Differentially Private Gradient Flow based on the Sliced Wasserstein Distance
Authors: Ilana Sebag
Muni Sreenivas Pydi
Jean-Yves Franceschi
Alain Rakotomamonjy
Mike Gartrell
Jamal Atif
Alexandre Allauzen
Venue: Accepted by TMLR ()

Abstract: Safeguarding privacy in sensitive training data is paramount, particularly in the context of generative modeling. This can be achieved through either differentially private stochastic gradient descent or a differentially private metric for training models or generators. In this paper, we introduce a novel differentially private generative modeling approach based on a gradient flow in the space of probability measures. To this end, we define the gradient flow of the Gaussian-smoothed Sliced Wasserstein Distance, including the associated stochastic differential equation (SDE). By discretizing and defining a numerical scheme for solving this SDE, we demonstrate the link between smoothing and differential privacy based on a Gaussian mechanism, due to a specific form of the SDE's drift term. We then analyze the differential privacy guarantee of our gradient flow, which accounts for both the smoothing and the Wiener process introduced by the SDE itself. Experiments show that our proposed model can generate higher-fidelity data at a low privacy budget compared to a generator-based model, offering a promising alternative.

Published in Transactions on Machine Learning Research (01/2025)
Differentially Private Gradient Flow based on the Sliced Wasserstein Distance
Ilana Sebag
i. Lscbag@criteo.com
Criteo AT Lab, Paris, France Miles Team, LAMSADE, Université Puris- DariisDee,,,e PSL University, CNRS, Paris, France
Muni Sreenivas Pydi
musni sagggkpppplllmmmsdddduugeeee
Miles Team, LAMSADE, Université Paris- Dorss-Dap PSL University, CNRS, Paris, France
Jean- Yves Franceschi Criteo Al Lab, Paris, France
byrjo.faaneechiieerttt..omm
Alain Rakotomamonjy
avakatomammmityy@ertttcccmm
Criteo AT Lab, Paris, France
Mike Gartrell
make.gortteellssiimmnnv...
Sigma Nova, Paris, France (Work done while working at Criteo Al lab)
Jamal Atif
Miles Team, LAMSADE, Université Paris-Dauphant PSL University, CNRS, Paris, France
Alexandre Allauzen
alezandre wwwwwwwwwwwdoppone.eeoooooomo
Miles Team, LAMSADE, Université Paris- Paris-Dauphine, PSL University, CNRS, Paris, France ESPCI PSL, Paris, France.
Reviewed OpenReview: https: // openreview. net/ forum? id= asOHc1LCpD
Abstract
Safeguarding privacy in sensitive training data is paramount, particularly in the context of generative modeling. This be achieved through either differentially private stochastic gradient descent or a differentially private metric for training models or generators. In this paper, we introduce a novel differentially private generative modeling approach based on gradient flow in the space of probability measures. To this end, we define the gradient flow of the Daassian-smoottlee differential equation (SDE). By discrotizing and defining numerical scheme for solving this SDE, we demonstrate the link between smoothing and differential privacy based on Gaussian mechanism, due to specific form of the SDE's drift term. We then analyze the differential privacy guarantee of our gradient flow, which accounts for both the smoothing and the Wiener process introduced by the SDE itself. Experiments show that our proposed model can generate higher-fidelity data at a low privacy budget compared to generator-based model, offering a promising alternative.
Sliced Wasserstein Distance, including the associated stochastic
1 Introduction
The widespread use of deep learning in critical applications has heightened concerns about privacy limitations, with various privacy attacks exposing vulnerabilities in machine learning algorithms (Shokri et al., 2017; Hu et al., 2022; Lacharité al., 2018; Mai al., 2018), including doep-learning-bart et al., 2020b; Carlini al., 2023). Differential Privacy (DP) has emerged as key solution counter privacy attacks, providing a robust framework to safeguard training data privacy (Dwork et al., 2006; Dwork, 2011;
generative models (Chen

---

Published in Transactions on Machine Learning Research (01/2025)
Dwork & Roth, 2014). DP ensures that single data point's inclusion exclusion minimally affects analysis outcomes.
In machine learning, DP is usually achieved by applying calibrated noise to gradient steps involving sensitive training data, with Differentially Private Stochastic Gradient Descent (DP-SGD) being a prominent example (A babdi et al., 2016). While extensively explored in classification, the application of DP in generative models is an emerging research area, often employing DP-SGD variants for training standard generative models (Xie et al., 2018; Chen et al., 2020a; Long et al., 2021; Dockhorn et al., 2023; Ghalebikesabi et al., 2023); either in the context of generator learning (Cao et al., 2021) or diffusion models (Dockhorn et al., 2023).
An under-explored alternative for DP-SGD is to minimize a functional on the space of probability measures:
min PrivatcCost(pu.)) + AReg(u), HEP(O)
(1)
where v is the probability measure to be modeled, PrivateCost is a cost functional on the space of probability measures P(2) that can be computed in a private manner, and Reg is a regularization functional that prevents over-fitting to training samples. Some works solve this problem by using this cost as a loss for training a generator. They employ differentially private versions of metrics such as Maximum Mcan Discrepancy or Sliced Wasserstein Distance (Harder et al., 2021; 2023; Rakolomamonjy & Ralaivola, 2021). Here, we propose alternative line of work that has never been explored, which explicitly minimizes the functional in Eq. (1) using gradient flows.
In a non-DP scenario, gradient flows are commonly employed to address minimization problems like the one described in Eq. (1) (Liutkus et al., 2019; Arbel et al., 2019; Franceschi et al., 2023; Galashov et al., 2024) and are viable alternative to other generative models (Fan et al., 2022). They possess inherent stability and convergence properties, thereby reaping significant benefits in the optimization process (Ambrosio et al., 2005a;b; Garg& & Panagou, 2021). Building on these strengths, gradient flows are particularly well-suited for capturing complex data distributions while maintaining stability during training. For instance, they provide a continuous-time approximation of discrete methods like SGD, facilitating the design of a multi-step push-forward operator that maps a noisy distribution to the target. This approach smooths the optimization path and provides insights into discretization effects. These properties make gradient flows particularly interesting to study in the context of differential privacy. Also, we expect them to enable incorporating differential privacy directly into the metric, rather than relying on DP-SGD as in diffusion models.
In addition, gradient flows have not been previously explored and analyzed within the DP framework. All of the above motivates the following question:
Can we develop principled formalism for preeecypreseeving generative modeling through gradient flows?
In this paper we provide an affirmative response to this question by presenting the theoretical framcwork for a differentially private gradient flow of the sliced Wasserstein distance (SWD).
Our approach involves defining the gradient flow on the smoothed SWD (Rakotomamonjy & Ralaivola, 2021), which is strongly related to DP as made clear latter. Despite its seeming simplicity, Gaussian smoothing, akin to related works on smoothed Wasserstein distance (Goldfeld et al., 2020; Goldfeld & Greenewald, 2020; Ding & Biles-Weed, 2022), introduces theoretical complexities and raises questions about the technical conditions of our gradient flow, including the existence and regularity of its solution. We overcome these challenges and formally establish the continuity equation of the gradient flow of the smoothed SWD, resulting in a smoothed velocity field.
This allows us to choose the discretization of the associated Stochastic Differential Equation (SDE) so that the proposed flow ensures differential privacy. We highlight that after discretization, the smoothing in the drift term acts as a Gaussian mechanism. Furthermore, we show that the discretization further amplifies privacy via the Wiener process in the SDE. This results in a sequential algorithm for which the differential privacy budget can be carefully tracked. We experimentally confirm the viability of our proposed algorithm compared to a baseline generator- based model trained with the same private SWD.
Notations. Throughout the paper we use Ω to denote the sample space. We assume that Ω is a compact subset of Rd For any subset A C TRd we use P(A) to denote the set of probability measures supported on
2

---

Published in Transactions on Machine Learning Research (01/2025)
A equipped with the Borel o-algebra. For W, P(2), we use I( S P(29) to denote the set of joint distributions "couplings" between and v. For μ E P(Ω) and a measurable function M: 52 -> Q2 the push-forward operator # defines a probability measure M# E P(2) such that M* = {u(M ((A) for all measurable S Ω. For r > O, we use r)) to denote a closed ball of center and radius r.
2 Background and Related Work
2.1 Sliced Wasserstein Distance
The Wasserstein distance between two probability measures ,, E P(9) is defined as (Peyré & Cuturi, 2019):
u)( = - inf (FET(() / || 21lBdT(1,
(2)
For p 2, i.e., for the squared Euclidean cost, celebrated result of Brenier (1991) proves that the optimal way to transport mass from j to is through measure-preserving transport map M: Ω - Ω of the form (xx)= = = Vvi(z), where th: Q + Ω is convex function termed the Kantorovich potential between μ and v, and M pushes onto v, i.e. M* v. In the one-dimensional case, the optimal transport map has a closed-form expression given by (Peyré & Cuturi, 2019)
M(z) = F F
33)
where Fu and Fy are the cumulative distribution functions (CDFs) of μ and v, respectively, In higher dimensions no such closed form exists.
The sliced Wasserstein distance (SWD) (Rabin et al., 2012) takes advantage of this simplicity of OT in one dimension by computing a distance between μl, V E P(2) through their projections Pop P E P(R) onto the unit sphere gd-1 = {0 Rd ||0]|2 = 1}. Here pº: Ω + R denotes the projection operator defined (x() = (z, 0). Formally,
SWi(u,D) = f J W vdd
(4)
where d0 is the uniform probability measure on sd-1, Like W2, the SWz also defines metric on P(2)) (Nadjahi et al., 2022).
2.2 Wasserstein Gradient Flows
A Wasserstein gradient flow represents an extension of the concept of gradient descent applied to functional within the domain of probability measures. More formally, it constitutes a continuous sequence (pe) of probability distributions within Wasserstein metric space (P(S)), Wz). Fx decreases along (Dt). more formally: the sequence follows a continuity equation (Santambrogio, 2016) with a general form of:
0p: = div (pr Vw,FF = div (e. Vw, + AApr, Øt
(5)
where = XH, F is the functional to be minimized and XH is an entropic regularization term. H is the negative differential entropy ensuring that the model can generalize and avoid overfitting: H(μ)) So p(x) log p(z)dz. À 2 signifies the strength of the entropic regularization. Pt. is the density of the probability flow (14)22 at time l. Vw,F(pp): Rd + Id is the Wasserstein gradient of F. Depending on the form of Fx. this continuity equation can be associated with a SDE which relates the evolution of (Pt) and particles (X))22 (Jordan et al., 1998):
dx, F(o)x(xx + 2 22dWW,
(6)
3

---

Published in Transactions on Machine Learning Research (01/2025)
where (Wi), is a Wiener process, and X₃ 14-
Wasserstein gradient flows possess a rich theoretical background (Ambrosio, 2008; Santambrogio, 2016). Notably they have been used for generative modeling, where initial distributions are represented by samples that evolve according to a partial differential equation (PDE) governing the gradient flow and defined on several metries such as the maximum mean discrepancy (Arbel et al., 2019) or Wasserstein-based metries (Mokrov al., 2021). Liutkus al. (2019) also present a non-parametric generative modeling method relying the gradient flow of the sliced Wasserstein distance, while Bonet et al. (2022) introduce flow defined in the sliced Wasserstein space by proposing numerically approximated Jordan- Kinderlehrer -ttoo (JKO) type scheme.
In this work we define the gradient flow of the smoothed sliced Wasserstein distance. Smoothing each sliced measure allows us to introduce differential privacy, at the expense of additional theoretical challenges on the definition of the flow.
2.3 Differential Privacy
Definition A random mechanism M: R is (c, if for any two adjacent inputs di, d2 E and any subset of outpuls S C R,
P[M((1) E S ccP()(d))) ES
(7)
Adjacent inputs refer to datasets differing only by a single record. DP ensures that when a single record in datasct is swapped, the change in the distribution of model outputs will be controlled by e and 8. controls the trade-off between the level of privacy and the usefulness of the output, where smaller values offer stronger privacy but potentially lower utility (e-g. in our specific case, low-quality generated samples). 5 is a bound on the external risk (e.g. information is accidentally being leaked) that won't be restricted by s; it is an extra privacy option that enables control of the extent of the privacy being compromised. In practice, we are interested in the values of 5 that are less than the inverse of a polynomial the size of the database (Dwork & Roth, 2014).
A classical example of DP mechanism is the Gaussian mechanism operating on function f: D Rd as:
My(d) = ((F(())
(8)
We define the fo sensitivity of f as A2(f)= = maxd, du: adjacenteD (l - fd2)lle. For c² > 2l1((1225/8) and a 2 c Aall)) the Gaussian mechanism is (s, ()DP (Dwork & Roth, 2014).
Several works deal with differentially private generative modeling, with most adopting DP-SGD (Xie et al., 2018; Chen al., 2020a; Long et al., 2021; Dockhorn et al., 2023; Ghalcbikesabi al., 2023). This approach is commonly employed in the context of generator learning (Cao et al., 2021) and diffusion models (Dockhorn et al., 2023).
An alternative solution for a DP generative model is to consider a DP loss function which the generator is trained. However, there is limited research on defining rigorous metrics that can be computed in a private manner, resulting in a gap for peeecc-preserving machine learning algorithms. While Lê Tien et al. (2019) use random projections to make Wi computation differentially private, they do not provide a theoretical analysis of the resulting divergence. Instead, Harder et al. (2021; 2023) have considered differentially private Maximum Mean Discrepancy as generator loss. Rakotomamonjy & Ralaivola (2021) introduce a Gaussian-smoothed version of SW2, defined as
f W2 Eo, Psu v+ Eo )d0
S.SW((,,)) = Iga-1
where &o ~ NN(, oZIx). They demonstrate that this distance and some extensions (Rakotomamonjy et al., 2021) are inherently differentially private, as the smoothing acts as a Gaussian mechanism. This allows for the scamless integration of this differentially private loss function into machine learning problems involving distribution comparisons, such as generator- generative modeling. In this work, we extend this trend by formalizing the gradient flow of the Gaussian-smoothed Sliced Wasserstein Distance.
4

---

Published in Transactions on Machine Learning Research (01/2025)
3 Differentially Private Gradient Flow
In this section we present the theoretical building blocks of our method, which consists of building a discretized gradient flow on the smoothed sliced Wasserstein distance. This smoothing and discretization will lead to a differentially private drift (vector field) in the gradient flow. In Section 3.1 we introduce the smoothing and the gradient flow of the Gaussia--mmothhed sliced Wasserstein distance of Rakotomamonjy & Ralaivola (2021) defined in Eq. (9), and prove the existence and regularity of its solution. In Section 3.2 we present a particle scheme to simulate the discretization of this flow and elaborate the link between the discretization, smoothing, and differential privacy. In Section 3.3 we present the privacy guarantee.
3.1 Gradient Flow of the Smoothed Sliced Wasserstein Distance
In this subsection study the following functional over the Wasserstein space (P(S), W2):
1 -G + XH(()),
(K) 2
(10)
where v E P(22 is the target distribution to be modeled and o > 0 is the smoothing of the probability measures in the inner optimal transport problem in S.SWg.
We will show later how and a relate to the privacy paramcters (c, in the discretized flow. The main result of this subsection the following: first, we establish the existence and regularity of Generalized Minimizing Movement Scheme (GMMS) for Eq. (10), and then we demonstrate that this GMMS satisfies the corresponding continuity equation.
Theorem 1. Let E P(B(O,r)) r)) have strictly positive smooth density. For X > 0 and r > vd, let the starting distribution Pa E P(B(Orr) have a density E (BO0, 1)). There exists minémizing movement scheme (pe)e20 associated with Eq. (10). Further, (Ho))20 admits densities (pt)ezo following a continuity equation:
Pp: = div(u, pi) + AApr,
(11)
Ōt
with:
1. (x) = 0.0) (z. = Jg fSLn (o)
(12)
(10) is the Kantorovich potential (see Section 2.1) between po * Eo and P u with &o ~ NNO, O0) Here, 0 and ils derivalive is given by Brenier (1991):
()))))(()) = -Fpgues. o FFggur, (2),
(13)
where Fp denotes the CDF of p.
Proof sketch. (1) We begin by showing that there exists a GMMS (see Appendix for the precise definition) for the functional in Eq. (1I). For this we show that the following optimization problem admits solution for any h > 0:
W (p.
Min E argmin NEP(N)
FX, (u) +
(14)
2h
Notice that the above problem is simply the implicit Euler scheme for deriving the gradient flow of FX,, over the Wasserstein space (P(2), W2). Since P(B(0, 1)) is compact for weak convergence, it is enough to show that Fx, is lower semi-continuous in W2. By Lemma 9.4.3 of Ambrosio (2008), 7l is lower semi-contimuous. By Rakolomamoniy & Ralaivola (2021), G.SWg(u, D) is symmetric and satisfies the triangle inequality. Moreover, G.SW2(1,)) < SW2(p, v) for any a 2 O. Hence for any g.g' E P(B(O, 1)),
(.. G.SWA(E P) S G.SWA(E.E") < SWa(S,E) $ c.W(E,)))
(15)
5

---

Published in Transactions on Machine Learning Research (01/2025)
where Ca >> 0 is constant only dependent on the dimension d and the last inoquality follows from Prop. 5.1.3 in Bonnotte (2013). Hence there exists a minimum á E P()(O,r)) of G(n), admitting a density ú, because otherwise H(2) = 0c. Further, we prove that the solution is "sufficiently regular" in Lemıaa 3 of Appendix A. Due to the distinct nature of the GWSW metric, proving the existence and regularity of the solution has never been done before and constitutes a novel contribution. To establish regularity (Temma 3), we adopt proof strategies inspired by Bonnotte (2013)'s previous work, but we need to adjust them fit our specific 9,SW case, requiring a complete construction from scratch.
(2) Next, show that the GMMS whose existence and regularity were previously established indeed satisfies the continuity equation in Eq. (11). A crucial ingredient in this step is the analysis of the first variation of the Gausian-smmoothed SW2 distance which is given in the following proposition (see Appendix A for proof).
Proposition 1. Let μ4 V E P(Ω). For any diffeomorphism Ç of
9wSW [d + e) G,SW3(U, = (V7, (W..))( ((=)) lim ビI
(16)
2x
Using the above result we then get the desired flow equation by closely following the proofs of Theorem S6 in Liutkus et al. (2019) and Theorem 5.6.1 in Bonnotte (2013).
Theorem shows that there is a continuous sequence of probability measures (H4)220 that constitutes a minimizing movement scheme for the functional in Eq. (10). Moreover, it shows that the probability density Pt of the minimizing movement scheme satisfies the continuity equation given by Eq. (11). Theorem 1 is generalization of Theorem of Liutkus et al. (2019) and Theorem 5.6.1 of Bonnotte (2013), which we retrieve when - 0. However, the proof does not trivially follow as corollary from these results, since we consider projected and then smoothed measures instead of directly applying previous results on smoothed measures before projection. Hence, we need two new pieces in the proof that are not present in prior works: (1) existence and regularity of solutions to the functional minimization problem, and (2) analysis of the first variation of the squared Gaaussan-smoothee SW metric.
One key point of our approach is that the drift term in the continuity equation relies on the Kantorovich potential v/
Hr.0 associated with the convolution of the Gaussian- and projected measures, specifically / &o and Pd = When transitioning from the continuous PDE to discrete-time SDE this Gaussian smoothing becomes crucial, ultimately leading to the Gaussian mechanism and ensuring differential privacy. We show in the next subsection that the combination of smoothing in projected measures and the discretization jointly establishes differential privacy.
3.2 Discretized and Private Gradient Flow Algorithms
In this subsection, we show how the gradient flow of Theorem 1 is discretized and how the smoothing acts as a Gaussian mechanism. We then present two algorithms that implement this gradient flow.
The gradient flow of S,SW2 given in Eq. (11) corresponds to a nonlinear Fokker-Plank type equation, where the drift term is dependent on the density of the solution. The evolution of (u))2o in Eq. (11) corresponds to stochastic process (Xt)120 that solves the SDE in Eq. (6). The latter can be discretized using the Euler- Mareyaaaa scheme with the random variable initialization Xo ~ Ho as follows:
8 = holl (Ex, PKA) + v 22h
(17)
where h > 0 is the step size, Pahk is the distribution of &G, and {Zeh are i.i.d. standard normal random variables. The discrete-time SDE in Eq. (17) can then be simulated by a stochastic finite particle system {81). where E N is the index for the 7th particle, and the discrete time index k runs from 0 to Kh = T (Bossy& & Talay, 1997). The particles are initialized Xi ~ po iid.. where each particle follows the update equation:
X 11500) Pha) + V 22h2k+11
(18)
6

---

Published in Transactions on Machine Learning Research (01/2025)
with ((7 Min) being an estimate of v/*) (Xx, PrA); cf. Eq. (20) for the details.
Naturally, approximating the continuous time SDE in Eq. (6) by the pascrete-time update in Eq. (18) leads to some error, which we provide bounds for in the following theorem. In the infinite particle regime, i.e,, as N + oc, under some assumptions of regularity and smoothness of the drift terms v(a) and Dim, we can state the following.
Theorem 2. Suppose that the SDE in Eq. (6) has a unique strong solution' (Xx)telo.ry for any storting point Zo E S2, such that Xr pT. For T = Kh, let Pich be the distribution of Xx in the discrete-time SDE in Eq. (17). Under suitable assumptions of regularity and Lipschitaness on vl0) and wil) stated in the Appendices, the following bound holds for any À > TL2/8:
T
1PT PKhliT S X A- TL2/8 [LPh(ech + dA) + o2x]
(19)
where e1, C2 K,K 0 are conslants independent of Lime (bul may depend on a).
The above theorem follows in straightforward manner from Theorem 3 of Liutkus et al. (2019) for the SW2 flow. It is worth noting that the error bound in Eq. (19) is possibly tighter than the corresponding error bound in Theorem 3 of Liutkus et al. (2019) for the special case of a - O, because the constant L can be shown to be a non-increasing function of a. The error bound depends linearly on the dimension d and will tend to 0 for K = 0 and small step size h.
We now delve into the numerical computation of the particle flow in Eq. (18). To evaluate p(0), we need two approximations. The first one approximates the distribution pus by the empirical distribution of the particles at time k. Pku zs Pih = x/2 E Oxg where here (6 is a Dirac distribution). The second one replaces the integral over A E gd-1 in Eq. (12), with Monte Carlo estimate, for f, a set of projections drawn from the sphere gd-1,
1 N₂ No
(x)
Σ (w ((r,
(20)
where (15(0) with &o N(0, and (1ble) (=))
0, / is the discrelized derivative of the Kantorovich polential between P (()) is defined (Brenier, 1991):
((N) Pth Eo and P v * Eor
0, ((z) = = F p 700.
o F Pove (z)
(21)
This equation is key in the gradiont flow since it is the main block of the drift term in Eq. (20) and it plays an essential role bridging the smoothing and the privacy. Indeed, since convolution of probability distributions boils down to the addition of random variables, Eq. (21) can be considered as a Gaussian mechanism (see Section 2.3) applied to the projected measures. In practice, we compute P Pib * = and / all the No projection vectors sampled uniformly from gd-1 and X= [ ",,,, Y = respectively, the dala matrices composed of the n i.i.d. samples from the target distribution and the empirical distribution (N)) Then, the smoothed and projected distributions P v So and P A x Eo can be respectively written as X + Zx and YO + Zy with Zx Zy E RR being the i.i.d. Gaussian random variables with variance o², corresponding to the Gaussian mechanism.
(N)
" &., in the following way. Let θ = [oT....., 0%,1 E R No be the random projection matrir composed of vi Raxd
From an algorithmic point of view, the random projection matrix θ E d No can either be sampled at the start of every discrete time step, or sampled once at the beginning of the algorithm and reused in every
AA strong solution an SDE means that it is defined over given probability space and Wiener process, while weak solution would be defined over some probability space and Wiener process. The interested reader can refer to egg. (Oksendal, 2000).
7

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm 1: DP Sliced Wasserstein Flow with resampling of 0's: DPSWftow-r.
1: Input: Y = ... ". E R i.e. N ii.d. samples from target distribution v, mumber of projections Ne, regularization parameter X, variance o, step size h.
2: Output: X x= [xf,,..,,,!! E Raxd
3: // Iritialiize the particles
{ar tai ~ no: X | I II Raxd
// Iterations of the flow
do 7: {0,) ~ U -= ....... 8%
for k k= o=.....,KK
Sample Zx, Zy RaxNO from i.i.d. (0,
Compute the inverse CDF of Y Zy
10: Compute the CDF of XO + Zx
Compute of") (xx) using Eq. (20)
11:
12:
Ti +- (a;) + 22222 (O)
13:
end for
step. These two strategies lead to DPSWflow-r (Algorithm 1) and DPSWflow (Algorithm 2) (the latter is detailed in Appendix C.3). The main difference between both lies in the sampling of the projections 0 of the sliced Wasserstein. In DPSWflow-r, we resample Ng projections at each iteration of the flow leading to a more expensive itcration. In DPSWflow, we sample all the No projections in advance (typically a larger amount) and use them in all subsequent iterations by subsampling; e.g al each iteration, we subsample a set. of projections among the pre-computed No projections. This enables us save on the privacy budget as it implies "free itcrations" in term of privacy. The choice between resampling or pro-sampling projections not explicit in the prior non-DP work of Liutkus et al. (2019). They adopt pre-sampling in the provided algorithm of their paper, whereas they resample projections in their code. In contrast, in our paper, we explicitly highlight the importance of this choice, for both privacy and performance issues.
Before clarifying this difference, we provide details regarding how privacy guarantees are dealt with in the gradient flow.
3.3 Privacy Guarantee
In this subsection we analyze the DP guarantee of the particle scheme outlined in the previous subsection. our gradient flow algorithm, there exist two independent levels of Gaussian mechanisms: (i) in the drift term, at the level of random projection of data through the matrix θ and (ii) the addition of the diffusion term 22 in the particle update Eq. (18). Then, comment how privacy impacts each of our particle flows in Algorithms 1 and 2, defined by Eq. (18),
3.3.1 Privacy guarantee arising from the random projection
To track the privacy guarantee arising from the random projection, we define a randomized mechanism MN guxdd - N as:
MN..o = X0 Zo.
(22)
where θ is the random projection matrix and Z, E Ra consists of i..d. Gaussian random variables with variance 2² Given X composed of (x;}; ~ v, the position of these partieles can be updated based on the drift of Eq. (20) which leverages MN..o (X) ) for computing Eq. (21). Hence, if M is (E, 6)-DP, then by the proceprocessing property of DP (Dwork et al., 2006), the computation of the drift term for all n particles in one time step is also O-D. To derive the DP guarantee for MN (X), we use the following lemma.
Lemma (Rakotomamonjy & Ralaivola (2021)). For data matrices X, X' E Rad that differ in only the 2h row, satisfying |X -5xxx I 1, and random projection matria O Bd whose columns are randomly sampled from g0-1, and No being large enough (No > 30), the following bound holds with probability at least.
8

---

Published in Transactions on Machine Learning Research (01/2025)
1:
Ne |X0 X'Ol S w(NN, 6), with, w(Na,
24-6
2No(d -
(23)
d
d
d+2
where Zi-6 = ---==1111 - 6) and + the CDP of zero-menn wnit variance Gaussian distribution.
3.3.2 Privacy guarantee arising from the diffusion term
To track the privacy guarantee arising from the diffusion term, we define a Markov operator KR,A : Raxd Rexd 88
(x) = |ook (x+) + 22h22"
(24)
i=1
where (1() (x) the drift term as defined in Eq. (20) and Z NN Ig). The following lemma from Balle et al. (2019) can then be used to characterize the privacy guarantoe resulting from both sourees of randomness. Lemma 2 (Balle et al. (2019)). Let K: X + Y be Markov operator satisfying the following condition for any N, x' E X:
[K(x)) - K(x'))ly ST.
(25)
Then for any (e,8)-DP randomized mechanism M, K. M is (E, DD)
Leveraging Lemmas and 2, we get the following theorem on the privacy guarantee of Algorithms and 2. Theorem 3. Under the setup of Lemma 1, the particle update in Eq. (18) is cwo(No.5) .)) v 2x DD, where c
is a constant satisfying e > 2lm(1.25/6).
Proof. For data matrices X, X' E Raxd that differ in only the 7th row, satisfying ||X; - -Xill2 S 1,
DKL. (x), Kn,x
DkI. (N(H6O0)(X3) (X.)),2XX,)) 2hAL2), (X.) 2hd1a)
| (X,) h6)l)) (X:)l² - h X
(26)
2(2hA)
where the last inoquality follows from the observation that (10T0) S Hence,
(X) - Kx. S 12DKA DKL (X),a (X') (L,))))), -
h
(27)
2x
Plugging in the sensitivity bound from Lemma 1 into the Gaussian mechanism presented in Section 2.3, see that the mechanism MN... is (DP for cw((N.6) The desired result. then follows from application of Lemma to the post-processed mechanism KN, MN....
We observe that the privacy parameter & degrades linearly with the number of projections No. while the 5 parameter decreases with the step size of the discretization. The scnsitivity depends lincarly on 1 va Thus,
the higher the d, the more private the mechanism is.
3.3.3 On the impact of (re-)sampling on privacy
Algorithms and differ by one element: in the former we choose a small No which is being resampled each step of the flow, whilst in the latter choose large No among which we sub-sample during the flow. The idea of reusing the random projections between iterations in Algorithm 1 has important effect on the resulting performance at similar privacy guarantees. Our intuition is that the DP bound is tighter for DPSWflow-r as we are allowed lo use the moment accountant. (Abadi et al., 2016) to derive the DP bound: this composition of DP operations is cheaper in privacy than doing them all at once at the beginning of the gradient flow. More details on the privacy guarantees of Algorithm and Algorithm are given in Appendix C.4. Also, resampling enables unbiased gradient estimates across iterations, which, we suppose, is important for generation performance. Experimental results reported for both mechanisms in Section 4.5.
9

---

Published in Transactions on Machine Learning Research (01/2025)
4 Experiments
In this section evaluate method within generative modeling context. The primary objective is to validate our theoretical framework and showcase the behavior of our approach, rather than strive for state-of-the-ar results in generative modeling. Our study focuses on specific claim: in a privacy setting, a gradient flow performs better than generator-based models trained with the same metric. The code for the experiments can be found https: //gi tuub..u..... com/ commilaan http:/llaaaebaappppaaa/a
4.1 Toy Problem
We use a toy problem to illustrate how our differentially private sliced Wasserstein gradient flow behaves compared to the vanilla non-DP baseline. Here we set Nç = 200, h = 1, and À = 0.001. Examples are provided in Fig. 1. We see that the particle flow of the sliced Wasserstein can correctly approximate the target distributions that are composed of 5 Gaussians, as measured by SWD. With the Gaussian smoothing, for a = 0.5, the particles are still able to match the target distribution, although the samples are more dispersed than for the noiseless SWF, leading to a SWD value of 0.81 instead of 0.08. Finally, for a = 1, our approach struggles in matching the true distribution, although many particles are still within its level sets.
SWiD-11 1162
SWD-1 Ao
SWD-6,00
SWD-1.44
SWD=0.81
SWD-5
SWD-2 21
SWD-2
33
30
Figure 1: Examples of particle flows for (top) sliced Wasserstein flow, (middle) our DPSWflow with a 0.5, and (bottom) 1. Each panel shows the level sets (in black) of the target distribution, which is composed of 5 Gaussians, well as the particles (in green). The columns depict the particles after the (left) first step, (middle) 10-th, and (right) the 200-th steps of the flow.
4.2 Comparisons and Baselines
demonstrate our claims we test our method on three mechanisms: DPSWflow-r (Algorithm 1), DPSWflow (Algorithm 2) and DPSWgen. DPSWgen is a generator-baser model adapted from Rakotomamonjy Ralaivola (2021) trained using the differentially private sliced Wasserstein distance. While both DPSWgen and our method use the same metric to measure distributional similarity, they differ in how samples are generated: DPSWgen employs one-step generative model that maps Gaussian noise directly to the target distribution. In contrast, our method generates samples through gradient flow, iteratively transitioning from Gaussian distribution to the target distribution, acting as a multi-step generator. Therefore, it is differentially private because its drift term built through Gaussian mechanism. Both approaches are fundamentally distinct.
To maintain suitably low input space dimension, in order mitigate the curse of dimensionality and reduce computational cost, mechanisms and 2 are preceded by autoencoder with Z, as the latent space. Subsequently, they take the latent space 2, S Rd of the autoencoder as the input space, and ensure differential privacy using the DP gradient flow. To ensure fair comparison, DPSWgen is also preceded by the same autoencoder with Z, as the latent space. Subsequently, it also takes the latent space 2,, S IRd of the autoencoder as the input space.
We evaluate the three algorithms using the Fréchet inception distance (FID, Heusel al., 2018). In our results, present each method three levels of differential privacy: e (no privacy), e 10, and & 5,
10

---

Published in Transactions on Machine Learning Research (01/2025)
along with their corresponding FID scores. In this context the optimal generated images from each model are expected to yield the best achievable FID score when oc.
4.3 Sensitivity and Privacy Budget Tracking
For both DPSWflow-r and DPSW we monitor the privacy budget using the Gaussian moments accountant method proposed by Abadi et al. (2016), where we choose range of a's satisfying the constraint or 2 evo( Ng.6) where w is the sensitivity bound from Lemma 1, c >>>>>>>>> ll(1125//), and we use the moment accountant to obtain the corresponding s's. Also, to prevent privacy leakage, we normalize the latent space of the autoencoder (which is used as input of the flow and the generator) to norm 1, so we incur an additional factor of 2 in the sensitivity bound.
4.4 Settings and Datasets
In all three DPSWNow, DPSWNow-r, and DPSWgen models, we pre-train an autoencoder and then use DP sliced Wasserstein flow component, DP generator, respectively. In order to uphold the integrity of the differential privacy framework and mitigate potential privacy breaches, conducted separate pre-training procedures for the autoencoder and the flows generator using distinct datasets: a publicly available dataset for the autoencoder, and a confidential dataset. for the flows generator. In practice, we partitioned the training set X into two distinct segments of equal size, denoted as X and Xpriv, Subsequently, we conducted training of the autoencoder on X Then, we compute the encoded representation on X X in the latent space and use as input to DPSWflow, DPSWiowwr, and DPSWgen. Furthermore, as mentioned in Section 4.3, to prevent privacy leakage normalize the latent space before adding Gaussian noise, ensuring that the encoded representations lie on hypersphere.
We assessed each method on three datasets: MNIST (LeCun et al., 1998), Fash (F-MNIST, Xiao et al., 2017), and ColebA (Liu et al., 2015). The experiments performed on the MNIST and FashionMNIST datasets use the same autoencoder architecture as the framework proposed by Liutkus et al. (2019). The experiments conducted on the CelebA dataset use an autoencoder architecture adapted from a DCGAN (Radford al., 2016b). More details on these architectures are given in Appendix C.2.
4.5
Experimental Results
This section outlines our experimental findings, including the resulting FID scores (Table 1) and the generated samples for each of our experiments (Figures Figs. 2 to 4). We compute the FID the average of the FID scores obtained from five generation runs. Also, for each experiment, we use d > O: å = 10-5 for MNIST and FashionMNISTT, and J = 10-6 for CelebA. Fig. presents the results from our model and the baseline without any differential privacy applied. Figs. 3 and show the results for e = 10 and E = 5, respectively.
Fig. 2 presents the results from our model and the bascline without any differential privacy appliod. Figs. and 4 show the results for 10 and 5, respectively.
We observe that methods outperform the DPSWgen baseline for all privacy budgets tested in experiments, both in terms of FID scores and the visual quality of the generated samples. Furthermore, the variant of our approach with resampling (DPSWflow-r) consistently outperforms the variant without
Table 1: FID results for each bascline, dataset and privacy setting, averaged over 5 generation runs.
MNIST
F-MNIST
CELEBA
10
10
10
DPSWgen
114
124
198
138
170
199
171
209
214
DPSWflow-r DPSWilow
21
70
114
42
88
98
57
132
197
73
118
171
96
98
129
134
262
292
11

---

Published in Transactions on Machine Learning Research (01/2025)
4
4
2
7
2
3
/
2
/
6
5
9
2
7
4
/
6
7
9
7
2
3
3
3
3
9
3
2
3
3
9
6
Figure 2: Generated images from DPSWilow-r (upper row) and DPSWgen (lower row) for MNIST, Fashion- MNIST, and Celeba with no DP: 00.
$
d
8
5
V
<
3
7
5
5 8
4
6
SF
5
Ø
1
9
F
/
L
5
7
4
4
Figure 3: Generated images from DPSWflow-r (upper row) and DPSWgen (lower row) for MNIST, Fashion- MNIST, and Celeba with DP: & 10.
5
$
5
2
+
1
5
Figure 4: Generated images from DPSWflow-r (upper row) and DPSWgen (lower row) for MNIST, Fashion- MNIST, and Celeba with DP: & = 5.
12

---

Published in Transactions on Machine Learning Research (01/2025)
resampling (DPSWflow). These results support our consideration of the efficiency of resampling the projections as discussed at the end of Section 3.2. These experiments show that our approach is practically viable and can serve as promising basis for future work on private generative models.
5 Conclusion
In this paper we have introduced a novel theoretically grounded method for differentially private generative modeling. Our approach leverages gradient flows within the Wasserstein space, with a drift term computed using the differentially private sliced Wasserstein distance. To the best of our knowledge, we are the first to propose such a DP gradient flow approach. Our experiments have shown that our approach is practically viable, leading generated samples of higher quality than those from generator-based models trained with the same DP metric at the same level of privacy. With both strong theoretical foundation and experimental viability, believe that our method forms a promising basis for future work in the area of private generative modeling.
Broader Impact Statement
Some aspects of our contributions in this paper are theoretical in nature, and we do not foresee any adverse societal impacts resulting from them. Our experiments small datasets, and have negligible carbon pootprint. Our adherence to differential privacy principles ensures that our generative model aligns with privacy-preservie principles.
References
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Tyya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Compuler and Communications Securily- ACM, oct 2016. doi: 10.1145 29144/99777799997818888 URL
attps://doi.org http:///boo.orrr11.1111/////22/ 1111121299777999..111 2978318.
Luigi Ambrosio. Gradient flows in metric spaces and in the spaces of probability measures, and applications to fokker-planck equations with respect to log-concave measures. Bollettino dell' Unione Matematica Italiana, 1(1):223 240, 2 2008. URL http://eudml. org/ org//oc/290477.
Luigi Ambrosio, Nicola Gigli, and Giusoppe Savare. Gradient Flows. Lectures in Mathematics ETH Zürich. Birkhäuser Basel, 2005a. URL https://www2 stat. duk...ukeeeee www..e..d./ssaaaaaaabbbbaaaaee www.saaaaaaaaaaaiio..a. .ddf
Luigi Ambrosio, Stefano Lisini, and Giuseppe Savaré. Stability of flows associated to gradient vector fields and convergence of iterated transport maps. manuscripta mathematica, 121:1-50, 01 2005b, doi: 10.10077/002230-00000003
Michael Arbel, Anna Korba, Adil Salim, and Arthur Gretton, Maximum mean discrepancy gradient flow, 2019. URL https://arxiv. http:///axxvvooraabb/////ss attpaa//s....//////////eeeeeeeea 1t/.199060044 04370.
Borja Balle, Gilles Barthe, Marco Gaboardi, and Josoph Geumlek. Privacy amplification by mixing and diffusson mechanisms, 2019. URL https://arxiv harg/abs/1900 12264.
Clément Bonet, Nicolas Courty, François Septier, and Lucas Drumetz. Efficient gradient flows in sliced- wasserstein space. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https:
//openreview.neteee. http://......orrr///aaaaaa..aaa.
Nicolas Bonnotte. Unidimensiiona and evolution methods for optimal transportation. PhD thesis, Université Paris Sud-Paris XI; Scuola normale superiore (Pise, Italic), 2013. URL http:::://wwww..oaaaaa...eee org/
-bomottte/aoc/ppd--oonnet
Mircille Bossy and Denis Talay. A stochastic particle mothod for the mckoan-vlasov and the burgers equation. Mathematics of computation, 66/217):157-19,, 1997. URL https: //wp///wwwppppp/////// inria.iii... fr ft/mmeerreb//b/ei......... http:ichieeeeeeeeeo/aaaaaaaaaaaa pdf.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Yann Brenier. Polar factorization and monotone rearrangement of vector-valued functions. Communications on pure and applied mathemalics, 44(4):375-411 1991. URL https://www.cereet ceremade dauphine fr/ "carlier/Brenierg" .dff.
Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, and Karsten Kreis. Don't generate me: Training differentially private generative models with sinkhorn divergence, 2021. URL https://arxis: wwwpiiiiiorroooooooooooooo org/abs) 2111.01177.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Schwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, Anaheim, CA, August 2023. USENIX Association. ISBN 978-1-939133-37-3. URL https: htapp:://wwww usenix org/ httg/coofeeenceee wetpccceeeseeixexeexxeee usenixsecurity23/
presentation/ cerrriionncaanii.... carlini.
Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. Gs-wgan: A gradient-sanitized approach for learning differentially private generators. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 12673 12684. Cur- ran Associates, Inc., 2020a. URL httpp::/pproceednggg ing.ooeeednnngg...rr...s..p .eurips tataper_fileesspper///oo 3547aa66651128777a677611aaaaaa
pdf..
Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: taxonomy of membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC Conference Computer and Communications Security, CCS '20. ACM, October 2020b. doi: http:// /tx doi. www.ddi..oo.....o.. org/10. 100.111//33299..3111111
3412278 URL
Yunzi Ding and Jonathan Neles-Weed. Asymptotics of smoothed wasserstein distances in the small noise regime, 2022. URL
06452.
Tim Dockhorn, Tianshi Cao, Arash Vahdat, and Karsten Kreis. Differentially private diffussio models. Transactions on Mochine Learning Research, 2023. ISSN 2835-8856. URL https://openrevier.. forum?id-z2ppoKTEXX
Cynthia Dwork. A firm foundation for private data analysis. Commun. ACM, 54:86-98 95, 04 2011. doi: 00.1145/186639918666 1866758.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sei., (13-4):211-407 aug 2014. ISSN 1551-305X. doi: https://doi. 1tte:100151110400000000
URL
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to scnsitivity in private data analysis. In Theory of Cryptography Conference, volume Vol. 3876, pp. 265 284, 01 2006. ISBN 978-3-540-32731-8. doi: 10.1007/11681878 14.
Jiaojiao Fan, Qinsheng Zhang, Amirhossein Taghvaei, and Yongxin Chen. Variational wasserstein gradient flow, 2022. URL https://arxiv wwwp:bs/////////
httpaaaa///aa.aaa/a/eaaeaaaaaaaa 2ttbbb///222.22222 02424.
Jean- Yves Franceschi, Mike Gartrell, Ludovic Dos Santos, Thibaut Issenhuth, Emmanuel de Bezenac, Mickael Chen, and Alain Rakotomamonjy. Unifying GANs and score-based diffusion as generative particle models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openrevien
training, 2024. URL net//oorum/d--MMEERROD
Alexandre Galashov, Valentin de Bortoli, and Arthur Gretton. Deep mmd gradient flow without adversarial http://aiivv..rraaaa//./..../../
Kunal Garg and Dimitra Panagou. Fixed-time stable gradient flows: Applications continuous-time optimization. IEEE Transactions on Automatic Control, 66(5):2002- 2015, May 2021. ISSN 2334-3303. doi: 10. 10.1109/1ac.2020.3000 URL attp://dxx doi orw.oorg/....1111 1109/TAA
14

---

Published in Transactions on Machine Learning Research (01/2025)
Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Ira Ktena, Robert Stanforth, Jamie Hayes, Soham De, Samuel L. Smith, Olivia Wiles, and Borja Balle. Differentially private diffusion models generate useful synthetic images, 2023, URL
13861.
Ziv Goldfeld and Kristjan Greenewald. Gaussian-smoothhed optimal transport: Metric structure and statistical efficiency. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statisties, volume 108 of Proceedings of Machine Learning Research, pp. 3327 3337. PMLR, 26 28 Aug 2020. URL https: sttps://proceedingg mll. www.-ppaas/v00//// / gold html.
Ziv Goldfeld, Kristjan Greenewald, and Kengo Kato. Asymptotic guarantees for generative modeling based on the smooth wasserstein distance, 2020. URL https::/arxiss http::///aavvv.raaaaaa/e wwwggabs//002..o 01012.
Frederik Harder, Kamil Adamczewski, and Mijung Park. DP-MERF: Differentially private mean embeddings with random features for practical privacy-preserving data generation. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 1819 1827. PMLR, 13 15 Apr 2021. URL www.mmlr...... www..pprees//v1///aarreee htt.v//////aaaaaaa..aaaa html.
Frederik Harder, Milad Jalali, Danica J. Sutherland, and Mijung Park. Pre-trained perceptual features improve differentially private image generation. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openrevier net/forrumidd-RGi7ZkkM..
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium, 2018. URL https://arxiv.org 0////7706.880
Hongsheng Hu, Zoran Salcie, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. Membership inference attacks on machine learning: survey, 2022. URL ittps://arxivv http///xxvvooogabb////a/a wtgg:b////1...00//////// 07853.
Richard Jordan, David Kinderlehrer, and Felix Oito. The variational formulation of the Fokker-Plaan equation. SIAM Journal on Mathematical Analysis, 29(1):1-17 17, 1998.
Marie-Sarah Lacharité, Brice Minaud, and Kenneth G Paterson. Improved reconstruction attacks on encrypted data using range query leakage. In 2018 IEEE Symposium on Security and Privacy (SP), pp. 297-314. IEEE, 2018 URL https: //tprit/eprint.... iacr.aaroooo org/2017/701 pdd.
Nam Lē Tien, Amaury Habrard, and Mare Sebban. Differentially private optimal transport: Application to domain adaptation. In IJCAI, pp. 2852 2858, 2019. URL https: http:/www........o org/ 2tg/proocedings/20019/ 0395. pdf.
Yann LeCun, Léon Botton, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278 2224,, November 1998. URL http://visionn stanford. http://o_sppiing//ppprr////////e
pdf.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild, 2015.
URL /tatps///axxiv..r www.aawwww....vvaaaaaaaa wttp/bs///g...//////////
Antoine Liutkus, Umut Şimşekli, Szymon Majewski, Alain Durmus, and Fabian-Rober Stöter. Sliced- wasserstein flows: Nonparametric generative modeling via optimal transport and diffusions, 2019. URL
https://arxiv. httppa///aaaivooo///ba//////// /a////.bb/00......
Yunhui Long, Boxin Wang, Zhuolin Yang, Bhavya Kailkhura, Aston Zhang, Carl Gunter, and Bo Li. G-pale: Scalable differentially private data generator via private aggregation of teacher discriminators. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.), Advances in Neural Information Processing Systems, volume 34, pp. 19655-9977 Curran Associates, Inc., 2021. URL https://proceediie .pdf.
neurips.cc/papp http:///p///////////ta///aaaaaaa
15

---

Published in Transactions on Machine Learning Research (01/2025)
Guangean Mai, Kai Cao, Pong C Yuen, and Anil K Jain. On the reconstruction of face images from deep face templates. IEEE transactions on pallern analysis and machine intelligence, (5)::1188-11200 2018. 00832.
URL https://axxiv.org///
Petr Mokrov, Alexander Korotin, Lingxiao Li, Aude Genevay, Justin Solomon, and Evgeny Burnaev. Large- scale wasserstein gradient flows, 2021. URL httpa://arxiv. wttpexiivoooggba///./// wtg. httpaaa///111...0.0... 00736.
Kimia Nadjahi, Alain Durmus, Lénaïc Chizat, Sohcil Kolouri, Shahin Shahrampour, and Umut Şimşekli. Statistical and topological properties of sliced probability divergences, 2022. URL https://axxiv..n
abs/2003.05783
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. 2019.
Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. Poundations and Trends& in Machine Learning, 11(5-6):355 607, 2019, ISSN 1935-8237. doi: 10.1561/ 2200000073. URL http://dxx
doi. wttp./ooo...o/......./o 15511/2000000073.
Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and its application texture mixing, In Scale Space and Variational Methods in Computer Vision: Third International Conference, SSVM 2011, Ein- Gedii Isruel, May 29 June 2, 2011, Revised Selected Papers 3, pp. 435 446. Springer, 2012. URL https: http///haaaaa. sciencc/haa--047666666 document.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Leorning Representations, 2016a.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised reprrrestoonntn learning with deep convolutional generative adversarial networks, 2016b. URL
httpaaaaiwi...aaa////aa/1/a/ 06434.
Alain Rakotomamonjy and Liva Ralaivola. Differentially private sliced wasserstein distance. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8810-8820. PMLR, 18 24 Jul 2021.
Alain Rakotomamonjy Mokhtar Z. Alaya, Maxime Berar, and Gilles Gasso. Statistical and topological wrg//oo////aaaaaaao/
properties of gaussian smoothed sliced probability divergences, 2021. URL 10524.
Filippo Santambrogio. Euclidean, Metric, and Wasserstein gradient flows: an overview, 2016. URL https://arxiv.org/ ab69////////....//.///////
Reza Shokri, Marco Stronati, Congzhong Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models, 2017. URL https: : http///xxvvvv.//// wwwpaawwwwrg/b////........
Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Falcon. TorchMetries - measuring reproducibility in PyTorch, February 2022. URL
CaaeiiiiiinnnAttttttiitooee AI/torchmetrics.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- a novel image datasct for benchmarking machine learning algorithms, 2017. URL https://arrivv.og//abb/// /ttp://.0bb/000.0777....
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network, 2018. URL https: sttpps:/aatee http///xxvvvorrgee/e wwwp/o//////.//.//..50
Bernt Oksendal. Stochastic Differential Equations: An Introduction with Applications, volume 82, 01 2000. ISBN 978-3-540-60243-9, doi: 10. 10.1007/978-10 078177787788---66233788--88--
16

---

Published in Transactions on Machine Learning Research (01/2025)
A Proof of Theorem 1
We begin by presenting two propositions that generalize Proposition 5.1.6 and Proposition 5.1.7 of Bonnotte (2013), respectively. These propositions will play crucial role in the proof of Theorem 1, and constitute a key element of novelty in our proof, compared to the proof of Theorem 2 in Liutkus et al. (2019).
Indeed, the SwS metric is not a simple application of the sliced Wasserstein metric to Gaussian convoluted measures. The convolution with Gaussian measure in the G,SWg metric occurs within the surface integral, separately for each one-dimensional projection of the original measures. To reiterate, (,, v) * (W (o, Eo), where &o o). This distinction becomes clear when comparing equations 4 with 9. Hence, establishing DP gradient flow presenled unique challenge which makes distinct contribution. The distinct nature of G,SW? metric introduces two blockers that need to be circumvented before applying results from Bonnotte (2013) and Liutkus et al. (2019), namely the existence and regularity of minimizers to the functional in Equation 10. Both these steps are non-trivial.
Proposition 2. Lel ,4, E P(2). For any i E P(Ω),
lim 9,SW(( - E)μ + tTT, S.SW((M, f I volol (())((1,a)))(( - -1))d)
(28)
2x
Jg
where u/ a is a Kantorovich potential between 0z * No and 0p * No.
Proof. Since 85v * No is absolutely continuous with respect to the Lebesgue mcasure for any gd-1, there indeed exists Kantorovich potential v between 0sp No and 0.v * No. Since v/()) may be optimal between (1 =)μ EØ and v,
lim inf G,SW2((1 -E)p + E) - (,SW3(H,P)) 2 f (1 *
p)d0.
(29)
Conversely, let w/a.e) bea Kantorovich potential between с)μ + eñ] No. Then,
No and OivNNo with f whog d0;[(1-
8.
S. +ū 1 Se e 1/ (0 ((0, 4)d0.
(30)
As the proof of Proposition 5.1.6 Bonnotte (2013), ( uniformly converges to a Kantorovich potential
for (0.u * No,0,u 0.v Na) as + 0+ Hence,
G,SW( E)P + ED. GoSW S ff
wirdd (10, 4)d0.
lim sup +-
(31)
2s
and
Combining Eq. (30) and Eq. (31); get the desired result.
Proposition 3. Let μ1 / P(Ω). For any diffeomorphism Ç of S
lim G.SW3 + EClap, G. f I /
(()))
(32)
2s
where w/ is a Kantorovich potential between Ozp * No and 0.v x No.
Using the fact that (/ is Kantorovich potential between 0.p No and G.v No. get the following:
Proof.
((()) (00, =((x))) (0)) (0.) x)) (
G,SW3 + -
dpd0.
(33)
2s
gg--
2s
17

---

Published in Transactions on Machine Learning Research (01/2025)
Since the Kantorovich potential is Lipschitz, it is differentiable almost everywhere, and so, we get the following using Lebesgue's differentiation theorem:
lim inf 9.SW [ld + ESlyu, v) G.SWE(u.)) 5
f I (10,))0000
(34)
Conversely, we will now show the same upper bound on the lim sup. Let 70, E 11(00uu * No, 0.v * No)) be the optimal transport plan corresponding to wi As is done in Proposition 5.1.7 in Bonnotte (2013), we extend Te, a to Ta.o E El1(u, 2) such that (0 & 0) = 70,a. In other words, if random variables (X, are sampled from ne.a: then ((X,0) 0). (Y, 0)) will follow the law of 70 for every 0 E s Then, it follows that [0 + (C) @ To E [Td + 8lu,,, 0gu). Hence,
G.SWa (Id+ (Id++<hu,, 9. SW S [10.2 +((()) - 10)2-1862 d ().
Since Ta,a is constructed from na which in turn is based on the Kantorovich potential w/ vm.00 we have (0. v) (0, z) (6/)) ) ((())x)) for -a.e (z. v), because of the optimality of 18. for the one-dimensional optimal transport. Therefore,
G.SW ([] v) SoSM (4, v) s ff ( 1( p..0) ((0, x))x x))l doo (2, y)do.
-
Simplifying the right hand side of the above equation and taking the limit of E - 0+, get the following:
GoS + =Clap, v) G. GoSW3 (H, S ff J (101100 ((00)))))..(
lim sup
(35)
5-+1
2x
Combining Eq. (35) and Eq. (34), we get the desired result.
We reproduce the following definition from (Liutkus et al., 2019). Definition 2. (Generalized Minimizing Movement Scheme (GMMS)) Let r > and F R4 P(B((), r)) x r)) + be a functional. For h > O, let yrh [0, o0) - (B, r)) be a piecewise constant trajectory for F slarting al pa P(B(,,))) such that: (i) n² (0) pa, (ii) (1) = (nhh for n L1/h], and (vii) d(nn 11h+ mènimizes the functional ( F(h, (nh)), for all N.
We say that & is Minimizing Movement Scheme (MMS) for F starting at pu if there exists a family of piecewise conslant Irojeclories (p²) )n20 for F such that lim, 1²t (1) = ja(t) for all L E It..
We say that i is a Generalized Minimizing Movement Scheme (GMMS) for JF starting at po if there exists a family of piecewise constant trajectories (1²²) meN for F such that limn 1 (t) fi(t) for all E R4.. Theorem 4 (Existence of solution to the minimization functional). Let v E PBB 1)) and > vd. For any no E r()(,,r)) with densily po E L (B((,,)))) and h > 0, there exists a á E P(B(O,r) that mónimizes the following functional:
1 W3(H Po).
= + 2h
(36)
where F1) is given by Eq. (10). Moreover fa admils a densily P on B(0,r).
Proof. We note that P(B 1)) is compact for weak convergence (and equivalently for convergence in W2). Hence, showing that G(u) is lower semi-continuous P(B(O, 1)) would suffice to show the existence of a solution A.. By Lemma 9.4.3 of Ambrosio (2008), H is lower semi-contimooss By Rakotomamonjy & Ralaivola (2021), G,SW2 v) is symmetric and satisfies the triangle inequality. Moreover, G.SSW(u, & SW2(u, v) for any o 2 O. Hence for any E.5' E P(B(O, 1)),
(9,SW((E))) - SWW x) S G.SWA(E.E") S SWa(E,E) S caWVE,E'),
where Ca > a constant only dependent on the dimension d, and the last incquality follows from Proposition 5.1.3 in Bonnotte (2013). Hence, there exists minimum P P P(B(O,, r)) of g(p). Moreover, f must admit a density á because otherwise H(A)
18

---

Published in Transactions on Machine Learning Research (01/2025)
Lemma 3 (Regularity of the solution to the minimizing functional). Under the assumptions of Theorem 4, any minimizer fa of (()) in Eq. (36) must admil striclly posilive density ò >> 0 a.e., and Il S (1 + 1
Proof. By Theorem 4, a minimizer á of G(p) exists and admits density P. Let j E P(B(O, be an arbitrary probability measure with density p. For e E (0, 1) let Ps (1 ε)ă Ep and let Me E (O, 1)) be the probability mcasure corresponding to the density Pe. By the optimality of (, we have that G(A) S g(pe). Hence,
(()) - G(ps) lim
(P) (W
H(D) - -H((s)
Wz( - -W (Pz)
0 2 lim
+ A Alm sup e-+0+
+ lim 5
c101
2x
2he
f / (0) (0, p)do + sup
H(O) Hl(Ee)
/ Ød jt),
s-+0+
where the last equality follows by combining Proposition with Proposition 1.5.6 in (Bonnotte, 2013). Here, vl(o) is a Kantorovich potential between 0gû * Na. and 0g * No as in Proposition 2 and o is a Kantorovich potential between f and v for W2. Rearranging, we get the following:
H(A) -((m)) I
1 /
lim sup ++0
vd(A -)). P))
(37)
À B(O.r)
where ((x) = fau ( x)) + (z)). From this point, for any μo E P(B(O,r)) with density po that is smooth and strictly positive, get the desired result by following the proof strategy of Lemma 5.4.3 in (Bonnotte, 2013). For more general Ni with density Ao E (B(O, we again arrive at the desired result by following the proof strategy of Theorem S4 in (Liutkus et al., 2019), which proceeds by smoothing Po by convolution with a Gaussian.
Theorem (Existence GMMS). Under the assumptions of Theorem 4, there erists a GMMS (H1)22 in P()(O,r)) starting from po for the following functional:
1
F% (h, Uuzt; Myru) Fx (Hinz)
W W2 (Mnatt
(38)
2h.
Moreover, for any >> 0, Pr has a density Pt such that || S @dvv/|pool.s
Proof. The desired result follows straightforwardly by following the proof of Theorem S5 in Liutkus al. (2019) or Theorem 5.5.3 in Bonnotte (2013), with the support of Lemma 3 and Theorem 4.
Theorem 6 (Contimuity equation for GMMS). Under the assumptions of Theorem 4, let (uu)i20 be the GMMS given by Theorem 5. For gd-1, let w/ be the Kantorovich potential between Pop * &a and Ph Eo, with &o NO, For t 0, the density pt of pe satisfies the following continuity equation in weak sense:
Ap:
div(v)
(0) p.) AAp..
Øt
with:
of (x) = -000 (e) { 19 (1(l)) 0))000.
That is, for all & E C2 (I0, 00) B(0,r)),
1000 (
8 (t,z)
VE(t,x) -AA(tt)) (( p(())dzdt= - JBO..) ((0,x)po(z)dz. (39)
B(on)) 0t
19

---

Published in Transactions on Machine Learning Research (01/2025)
Proof. We will closely follow the proof of Theorem S6 in Liutkus al. (2019) and Theorem 5.6.1 in Bonnotte (2013). Just as in the proof of Theorem S6 in Liutkus et al. (2019), we will proceed in five steps.
Step (1): By the definition of GMMS, there exists a family of piecewise constant trajectories (u MEE for Fx,a such that lim 14 = jl for all E Rt. Let & E C ([0, (e) x B(O, r)) and let Ej (x) denote glkhn, 2). Using step of the proof of Theorem S6 in Liutkus al. (2019), we get:
/
05 5(0, ((0))))p(z)dz + T.²0 / Sear, ōt (z)ddt = lim hn EL (x) B(0,r)
An Pkh
(x)
No P(k-- 1)Am (I)
dr.
n-000 kml B(O,r)
hn
(40)
". Step (2): For any 0 s let w be the Kantorovich potential between Pi ho En and po v Eor with
So ~N(00, Using step the proof of Theorem S6 in Liutkus et al. (2019), we get:
lim hn Σ B(o.r) f n-+00
(41)
Step (8): From step 3 of the proof of Theorem S6 in Liutkus et al. (2019), we get:
(x) (x)dx= = 1000 / ()((,
lim In +2000 k=1 B(O.r)
(42)
Step (4): Let ob be the Kantorovich potential from the ((k-1)).. From the optimality phn, the first. variation of the functional (+ FX)((() + 1 W2 m&R,)) with respect to Ç al the point Pen. in the direction of the vector field VER is zero. Using Proposition 3 for the first variation of the G&SW term, Proposition 5.1.7 for the first variation of the W term, and Jordan et al. (1998) for the first variation of the term, we get the following:
25
1 (V))"()), V (z) Am (x) (z) -SRR.
f(ot2 (=)))dddpk) fon (4) A3n
0
hn B(o.r)
AEE (x). B(oor)
(43)
Proceeding as in step 4 of Liutkus et al. (2019) and using Eq. (43), we get the following:
Pkin
Ptr 11hm
lim n-0
hn
(x)
dr
hn
k=1
lim
hn
Σ A=1
20
S f
(yloohayy00)) ded)dd + hn EL B(o.r)
(cX()) (o)dxr
(o)oo)
(44)
--000
B(o.r)
Step (4): Combining Eq. (40), Eq. (41), Eqq (42) and Eq. (44), we get the desired result in Eq. (39).
B Proof of Theorem 2
We simply follow the proof strategy of Theorem 3 in Liutkus et al. (2019). We begin by restating the following two discrete-time SDEs:
8 = (iu(o)(8x)) (uRn)) 22hh
(45)
X h((o)) (Kkk))(K,, junn) v 22h
(46)
20

---

Published in Transactions on Machine Learning Research (01/2025)
where Eq. (45) is equivalent to Eq. (17), which in turn is the Buler-Maruyani discretization of the contimuous- time SDE in Eq. (6). Equation 46 is equivalent to the particle update equation in Eq. (18).
Similar to the proof of Theorem 3 in Liutkus et al. (2019), we define two contimuous-time processes (Y1)220
and (Ud)e20 defined by the following continuous-tin SDEs:
dY,- = 8 (Y) + 2XdW,.
(47)
dU, = o. (U)dt + 22dW.,
(48)
where
(Y( = -
(0) kb
(Yur: Pank pa
(49)
k=0
(U) = (l) (Unn, (ukh)) (1))(t))))))))..
(50)
k=0
In Eq. (49), Pun follows the distribution of Xx in the discrete-time process defined by the update equation in
Eq. (45). Therefore (Y))e20 is continuous linear interpolation of the discrete-time process (XE)KEN,-
In Eq. (50), PAh follows the distribution of X in the discrete-time process defined by the update equation in Eq. (46). Therefore (U))2a is contimuuu linear interpolation of the discreto-time process (Xx)aEN,-
Let Tx. TH. and गे denote the distributions of (Xx))Eto,T) (Y)) and (Ut)tE respectively, with T= Kh. We have the following lemma bounding the total variation distance between the pairs (Tx my) and (TV, at) from Liutkus et al. (2019):
Lemma 4 (Lemmas S1 and S2 in Liutkus et al. (2019)). For all X O, assume that the continuous-time
SDE in Eq. (6) has a unique strong solution (Xt)e20 for any starting point x E Rdd For t 2 0, define 110) ( == for ( (10, 2)) d00 Suppose there exists constants A, B, L, m, O, and K (0, 1), such that the following are true for any x, E Rdd, u, E P(2),, and all t 2 0:
(x) - // (x')ll < L([ x' + It - t'l),
[| (x. 11) (00) (xd |1) L([x 2'|l || v'llT)),
(5, u/ (0) (x)) 2 m| bb
] =1(")
E (1, - u (2, []]l 2<(L2||=|| + B²).
Define:
C. = H(Ho),
1 (bb 2B² 2B2 da),
C. = C. + 2(1 m
Ci = C2 + +B²)+1 1,
C2 = 2(T2C6 + B²).
Then, we have:
LPK
h
C2nKh
| - 12v S
+ +3ddh²)
4X
3
82
Kh
- - Ptv <
||
16A
21

---

Published in Transactions on Machine Learning Research (01/2025)
Theorem 7. Under the assumptions of Lemma and for X TL2/8:
T
l - PKAlliV & x- T L'hlczh + dA) + o2k]
(51)
where er, c2 L,K 0 are constants independent of time.
Proof. We will emulate the proof of Theorem in Liutkus et al. (2019).
TTllTT
2
Tli + +11 TFl
L
h₂ + +3xdh2)
+ C2RKh
L2Kh
| -
- 2
4A
82
KL2h -1 L2K
C1h²
C2kKh
+ 32dh2)
8A
2X
42
where the second inequality follows from Lemma and the last inequality holds for À > TL2/8. The desired result then follows by plugging T = Kh and rearranging.
C Experimental Details
In this section, we explain all experimental details required running the experiments (along with the code which is provided in the supplementary material). For this project use 1 NVIDIA GPU Tesla V100 which was necessary for the pretraining of the aulo-encoder only. All neural networks are coded using Py (Paszke et al., 2019).
C.1 Datasets and Evaluation Metric
MNIST is standard dataset introduced in LeCun et al. (1998), with clear license to the best of our knowledge, composed of monochrome images of hand-written digits. Each MNIST image is single-channel, of size 28 x 28. We preprocess MNIST images by exlending them to 32 x 32 frames (padding each image with black pixels), in order to better fit as inputs and outputs of standard convolutional networks. MNIST is comprised of a training and testing dataset, but no validation set. We split the training set into two siuelly-sized public and private sets.
MNIST is similar dataset introduced by Xiao et al. (2017) under the MIT license, composed of fashion-related images. Each FashionMNIST image is single-channel, of size 28 x 28, and we also preprocess them by extending them to 32 x 32 frames. Like MNIST, FashionMNIST is comprised of a training and testing dataset, but no validation set. We split the training set into two equally-sized public and private sets.
Notice that for MNIST and FashionMNIST, we scaled the pixel values to [0,1].
CelebA is a dataset composed of celebrity pictures introduced by (Liu et al., 2015). Its license permits use for non-commercial research purposes. Each CelebA image has three color channels, and is of size 178 218. We preprocess these images by center-cropping each to square image and resizing to 64 64. CelebA is comprised of training, lesting, and validation dataset. After conducting init experiments and analysis with the validation set, we removed it. We then split the training set into two equally-sized public and private datasots.
Fréchet Inception Distance (FID) was introduced by Heusel et al. (2018). measures the generative performance of the models we consider. In our code we use the PyTorch implementation of TorchMetrics (Skafte Detlefsen et al, 2022).
C.2 Structure of the Autoencoders for Data Dimension Reduction
The experiments performed on MNIST and FashionMNIST datasets utilize an autoencoder architecture per the framework proposed by Liutkus et al. (2019). Furthermore, we normalized the latent space before
22

---

Published in Transactions on Machine Learning Research (01/2025)
adding Gaussian noise, ensuring that the encoded representations lie hyper-sphere. We use the following autoencoder structure, which is the same as that used in Liutkus et al. (2019):
Encoder. Four 2D convolution layers with (num chan out, kernel size, stride, padding) set to (3, 3, 1, 1), (32, 4, 2, 0), (32, 3, 1, 1), (32, 3, 1, 1), each one followed by ReLU activation. At the output, a linear layer is set to the desired bottleneck size, and then the outputs are normalized.
Decoder. A linear layer receives from the bottleneck features vector of dimension 8192, which is reshaped as (32, 16, 16). Then, three convolution layers are applicd, all with 32 output channels and (kernel size, stride, panning) set lo, respectively, (3, 1), (3, 1, 1), (2, 2, 0). A 2D convolution layer then applied, with the specified output number of channels set to that of the data (1 for black and white, 3 for color), and a (kernel size, stride, panning) set to (3, 1, 1). All layers are followed by a ReLU activation, and sigmoid activation is applied to the final output.
Conversely, experiments conducted on the CelebA dataset employ an autocncoder/genne /ggnoerrrrgggrrooorrr architecture based the DCGAN framework proposed by Radford el al. (2016a). The structure is the following:
Encoder. Four 2D convolution layers are employed with the following specifications: (64,3,1,1), (,,,,,4,211), (64x4,4,2,1) and (64x8,4,2,1). Each convolutional layer is followed by leaky Rectified Linear Unit (LeakyReLU) activation function. Subsequently, linear layer is applied to obtain the desired bottleneck size. The outputs are then normalized.
Decoder. Four 2D convolution layers are employed with the following specifications: (64x8,4,110), (64.4,4,2.1),, (64x2,4,2,1) and (64,4,2,1). Each convolutional layer is followed by leaky Rectified Linear Unit (Ll.))Ree.U) activation function and batch normalization is added after the activation.
C.3 Additional Algorithm
A Algorithm 2 describes the DP Sliced Wasserstein Flow without resampling of the Os.
Algorithm 2: DP Sliced Wasserstein Flow without resampling of the Os: DPSWflow.
Input: Y == ......., I Rxd ie. N ii., samples from target distribution v, number of projections No, regularization parameter X, variance a, step size h.
Output: X X= [=f " Raxd
3: // Initialise the particles
4: { ~ no. X irf....... x J E Radd
5: {0,)5a ~ Unif(S ) -= 10 .o....., TT,
6: Sample Zy E RaxNN from i.i.d. (0,
7: Compute the inverse CDF of YO + Zy.
8: // Nenations
9: for k= = O....,K K 1 do
10:
Sample Mo projections among obtain T
Sample Zx RR from i.i.d. (0),
11:
12:
Compute the CDF of XY Zx
Compute ((a) using Eq. (20)
13:
14:
I; + (((())xa)) + /22h2, ~ (O,
15: end for
C.4 Understanding Performance Differences Between DPSWflow-r and DPSWflow.
Let us recall that the main difference between Algorithm and Algorithm lies in the sampling of the projections of the sliced Wasserstein. In the former, we resample No projections at each iteration. Each sample is handled independently using the Gaussian mechanism. In the latter, we sample all No projections
23

---

Published in Transactions on Machine Learning Research (01/2025)
in advance and use them in all subsequent iterations. This approach allows for more iterations since the privacy budget is determined at initialization, but it requires a stricter privacy bound to ensure the same guarantees.
Indeed, the composition DP operations used Algorithm 1 enable the use of advanced composition techniques like the moment accountant (Abadi et al., 2016), resulting in tighter privacy bounds. In contrast, Algorithm 2 relies on basic composition, which is less efficient and yields looser bounds.
Let us give mathematical intuition by expressing E, and respectively corresponding to the privacy budget for Algorithm 1 and Algorithm 2 for the same input noise Ø. To ensure fair comparison, we use the same number of projections in both cases, Given that, in Algorithm 1, we resample I' times the projections, we will be using 7 * Ng as the number of projections for Algorithm 2 (already accounted for when we resample). As is, we will show that E, <.
DPSWflow (Algorithm 2): Let us remind that the privacy guarantee is determined by the scnsitivity of the function A' (how much single data point changes the output) and the amount of noise added (via the noise scale σ). The privacy loss, measured as e is inversely proportional to
EX
(52)
2N 1) d42
Therefore, by plugging in the sensitivity of the sliced Wasserstein gradient flow A' x N2- = T' Ne, in the case of Algorithm 2 (DPSWflow), would obtain the following:
V
with
TNe 2 x
2o (d)
(53)
ood
02
d+2 2
DPSWilow-r (Algorithm 1): However, the structure of this algorithm enables to use the moment accountant technique which gives tighter bound for the privacy budget:
E, qVTA
(54)
where q b/d, 6 being the batch size, d the size of the dataset, T is the number of iterations, and A x
No
d + 2No(d-1) 1)
We obtain the following:
d42
2T
2N - 1) d 2
2
(55)
old
Comparison: Let us compare the privacy budget. y = b/d is usually small: for relevant experiments, the dataset size tends be large and batch size small. Comparing term by term we will have, for reasonable number of iterations T:
TN < TNo and 2T < VT
(56)
Therefore, we obtain & <, which we also observe/compute our experiments.
C.5 Additional Comments on Hyperparameters and Algorithms
In this subsection we give all of the hyperparameters necessary for reproducing the experiments conducted.
MNIST and FashionMNIST.. All three DPSWflow, DPSWflow-r, and DPSWgen models are evaluated with a batch size of 250 and for 5 10-5 and the latent space (of the autoencoder, used as input of the mechanisms) has size 8. DPSWflow is evaluated over 1500 epochs for all values of e, while DPSWflow-r
24

---

Published in Transactions on Machine Learning Research (01/2025)
and DPSWgen are evaluated on 35 epochs for and E 10, and 20 epochs for = 5. DPSWftow
uses No = 31 and Mo 25, while DPSWflow-r and DPSWgen use No- = 70. As explained in Section 4.3, the privacy tracker is different for DPSWilow compared to DPSWftow-r and DPSWgen. For DPSWflow we directly input the desired value for and use the sensilivity bound from Eq. (23), along with 2 c Aa(f) (from Section 2.3), to obtain the value of a which is used in our code. For DPSWflowrr and DPSWgen we used the following pairs of 0,6; a = 0, oo; a = 0.67, = 10; and o = 0.8, = 5.
CelebA. All three DPSWNow, DPSWNow-r, and DPSWgen models are evaluated with a batch size of 250, for 10 and the latent space (of the autoencoder, used as input the mochanisms) has size 48. DPSWflow is evaluated over 2000 epochs for every value of e, while DPSWilow-r and DgSWgen are evaluated on 30 epochs for every E. DPSWHow uses No- 250 and M, 220, while DPSWHoow-r and DPSWgen use No- 300. For DPSWflow-r and DPSWgen we used the following pairs of a,e: = 0, xc; a 0.58, = 10; and 0 = 0.72, 5.
Also, notice that the architecture of the generator of DPSWgen structured as follows: the input layer consists of a linear layer that transforms the input vector to a 250-dimensional vector. The first hidden layer applics ReLU activation function to introduce non-lincarity. The second hidden layer includes another linear layer that increases the dimensionality from 256 to 512, followed by Batch Normalization layer to stabilize and accelerate the training process, and ReLU activation function. The third hidden layer contains linear layer that reduces the dimensionality from 512 to 256, followed by ReLU activation function. The output layer comprises a final linear layer that maps the 256-dimensional vector to the desired output vector. This sequential arrangement of layers allows for progressive transformation and refinement of the input data, leveraging non-linear activation functions and normalization techniques to improve the model's learning capacity and stability.
25


Reviews:
[{"review_id": "MIbvIfCegk", "reviewer": "Reviewer_yUc6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper propose an alternative DP generative modeling via gradient flow\n- Theoretical analysis and empirical results are provided\n- The paper is clear and easy to follow\n\nWeaknesses\n- What are the practical values of L, c1, c2 in Theorem 2? If these values are large, the penalty parameter \\lambda will also be very large, making it impractical.\n- The parameter \\delta is used with different meanings in Sections 3.2 and 3.3.\n- What is the data dimension d used in the experiments? While an autoencoder can reduce d for the provided image datasets, it may negatively impact performance on other types of data. It would be helpful to evaluate and illustrate performance with a larger d.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an alternative generative modeling approach based on gradient flow, which is defined by Gaussian-smoothed Sliced Wasserstein Distance, including the associated SDE. The authors define a numerical scheme for solving this SDE, and analyze the differential privacy guarantee of the proposed gradient flow. Experiments are conducted the demonstrate the results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7LWFZSY3Ef", "reviewer": "Reviewer_o1Ec", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Self-claim: I know DP generator very well, but I do not know too much about the gradient flow/SDE part (so please refer to other reviewers for this part).\n\n## Strength\n1. Training DP generator via gradient flow is new, as far as I know.\n2. This paper includes a complete theoretical analysis and experiments on toy and real datasets. Algorithm1, figures, and tables are clear to follow.\n\n## Weakness\n1. Motivation: \n    + I do not find sufficient justification for choosing gradient flow as the generative modeling method, other than \"it is under-explored\". What is the benefit of this method? \n    + Is there any other motivation for choosing Gaussian-smoothed SWD (I mean other than DP)?\n3. Writing of gradient flow/SDE: Probably because I have limited knowledge in this field, I still do not get the connection between gradient flow and SDE. For example,\n    + \"Depending on the form of $F_\\lambda$, this continuity can be associated with an SDE...\". I cannot get how they are associated.\n    + What is the connection between eq. 11, 13 and 17. Is eq. 17 an existing result? If so you need to add reference.\n4. Discussion with diffusion models: diffusion models are also strongly connected with SDE, e.g. reversing the backward diffusion process is to find a solution to an SDE. What is the difference between gradient flow and diffusion?\n2. Baselines: the experimental comparison is not complete. The only baseline is DPSWD, which I initially thought was similar to the proposed method. However, they are \"fundamentally distinct\". If so, why other distinct methods are not compared? For example, you can find a lot more related works in the numerical comparison table in [Dockhorn et al, 2023].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores an alternative approach for training a DP generator other than DP-SGD, i.e. a gradient flow of sliced Wasserstain Distance (SWD). Particularly, a smoothed version of SWD is automatically a Gaussian mechanism with DP guarantees, so we can use it to train a DP generator. A formal DP guarantee is analyzed for each update. Experiments are conducted on both toy dataset and real image datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FBTlf7Z0xl", "reviewer": "Reviewer_YyB2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors study an interesting problem and their solution is also very interesting. In particular, considering the minimization problem on the space of probability measures, this is less understood compared to DP-SGD. I think that the theoretical guarantees proved in this paper might lead to better performance than the generator-based model. \n\nThe proof in the paper also has some novel attributes. For example, they show the existence and regularity of solutions to the functional minimization problem which I am not sure exists in the literature.\n\nI vote for acceptance. I however have a few clarifying questions.\n\nI am trying to understand the proof of the regularity of the minimizer. Can the authors give more details as its relation to previous works or is the proof entirely new? If it is completely new, I feel that the authors undersell the importance of this result. \n\nDefine strong solution to the SDE before stating Theorem 2. If it was defined earlier, I have missed it. Mention somewhere that the assumptions of strong solution is standard in SDE literature. \n\nWhy do you get a tighter bound for Algorithm 1? Is there an intuitive reason for it?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies DP generative modeling in the space of probability measures. They define the GF of the Gaussian-smoothed sliced Wasserstein distance and the natural stochastic process involved with it. They use natural discretization based on Euler's method to solve the continuous-time SDE. The DP guarantee of the corresponding GF is more straightforward after this.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above and the pointed questions.", "broader_impact_concerns": "The paper has some interesting new directions, and I believe it definitely deserves acceptance.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eOoKNuydTe", "reviewer": "Reviewer_q2Vr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is written very clearly and from the beginning, it is clear what the contributions of this paper are.\n- The theoretical results are strong and the contribution is novel, as the authors say it has not been done before\n- The experiments clearly highlight that their method is practical and viable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel differentially private generative modeling approach based on gradient flow. The main contributions are the following:\n- The authors define the gradient flow of the Gaussian-smoothed Wasserstein Distance and the associated SDE\n- The authors show a link between smoothing and differential privacy based on a Gaussian mechanism\n- In particular, the authors present two algorithms that implement their gradient flow\n- The also analyze the differential privacy guarantee of their proposed particle scheme\n- They perform experiments to show/validate their results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the reviewers are not super enthused by the technical novelty of the paper, but overall they all agree that the paper's claims are validated by strong evidence.
Decision: Accept (poster)