Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training
Authors: Kristjan Greenewald
Yuancheng Yu
Hao Wang
Kai Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: differential privacy
synthetic data generation
tabular data
GAN
f divergence
slicing
Primary area: privacy
TL;DR: We enable private training of GANs using random slicing, avoiding the shortcomings and challenges of DPSGD.

Abstract: Training generative models with differential privacy (DP)  typically involves injecting noise into gradient updates or adapting the discriminator's training procedure. As a result, such approaches often struggle with hyper-parameter tuning and convergence. 
We consider the \emph{slicing privacy mechanism} that injects noise into random low-dimensional projections of the private data, and provide strong privacy guarantees for it. These noisy projections are used for training generative models.
To enable optimizing generative models using this DP approach, we introduce the \emph{smoothed-sliced $f$-divergence} and show it enjoys statistical consistency.  
Moreover, we present a kernel-based estimator for this divergence, circumventing the need for adversarial training. 
Extensive numerical experiments demonstrate that our approach can generate synthetic data of higher quality compared with baselines. Beyond performance improvement, our method, by sidestepping the need for noisy gradients, offers data scientists the flexibility to adjust generator architecture and hyper-parameters, run the optimization over any number of epochs, and even restart the optimization process---all without incurring additional privacy costs.

Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training
Kristjan Greenewald MIT-IBM Watson AI Lab, IBM Research kristjan. g.w.neweeeee111111 bm. com
Yuancheng Yu UIUC yyu51@1llinois.cm edu
Hao Wang
Kai Xu
MIT-IBM Watson AI Lab, IBM Research hao@ibm.. com
MIT-IBM Watson AI Lab, IBM Research xuk0ibm. com
Abstract
Training generative models with differential privacy (DP) typically involves inject- ing noise into gradient updates or adapting the discriminator's ' training procedure. tuning and conver-
As a result, such approaches often struggle with gence. We consider the slicing privacy mechanism that injects noise into random low-dimensional projections the private data, and provide strong privacy guar- antees for it. These noisy projections are used for training generative models. To enable optimizing generative models using this DP approach, we introduce the smoothed-sliced f-divergence and show it enjoys statistical consistency. Moreover, we present a kernel-based estimator for this divergence, circumventing the need for adversarial training. Extensive numerical experiments demonstrate that our approach can generate synthetic data of higher quality compared with baselines. Beyond performance improvement, method, by sidestepping the need for noisy gradients, offers data scientists the ficxibility to adjust generator architecture and hyper-panne ppeer-prrmmters, the optimization any number of epochs, and even restart the optimization process- all without incurring additional privacy costs.
Introduction
Just as oil fueled the industrial revolution, data now propels innovation and progress today's digital life. However, data sharing faces challenges due privacy risks and regulatory policies (c.g., GDPR, CCPA, FTC). Synthetic data offers a promising solution as closely resembles real data, retains its formats and essential properties, and can be seamlessly integrated into existing workflows. Nonetheless, modern generative models are vulnerable to privacy attacks that could expose individuals' sensitive information in the original data [HMDDC17, CYZF20, SOT22]. Today, differential privacy (DP) [DR14] stands as the de facto standard for privacy protection and it plays crucial role in guiding the design of synthetic data. For example, DP was used in the release of microdata from Israel's National Registry of live births in 2014 4 to protect the privacy of mothers and newborns [HC24), well as in the release of global perci-pperpttatorr data ensure the privacy of victims [fM22]. Additionally, a recent publication [NDL++23] by the National Institute of Standards and Technology (NIST) recommends the use of DP algorithms in generating synthetic data to provide robust privacy protection against rapid developments in privacy attacks.
Existing approaches to training generative models often integrate DP by injecting noise into gradient updates or adapting the discriminator's 's training procedure [XLW+18, TKP19, CBV+21, TTCY21, BJWW+19, JY VDS1, TWB+ 19, BKZ23, NPA23, LJW+20, +2 COF20]. They provide several benefits, such as the ability to generate diverse data types (including continuous data, time-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

series, and images), scalability to high-ding sish--imensional data, and accelerated runtime using GPUs. However, fine-tuning hyper--aramenn parameters within these frameworks can be challenging [NWD20, PS22, I.T19]. Additionally, they encounter dilemma when determining the number of training epochs: with a fixed privacy budget, increasing the number of epochs requires injecting more noise into gradient update per iteration, while fewer epochs might not be sufficient for the optimizer to converge. In such setting, if the training does not converge, the only recourse may be to increase the privacy budget. This motivates fundamental question:
How can we train generative models with DP guarantees while ensuring easy fine-tuning, stable convergence, and high utility?
In this paper, we introduce a new learning paradigm for training privacy- privacy- preserving generative models. Our approach decouples the training process into two steps: (i) computing noisy -iw-dmmesiional projections of the private data along random directions, and (ii) updating the generative models to fit these noisy projections. We establish DP guarantees for the first step and leverage the random projection to further tighten our privacy bound. This decoupling strategy offers several advantages. The post-process post-proeessing property of DP ensures that any deep learning techniques can be applied in the second step. In other words, our approach is model-agnostic, allowing for smooth integration into existing training pipelines of generative models. With our method, data scientists have the flexibility to adjust generator architecture and hyper-parameterd optimize the generative model for any number of epochs, and even restart the optimization all without worrying about additional privacy costs.
Based on this paradigm, we are motivated to introduce new information -teoorethcc measure: the smoothed-slich f f-divergence. This divergence (randomly) projects the original and synthetic data distributions onto lower-dimensional spaces, followed by smoothing with isotropic Gaussian noise, and averaging their f-divergence over all projections. We prove that using this divergence as the loss function in generative model training is equivalent to the aforementioned two-step training process. Additionally, we present kernel-based, differentiable estimator for this divergence. It circumvents the need for adversarial training in generative models, thereby enhancing convergence stability and robustness to different choices of spper-parameters. Finally, we establish the statistical consistency of training generative models using this divergence.
In terms of the slicing mechanism, we build upon the work of [RL21]. They introduced the smoothed- sliced Wasserstein distance and applied it to generative models and domain adaptation tasks. However, their approach limited to 1-dimensional projection spaces (A- 1), exploiting the closed-form expression of Wasserstein distances in ID. Moreover, their privacy analysis contains a significant flaw in its derivation (see Remark for detailed discussions). contrast, we propose a generic framework for training privac--rresevving generative models, applicable to any k-dimensional projections. Empirically, we observe that setting the projection dimension to a small number (e.g., k 2, 3) significantly improves the quality of synthetic data compared with A = 1 under the same privacy budget. Additionally, our kernel-based estimator eliminates the need for adversarial training in generative models, facilitating stable convergence. Finally, we present completely new proof for establishing the DP guarantees of our framework. This analysis also applics to any smooth- sliced divergence objective, for instance, we provide corrected (and improved!) DP bounds for the smoothed-sliced Wasserstein framework of [RI.21].
We demonstrate the effectiveness of our method through numerical experiments. We compare generative models trained using our method with those trained by standard privacy mechanisms (DP-SGD, PATE, or smoothed-sliced Wasserstein distance) among various real-world datasets. The results indicate that our approach consistently produces synthetic data of higher quality compared with baseline methods.
In summary, our main contributions are:
We introduce new framework for training privacy-preserving generative models. It offers easy hyper-parameter tuning and allows for optimizing generative models over any number of training epochs without extra privacy costs.
We propose new information-theoreti divergence and provide kernel-based estimator. This estimator enables the training of generative models without relying on adversarial training, enhancing convergence stability.
2

---

For the slicing mechanism, we extend the work of [RL21] and allow for projecting data onto any k-dimensional space. Using an entirely new proof technique, we provide DP guarantees.
which in the &- = setting correct (and strengthen!) the privacy analysis in [RI.21].
We validate our method through numerical experiments. The results show that our method produces synthetic data of higher quality compared with baselines.
Additional Related Work
Recent work has proposed alternative approaches for training DP generative models without adversar- ial networks. For example, [CBV+21] considered using the Sinkhorn divergence as the loss function, but their method adds noise gradient updates to ensure DP, which leads the challenges discussed in the introduction. Another line of work [HAP21, 22, HISP23, YAS 23] used the maximum mean discrepancy (MMD) to train DP generative models. They inject noise into the embedding of the private data distribution to maintain DP. However, minimizing their loss function to zero does not guarantee perfect matching between the synthetic and real data distributions, due either not using a characteristic kernel or the approximation errors stemming from using umite-dimensione feature map- pings to approximate the kernel function. In contrast, Proposition 1 proves that our smoothed -liced f f-divergence equals zero iff the synthetic and real distributions are identical; Corollary I establishes the statistical consistency of training generative models using our loss function. Additionally, we amplify our privacy bound by leveraging random projections in Theorem 1.
There significant research introducing DP mechanisms tailored to generating tabular synthetic data [e.g.. ZCP+ 17, MSM19, MMS21, MMSM22, GAH+ 14, VAA+22, DAHY24J. They select set of workload queries (e.g., low-order marginal queries) and generates synthetic data to minimize approximation errors on these queries. These methods often maintain statistical properties of the original data with high accuracy, particularly for the selected workload queries; downstream predictive models trained on such synthetic data often achieve high performance when deployed on real data TTMH*21, WSH 23]. However, they only apply to categorical features rely on special generative model architectures, or struggle to scale effectively to high-dimensional data. More broadly, there are several works analyzing DP synthetic data from theoretical perspective or investigating other properties of synthetic data (e.g., missing values) [NTZ13, UV20, GMHI20, CLWX21, BSV22b, BSV22a, RTMT21, MZKH23].
ABK ZWL LVW21,
A burgeoning line of work has explored slicing and smoothing to improve sample complexity in estimating divergence and optimal transport measures. These measures often suffer from extreme curses of dimensionality (e.g., n --//d for Wasserstein distance [Dud69])- Previous studies have shown that both slicing @RPDB11, VFT+ 19, 2,C**1,NDC*+2 GG21) and smoothing [GGNWP20, GG20] facilitate convergence at the parametric rate for both f-divergences and optimal transport distances, while preserving key properties of the original divergence (c.g., being zero iff the two distributions are identical). These modified divergences are also used as objective functions for training generative models. In our context, the use of smoothed-sliced divergence is less motivated by sample complexity (indeed using both smoothing and slicing would unnecessary for achieving the parametric rate). Instead, it is motivated by the slicing privacy mechanism. Also, while we enjoy the sample complexity benefits of slicing, we do not benefit from the sample complexity improvements of smoothing [GGK20]. This is because achieving DP requires injecting a finite number (typically just one) of noise realizations per data point. Full smoothing results are still useful, however, for our consistency results in the asymptotic regime.
While the I-dimensional sliced Wasserstein distance can be computed via a simple sorting algorithm [DZS18], there is in general no closed-form sample-based estimator for f-divergence, even in one dimension. While nearest neighbor [WKV09] and kernel-based [MSGHI18] estimators do exist, they often suffer from scalability issues and are not friendly to gradient- based optimization. Hence, it is common practice to use dual forms of f --divergence in deriving adversarial-bassee training procedures for generative models [NCT16]. In the present work, we avoid this costly adversarial training using our moment matching estimator.
The supplementary material of this paper includes: (i) omitted proofs of all theoretical results and (ii) supporting experimental results.
'There are a few exceptions [e.g.. LTVW23, VAA+22] but they only generate pre-determined number of synthetic samples, rather than providing generative model, as ours does.
3

---

2
Preliminaries and Problem Setup
In this section, review the concepts of differential privacy (DP), f-divergence, and moment matching method used for estimating faaiaaiinnndvveeeee f-divergence.
2.1 Differential Privacy
We denote the original dataset as a matrix X E Rnxd, where n denotes the number of records and d' represents the number of real-valued features per record.
Definition (Dataset adjacency). Two datasets X and X' are considered adjacent if they differ in single row, say the i-th row, such that [X,,: x,,lla where X,; and X are the i-th row of X and X', respectively.
Next, we recall the definition of differential privacy (DP) [DR14].
Definition 2 ((c, 8) differential privacy). A randomized mechanism M Rexd 4 0 satisfies (c, difffffferet privacy if for any adjacent datasets X, X' and all possible outcomes S S O, we have:
Pr(M(X) E 3) S exp(e) Pr(M(X) e 8) 6
DP has many compelling properties. The post-processing property states that if mechanism M is (e, 6D, its outcome remains (e, 8)DD even after applying (potentially randomized) function; the basic composition rule states that given sequence of mechanisms M1.... if M, is (e,, DDP,
then their composition M(D) (M1(D), ..... Mp())) will satisfy (E 6, Ek= D,P.DP
2.2 f-divergence and Moment Matching
We first recall the definition of f-divergence [Chapter 7 in PW23].
Definition Let / (0, x))
be convex function with f(1) 0 and f(0) - f(0+). Let Px and Qx be two probability distributions on X. If P < Q, then the f-divergence is defined as (PII Eg
where aQ a Radon-Nikodyn derivative. Additionally, we denote
3Q
density ratio by r(z)
(z).
f-divergences have many nice properties. For example, it is always non- non-negative; and assuming f is strictly convex then (y( = 0 if and only P= Q.
There is burgeoning field of research focusing on f-divergence estimation [see e.g., GGNWP20, WY20, SXGS20J. Here we revisit a framework based on kernel mean matching, special instance of moment matching methods [Chapter SSK12J. Given a reproducing kernel K(æ, x'), one can solve the following optimization problem to estimate the density-ratio function:
/ [(zz,))ar )dP(x) - f K(x,))) (r())d)()))))))))))
min FE
R
where || | denotes the norm of reproducing kernel Hilbert space R. One example of reproducing (- -- Given {< drawn from P and 2n
kernels is the Gaussian kernel K(z,x') exp
{= drawn from Q, we can optimize empirical version of the above optimization to obtain an analytical solution:
12 K K Kgpplay
A
n2p
9.9
where fg E R. is the (empirically) optimal density ratio values at samples drawn from Q, K R Xn and g. IR" up the kernel Gram matrices:
[fgb] f (x?). [Kggl,s K (mg..²)) K.ppyy, = (z3,,=²)
We extend the definition of f to vector by applying it to each element of the vector. Then we have
(P(())
1 IT n.
f(
1 _ nh
ng P 'K Kg 1n,
(1)
nh
np
9.9

---

3 Main Results
We introduce a new information cooomation--hhoreticc measure- the smoothed-sliced f-divergence. We show that this divergence measure the difference between distributions, with only access noised k-dimensional slices of the distributions. This finding motivates new DP mechanism: the k-slicing privacy mechanism. Finally, using the non-adversarial estimator (I), apply the smoothed-sliced |-divergence as new loss function to train privacy-preserne generative models.
3.1 Smoothed-sliced f-divergence
We start with giving a formal definition of the smoothed-sliced f /-divergence.
Definition 4. Denote the Stiefel manifold of d k matrices with orthonormal columns by Sy (RR). Let ~ (S (R²)) and N -N(00, o2I)). The smoothed f-divergence between distributions Px and Qx on R2 defined as
SD f.k,0 (xx] & Dy DI(PorrsHNell@orxxiell)))
vol((x(R / (RM) Dy Dj(Prxxnll@oxxxxide
Next, we discuss some basic properties of this new divergence.
Proposition The smoothed sliced f f-divergence is non-negative: J. (PxllQx) 2 for any k 2 and a 2 O. If f is strictly convex at and Px, Qx have moment generating functions, then SD (xx xxx)
if and only Px = Qx
Given a set of real data {a (i.e., rows of X) and synthetic data {-="")>>>> Ji=1: let Dj denote any estimator of k-dimensional f-divergence (e.g., (1)). We draw random directions 0, and additive noise Va and Ua.. Then can estimate the smoothed-sliced f-divergence by
SD o (Pxm Pxx
Σ Dy ((=;" + U. 0 (x 1
As SD f. is an empirical average of m estimates of k-dimensional divergences, this estimator will inherit any statistical convergence bounds that apply to the chosen Dj-
3.2 Slicing Privacy Mechanism
The loss function in (2) accesses the original data solely through their noisy projections along random directions. This observation motivates the following k-slicing privacy mechanism.
Definition Let &s denote the dimension of the random projections, and m denote the number of them, yielding m² = mk. Let X E =xd represent the original dataset where we assume |X,,; l|2 S 1 for all i. Let U ddxm' and V E Raxm' be random slicing and noise matrices with each element independently drawn from U2, ~ ~N(0 d and V1,5 ~ (O0, o?), respectively. The slicing privacy mechanism is defined
M(X) & (U,XU+
Our mechanism outputs the random slicing matrix U, as will be used to project the synthetic data onto the same spaces during the training of generative models. In contrast, [RL21] did not include in their privacy mechanism, leading them to give an incorrect derivation of the privacy guarantee. Similarly, approach differs from using the LiinsonnLindenstrauuss transform to preserve DP [BBDS12, EKKL20], as these methods do not reveal the random matrix.
Remark 1.
In Definition 5, we draw random projections from a Gaussian distribution instead uniformly from the Stiefel manifold. This choice simplifies the sampling procedure and streamlines our privacy analysis by making the the mechanism output jointly Gaussian with zero mean, conditioned on the data. Next, establish the DP guarantees for our noisy slicing mechanism.
5

---

Theorem 1. Assume - 2a2 - a) < d. For all ò E (0,1) and a > 1, the mechanism M(X) (U,XU + V) in Definition 5 satisfies
m'a
In(1/6) DP.
(d
where (e, (), is defined in Definition with dataset adjacency specified in Definition 1.
Our privacy bound relies on k (the dimension of projection spaces) and m (the number of slices) only through their product m' mk, which reveals trade-off: with fixed privacy budget, increasing aids generative models in capturing higher-order information of the original data, while increasing m improves the accuracy of learning lower-order information. This trade-off resembles marginal-basse -based mechanisms, where adjusting k to enable generative models to learn the k-way marginal queries [Definition in LVW21] involves similar effect. However, we remark that even one-dimensional slices suffice to fully characterize a distribution since S... ( Pxm)) Proposition In contrast, matching all w-way marginal distributions for & < d does not necessarily ensure Px Px
implies Px Px" by
We prove in Appendix Proposition 3 that the slicing privacy mechanism is ma 1n(1//), -DP if the projection matrix U is deterministic. Comparing it with Theorem I, we observe that by randomly selecting the projection matrix, we can achieve a tighter privacy bound by reducing factor of d-y (for the first-term), even U is disclosed by the privacy mechanism. The rationale behind this lies in the fact that deterministic projection matrix allows the model designer/adversary target specific individual records through carefully designed projection directions.
Remark (Choosing (t). The bound in Theorem 1 can be optimized terms a for a desired fixed and this optimization can be done numerically. Alternatively, there an ad hoc strategy for "approximately optimizing' a. Let us assume y S d/2, i.e. a² - I do² /2. Then
S m'a In(1/5)
odd
Minimizing the right hand side expression w.r.t. a > 1 yields an optimal
(2d1((1
a 1==1+
ml
Substituting it into the (true) expression for yields
m'
d
m/ ln(1/8)
m'
m' ln(1/))
1+
+2
20 (d - 7²))
2(d 7)
a2d
2dd
o²d
where 7" = 2a a).
3.3 Training Generative Models
We outline our approach for training
generative models using the smoothed-slice fdiivergence (see Algorithm for more details). First, we transform the output of the slicing privacy
mechanism M(X) [X) = (U, XU + V) into O = ((O., os.s) [ess)))mem Here 0. U (.( E Rdxk is the random slicing directions and o,,, = (XU + () E IR is the (noisy) projection of the i-th data point onto the s-th slice.
Let G. be a generative model with trainable parameters w E Ω. At each iteration, we sample mini-batch from O as (0.,, o,,s) } (e]m) where b is the batch size.
To produce synthetic data, we draw random noise zj for j E [b] and feed them into the generative model 2;," (2,). Additionally, we draw random noise 0j, ~ 00,
o²I,) ) for j E [b] and [m] that will be added to the projected synthetic data.2 Given these samples, we train the generative model G. to generate samples close the real data, using an estimator for the sliced
This noise can be resampled every epoch as it does not touch the real data.

---

f-divergence to measure the discrepancy between the noisy-sliced real data o.,, and the noisy-sliced synthetic data 0, + yyss- The smoothed slice f-divergence (2) yields the following loss function:
L(w) = 1 --=->,,,-- Dy (1x) + +
#=1
The following corollary is direct application of Proposition
Corollary (Consistency). Consider dataset X Rxx of n. i.i.d. samples from d-dimensional distribution Px whose moment generating function exists, and the slicing privacy mechanism M(X) as in Definition 5. Suppose the noise level is fixed while - co and suppose that the chosen f f-divergence estimator D is consistent and f is strictly convex Then limm,,, L(w*) = for some w" E O if and only Gw Px.
ou. Additionally,
Inserting our kernel-based estimator (1) yields for kernel
I(w) = mb #=1
Σ 1 ((KK, + TI)) - 'K (k,wll)))) 1))
(5)
where we compute the kernel Gram matrices K,, Kew E Rbxb along each slice:
KKll..- = K (o.,,; s), KKawkis = K (o 2 0, 0g,0), for E [m]
(6)
To ensure the stability of computing matrix inversion, we include 7I, where > 0 is small constant. Given that the domain of I is [0, 0o), we clip the density ratio estimator to make non-negative.
Note that to compute the kernel Gram matrices K, and K we need to apply the Gaussian kernel, which requires constant Tg. A heuristic of choosing this 0g is make it the median distance between all samples-note that this will lead to different kernels along different slices. In practice, we use an ensemble of og and average their density ratio estimators.
Remark 3.
Algorithm 1 Training DP generative modes with the smoothed-sliee f f-divergence.
Input: training data X = [a: Jim slicing dimension k; number of slices m; batch size; max number of iterations 7'; learning rate y, privacy budget e, 6.
Apply the noisy slicing mechanism M(X) with (c,8)-DP, finding noise variance oee,) and transforming the output into {(.,, or.,) )etm))-et
for t= 1. T do
Sample mini-batch of data from [(O,, (x()),))())) choosing hatch subset of the i. Generate synthetic data X.yn by feeding random noise into Gw.
Slice noise the synthetic data as (syu)) 0, X.gn + (c.s) (0, I), redrawing noise each time (no privacy cost).
Compute the kernel Gram matrices (6) under Gaussian kernels in each slice.
Compute the loss function (ww) in (5). Run stochastic toradiet optimization w +- w vVV.)(
Output: generative model Gu-
4 Numerical Experiments
We validate our approach and compare it with baselines through numerical experiments. Additional experimental results and details of our setup are reported in Appendix C.
4.1 Synthetic Tabular Data
We compare Algorithm 1 with four DP mechanisms: DP-SGD [XLW*18, 18, 2LP+20], PATE [JYVDS19], MERF [HAP21] and SliceWass [RL21]. Like our algorithm, these baselines can
Baselines.
PPiivccy guarantees will be vacuous in this limit but this can be mitigated by privacy amplification (see Lemma in appendix). Specifically, can subsample n data from larger dataset of size n' and let n' /nn
7

---

Single Attribute Similarity
Pairwise Attribute Similarity
Classifier Fl Score
Dataset
DP Mcchanism
KSComp
TVComp
ContSim
CorrSin
LogitRegreasse
Income
Algorithm
=0.05
0.70 =0.01
0.35 +0.03
0.96 #0.04
0.31 #0.30
SliceVass
0.24 #0.05
0.61 +0 03
0.27 +0.02
0.93 #0.07
0.13 #0.14
DP-SGD
0.29 #0.06
0.38 #0.02
0.10 #0.01
0.75 #0.07
0.00 #0.00
PATE
0.15 #0.03
0.40 #0.06
0.13 +0.03
0.90 +0.07
0.16 +0.18
KERF
0.81 L0.01
0.51 10.04
0.10 10.02
0.56 10.00
0.28 10.001
Coverage
Algorithm
0.74 =0007
0.87 1:002
0.63 +0002
0.91
0.41
SliceVass
0.72 #0006
0.85 ::0 01
0.60 2:001
0.91 20:05
0.41 #:002
DP-SGD
0.44 1:0.0
0.63 110000
0.35 1..10
0.73 10:15
0.24 1.0.8
PATE
0.35 =0.05
0.51
0.26 +0.02
0.85 +0.009
0.32 #0.22
KERF
0.36 =0. 18
0.52 00.04
0.18 +0.03
0.69 10.15
0.29 #0.19
Nobility
Algorithm
0.68 =0.06
0.85 #0001
0.50 +0.02
0.86 #0.01
0.69
SliceVass
0.74 =0.04
0.84 +0002
0.50 +0.02
0.86 +0.08
0.67 +0.06
DP-SGD
0.52 1..00
0.70 .10.06
0.34 10.06
0.74 10.13
0.67 10.19
PATE
0.08 20.03
0.54 1:000
0.23 10.02
0.83 40.02
0,41 4:042
KERF
0.34 2:005
0.59 =: 01
0.21 20.01
0.68 20.04
0.59 1:034
Employment
Algorithm
=0.08
#0001
0.64 +0002
0.47 #0.07
SliceVass
0.74 =0.04
0.83 +0.01
0.62 +:000
0.50 #0.10
=0.04
20.05
+0.05
#0.31
PATE
0.39 #0.32
0.54 #0.04
0.30 #0.04
0.38 #0.28
KERF
0.96 =0.01
0.67 #0.08
0.34 #0.02
0.67 #0.08
TravelTine
Algorithm
0.45
0.75 10.01
0.45 10.01
0.87 10:05
0,42
SliceWass
0,33 =0.01
0.62 =0 02.002
0.33 +0.01
0.84 #0.05
0.42 #0.14
DP-SCD
0.37 2:008
::0
0.22 10.03
0.94 10:07
0.37 10.33
PATE
0.37 10.04
0.40 10.
0.15 1.000
0.91 10.02
0.34 10.20
MERF
0.61 =0.04
0.37
#0.02
0.07 #0.00
0.62 #00022
0.14 #0.28
Table 1: We compare synthetic tabular data generated by our Algorithm with baselines, all under the same privacy budget e 5. We demonstrate them on the US Census data derived from the American Community Survey (ACS) with various evaluation metrics. These metrics range from 0 to 1, with higher scores indicating better performance. Since Employment has only one numerical column and CorrSim requires at least two numerical columns, we skip its values. The highest scores are highlighted in bold. If two methods have the same average score, only the one with lower standard deviation is highlighted. We remark that KSComp and CorrSim are less significant since they are designed for numerical columns. However, the benchmark datasets contain limited number of numerical columns, with the majority being categorical (see Table 2).
handle both numerical and categorical data types without requiring data discretization and return a generative model that can produce any number of synthetic data. The implementations of the first two are adapted from an open-source Python library [Sma23], and the MERF implementation is from https:// / /gi thub. http/PaaaaabLLDDP/EEEE mechanism with their main algorithm, which fixes the flaw in their privacy analysis and improves their privacy guarantees (and therefore quality results). Additionally, since their public Github repo does not include their synthetic data code, we implement their algorithm ourselves.
For SliceWass,
combine our slicing privacy
Data. We validate both our method and baselines using the US Census data derived from the American Community Survey (ACS) Public Use Microdata Sample (PUMS). Using the API of the Folktables package [DHMS21], we access the 2018 California data. Additionally, Folktables package provides five prediction tasks (Income, Coverage, Mobility, Employment, TravelTime) based

---

on a target column and set of mixed-type features. Details about these data, including the number of records and columns, are provided in Table 2 in the appendix.
Evaluation metries. We follow the evaluation principles in [TMH 21] for assessing the quality of synthetic data and leverage the APIs from an open-source library in our implementation [Dat23].
KSCompl
ement ement (numerical columns) and TVComplement (categorical columns). They measure the (average) similarity of one-way marginals (i.e., histograms of individual columns) between real and synthetic data.
Conting seatiinnnnyimmiii iariitittt...... It measures the (average) similarity of pairs of categorical columns between real and synthetic data.
It measures the (average) correlations among numerical column pairs and computes the similarity between real and synthetic data.
Correlation watatnnniimlaiiiiii...teeiiiiiii
BinaryLogist ceetiiieegeessiooi It measures the downstream classifier's F-1 score when trained on synthetic and test on real data.
Note that the benchmark datasets contain limited number of numerical columns, with the ma- jority being categorical (see Table 2). Hence, the applicability of Correlati wwtpllaainnimmmaaaaa.a and KSComplement constrained they are tailored for numerical columns.
Main results and observations. We present the experimental results for e = 5.1 in Table I and show results for = 8.1 in Appendix C. As shown, the two methods using the slicing privacy mechanism (Algorithm and SliceWass) consistently outperform the other baselines. MERF has better numbers for three instances of KSComp and one of Logi tRegression, but is signficantly worse in other instances and for the remaining metrics. For both Algorithm and SliceWass we used the same hyper-parameter initialization and neural network architecture across all datasets. We believe that with more extensive hyperparameter tuning, which incurs no extra privacy cost as previously discussed, their performance could be even further improved.
Algorithm 1 exhibits a more favorable performance compared with SliceWass in most settings. The rationale behind this observation lies in the fact that SliceWass is limited to 1-dimensional slices (k: 1), relying on the closed-form expression of Wasserstein distances in 1D. In contrast, Algorithm 1 is applicable higher-dimensional slices. This capability enables the generative models to capture higher-order statistical information, leading to better performance in the pairwise ContingencySind larity statistic and higher robustness in the SetryLoooissiiRegressst downstream task.
4.2 Synthetic Image Data
We conduct experiment to generate DP synthetic im- age data using the MNIST dataset [LBBH98]. We train separate generative models for each of the 10 classes in MNIST, with 10% of the data randomly sampled for each class. We evaluate the quality of the synthetic images by measuring the downstream accuracy of a classifier trained on the synthetic data and deployed the real MNIST test set. We compare Algorithm and SliceWass (combined with slicing privacy mechanism), with MERF, which is a state-of-the-art MMD-based method for generating synthetic images [HAP21]. We observe that MERF out- performs the two
64-
64
62-
Alputtmm
VIAF
We compare accuracy for downstream classification tion of privacy budget (e) for synthetic MNIST data created by MERF with and SliceWass. Note
Figure 1:
func-
algorithms at lower privacy budgets but underperforms higher budgets. We hypothesize that MERF achieves superior performance in the low-budget regime since privatizes only the mean embedding of the data for each class; its generative model designed to recover this mean rather than the full data distribution. As using only the mean involves very little privacy budget, they are able to add less noise and outper- form in the low-budger regime. Our approach, which models the full data distribution, does not
A Algorithm that the two biccing-echhansm-e approaches outperform MERF for higher privacy budgets.

---

benefit from these savings. However, as the privacy budget increases, our method better captures the true data distribution, leading improved results over MERF.
4.3 Domain Adaptation
0.44
Following the experiments in [RL21], for completeness we also apply our smoothed-sliced f-divergence in do- main adaptation tasks. We benchmark our approach with SliceWass [RL21] using the implementation available on their Github repo, modified to use our DP bound. This experiment aims to privately train a classifier using labeled data from source domain and unlabelled data from tar- get domain. We consider the target and source domains being MNIST and USPS, as well as the reverse. Results for varying e are depicted Figure 2. As shown, our Algorithm using 100 slices improves SliceWass.
0.90
USFS
ShoeWees USPS
Figure 2: Unsupervised domain adapta- tion between from MNIST to USPS and vice versa.
5 Final Remarks and Limitations
In this paper, we consider the slicing mechanism for training privacy-pree preeeerirn generative models and derive strong privacy guarantees for it. This mechanism motivates divergence to yield a smoothed-sliced f-divergence that can be estimated with a kernel-based density ratio estimator. Our approach circumvents the need for injecting noise into gradient updates and avoids adversarial training for optimizing generative models. It provides flexibility in selecting neural architectures and tuning sypee-parameters without incurring additional privacy costs. Through experiments on synthetic data generation, we validate our approach and compare it with existing baselines. Our findings suggest that the slicing privacy mechanism powerful tool for training generative models to create private synthetic data, and the smoothed-sliced f-divergence provides a promising avenue for advancing the field of dimensional datasets. Additionally, we hope our effort can be of particular interest to the information theory community and open up a new application frontier for classical information -ooomatinn-theortiicc tools.
combine it with f-
data synthesis in sensitive, high-
There are several promising avenues worth exploring. The f-divergence has many nice properties, including (strong) data processing inequalities, inequalities among different f-divergences, and variational representation. It would be interesting to investigate whether similar properties extend to the smoothed-sliced f-divergence S..2. Additionally, deriving cample-complexity bounds for estimating SD fA.o² from finite samples would offer valuable insights. Finally, the use of synthetic data presents both opportunities and challenges. On the one hand, synthetic data is well-suited for various tasks like early model development, educational demonstrations, simulation, and testing. On the other, synthetic data can never fully replicate all aspects of the original data, and adding DP guarantees introduces an additional layer of complexity requiring a careful balance between privacy and utility. Therefore, any machine learning models trained on synthetic data or any insights drawn from synthetic data should undergo thorough evaluation before deployment in real-world scenarios. If biases are detected, crucial to diagnose their source- -whether stemming from the synthetic data generation algorithm, the noise added to the real data, or other factors- to ensure that the models perform robustly and reliably in practice.
Acknowledgement
We would like thank Akash Srivastava for his insights connecting sliced divergences to synthetic data generation, and for sharing [RL21] with us.
10

---

References
[ABK*21] Sergul Aydore, William Brown, Michael Kearns, Krishnaram Kenthapadi, Luca Melis, Aaron Roth, and Ankit A Siva. Differentially private query release through adaptive projection. In International Conference on Machine Learning, pages 457-467. PMLR, 2021.
[BBDSI2] Jeremiah Blocki, Avrim Blum, Anupam Datta, and Or Sheffet. The Johnson- Lindenstrauss transform itself preserves differential privacy. In 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science, pages 410-419, IEEE, 2012.
[BG188] Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsam- pling: Tight analyses via couplings and divergences. Advances in neural information processing systems, 31, 2018
[BJW 19] Brett K Beaulieu-Jones, Zhiwei Steven Wu, Chris Williams, Ran Lee, Sanjeev P Bhavnani, James Brian Byrd, and Casey Greene. eiiacc-preseevve generative deep neural networks support clinical data sharing. Circulation: Cardiovascular Quality and Outcomes, 12(7)=000112 122, 2019.
[BKZ23] Alex Bie, Gautam Kamath, and Guojun Zhang. Private GANs, revisited. Transactions on Machine Learning Research, 2023.
[BS16] Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography: 14th International Confer- ence, TCC 2016-B, Beijing, China, October 31-November 3, 2016, Proceedings, Part I. pages 635-658. Springer, 2016.
[BSV22a] March Boedihardjo, Thomas Strohmer, and Roman Vershynin. Privacy of synthetic data: statistical framework. IEEE Transactions on Information Theory, 69(1):200- 527, 2022.
[BSV22b] March Boedihardjo, Thomas Strohmer, and Roman Vershynin. Private sampling: a noiseless approach for generating differentially private synthetic data. SIAM Journal on Mathematics of Data Science, 403):1082-1155 2022.
[CBV+21] Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, and Karsten Kreis. Don't ' gener- ate me: Training differentially private generative models with sinkhorn divergence. Advances in Neural Information Processing Systems, 34:1 14:12480-1 2021.
[CLWX21] Kuntai Cai, Xiaoyu Lei, Jianxin Wei, and Xiaokui Xiao. Data synthesis via dif- ferentially private markov random fields. Proceedings of the VLDB Endowment, 2021.
[COF20] Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. Gs-wgan: A gradient- sanitized approach for learning differentially private generators. Advances in Neural Information Processing Systems, 33: 33:12673-1266 2020.
[CYZF20] Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 343-362, 2020.
[DAHY24] Konstantin Donhauser, Javier Abad, Neha Hulkund, and Fanny Yang. Privacy- preserving data release leveraging optimal transport and particle gradient descent. arXiv preprint arXiv:240J. 10l::2401.7777 17823, 2024.
[Dat23] DataCebo.
Synthetic data metrics. https: htops://ddcss sdv wadt.dder/dmetrrics/// 2023.
[DHMS21] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34:6478-6499 6499, 2021.

---

[DR14] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends@ Theoretical Computer Science, 9(3-4):211-40) 2014.
[DR16] Cynthia Dwork and Guy N Rothblum. Concentrated differential privacy. arXiv preprint arXiv:1603.01 2016.
[Dud69] R. M. Dudley. The speed of mean Seivenko-Cantelly convergence. Ann. Math. Stats., 40(1))40-500 Feb. 1969.
[DZS18] Ishan Deshpande, Ziyu Zhang, and Alexander G Schwing. Generative modeling using the sliced Wasserstein distance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3483-3491 2018.
[EKKL20]] Marek Eliák, Michael Kapralov, Janardhan Kulkarni, and Yin Tat Lee. Differentially private release of synthetic graphs. In Proceedings of the Fourleenth Annual ACM- SIAM Symposium on Discrete Algorithms, pages 560-578. SIAM, 2020.
[fM22]
International Organization for Migration. The global synthetic data dashboard.
https://www www.aaacclllabbratiii.... org/ Elobal-victime perpetratoo-ssynheetree dataset, 2022.
[GAH* + 14] Marco Gaboardi, Emilio Jesús Gallego Arias, Justin Hsu, Aaron Roth, and Zhi- wei Steven Wu. Dual query: Practical private query release for high dimensional data. In International Conference on Machine Learning, pages 1170-1178. PMLR, 2014.
[GAL13] Manuel Gil, Fady Alajaji, and Tamas Linder. Rényi divergence measures for commonly used univariate continuous distributions. Information Sciences, 249: 149:124-118 131-131, 2013.
(GG20) Ziv Goldfeld and Kristjan Greenewald. Gaussian-smoothed optimal transport: Metric structure and statistical efficiency. In International Conference on Artificial Intelligence and Statistics, pages 3327-3337. PMLR, 2020.
[GG21] Ziv Goldfeld and Kristjan Greenewald. Sliced mutual information: A scalable mea- sure of statistical dependence. Advances in Neural Information Processing Systems, 34:17567-17578, 2021.
[GGK20] Ziv Goldfeld, Kristjan Greenewald, and Kengo Kato. Asymptotic guarantees for generative modeling based on the smooth Wasserstein distance. Advances in neural information processing systems, 33:2527-2539, 2020.
[GGNWP20] Z. Goldfeld, K. Greenewald, J. Niles-Weed, and Y. Polyanskiy. Convergence of smoothed empirical measures with applications to entropy estimation. IEEE Trans. Inf. Theory, 66(7):4368 4391, Jul. 2020.
[GMH120] Chang Ge, Shubhankar Mohapatra, Xi He, and Ihab Ilyas. Kamino: Constraint-aware differentially private data synthesis. arXiv preprint arXiv:2012. 2020.
[HAP21] Frederik Harder, Kamil Adamczewski, and Mijung Park. Dp-merf: Differentially private mean embeddings with randomfeatures for practical pereecypp data generation. In International conference on artificial intelligence and statistics, pages 1819-1827. PMLR, 2021.
[HC24] Shlomi Hod and Ran Canetti. Differentially private release of israel's national registry of live births. arXiv preprint urXiv:2405.00267. 2024
HJSP23] Frederik Harder, Milad Jalali, Danica J Sutherland, and Mijung Park. Pre-trained perceptual features improve differentially private image generation. Transactions on Machine Learning Research, 2023.
[HMDDC17] Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Mem- bership inference attacks against generative models. arXiv preprint arXiv: 170550705 2017.
12

---

[JYVDS19] James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. PATE-GAN: Generating synthetic data with differential privacy guarantees. In International conference on learning representations, 2019.
[KLN 11] Shiva Prasad Kasiviswanathan, Homin K Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What can we learn privately? SIAM Journal on Computing, 40(3):793-822 826, 2011.
[LBBH98] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):33) 36(11):2278- 2324, 1998.
[LJ Zinan Lin, Alankar Jain, Chen Wang, Giulia Fanti, and Vyas Sekar. Using gans for sharing networked time series data: Challenges, initial promise, and open questions. In Proceedings of the ACM Internet Measurement Conference, pages 464 483, 2020.
[LQS112] Ninghui Li, Wahbeh Qardaji, and Dong Su. On sampling, anonymization, and differ- ential privacy or, k-anonymization meets differential privacy. Proceedings of the 7th ACM Symposium on Information, Computer and Communications Security, pages 32-33, 2012.
[LT19] Jingcheng Liu and Kunal Talwar. Private selection from private candidates. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing. pages 298-309, 2019.
[LTVW23] Terrance Liu, Jingwu Tang, Giuseppe Vietri, and Steven Wu. Generating private synthetic data with genetic algorithms. In International Conference on Machine Learning, pages 22009-22027. PMLR, 2023.
[LVW21] Terrance Liu, Giuseppe Vietri, and Steven Z Wu. Iterative methods for private syn- thetic data: Unifying framework and new methods. Advances in Neural Information Processing Systems, 34:690-702, 2021.
[LZC+21] T. Lin, Z. Zheng, E. Chen, M. Cuturi, and M. Jordan. On projection robust optimal transport: Sample complexity and model misspecification. In International Conference on Artificial Intelligence and Statistics (AISTATS-2019), pages 262-270, Online, 2021.
[Mir17] Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pages 263-275. IEEE, 2017.
[MMS21] Ryan McKenna, Gerome Miklau, and Daniel Sheldon. Winning the NIST contest: A scalable and general approach to differentially private synthetic data, arXiv preprint arXiv:2108 04978, 2021.
[MMSM22] Ryan McKenna, Brett Mullins, Daniel Sheldon, and Gerome Miklau, Aim: An adaptive and iterative mechanism for differentially private synthetic data. arXiv preprint arXiv:2201. 12677, 2022.
[MSGHI18] Kevin R Moon, Kumar Sricharan, Kristjan Greenewald, and Alfred O Hero III. En- semble estimation of information divergence. Entropy, 20(8):560, 2018
[MSM19] Ryan McKenna, Daniel Sheldon, and Gerome Miklau. Graphical-model based estima- tion and inference for differential privacy. International Conference on Machine Learning, pages 1335-4444. PMLR, 2019.
[MZKH23] Shubhankar Mohapatra, Jianqiao Zong, Florian Kerschbaum, and Xi He. Differentially private data generation with missing data. arXiv preprint urXiv:2310. 11548, 2023.
[NCTI6] S. Nowozin, B. Cseke, and Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. In Proceedings of the Annual Conference on Advances in Neural Information Processing Systems (NeurIPS-2016) pages 271 271-279, Barcelona, Spain, Dec. 2016.
13

---

[NDC 20] K. Nadjahi. A. Durmus, L. Chizat, S. Kolouri, S. Shahrampour, and U. Simsekli. Statistical and topologiaal properties sliced probability divergences. Proceedings of the Annual Conference on Advances in Neural Information Processing Systems (NeurIPS-2020), Online, Dec. 2020.
[3]L+23] Joseph P Near, David Darais, Naomi Lefkovitz, Gary Howarth, et al. Guidelines for evaluating differential privacy guarantees. Technical report, National Institute of Standards and Technology, 2023.
[NPA23] Valtteri A Nieminen, Tapio Pahikkala, and Antti Airola. Empirical evaluation of amplifying privacy by subsampling for gans to create differentially private synthetic tabular data. TKTP 2023: Annual Symposium for Computer Science, 2023.
[NTZ13] Aleksandar Nikolov, Kunal Talwar, and Zhang. The geometry of differential privacy: the sparse and approximate cases. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing, pages 351-360, 2013.
[NWD20] Marcel Neunhoeffer, Zhiwei Steven Wu, and Cynthia Dwork. Private post-gan boosting. arXiv preprint arXiv:2007.11 1934, 2020.
[PS22] Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential privacy. In International Conference on Learning Representations, 2022.
[PW23] Yury Polyanskiy and Yihong Wu, Information theory: From coding learning. Cambridge university press, 2023+
[RL21] Alain Rakotomamonjy and Ralaivola Liva. Differentially private sliced Wasserstein distance. In International Conference on Machine Learning, pages 8810-8820. PMLR, 2021.
[RLP+20] Lucas Rosenblatt, Xiaoyan Liu, Samira Pouyanfar, Eduardo de Leon, Anuj Desai, and Joshua Allen. Differentially private synthetic data: Applied evaluations and enhancements. arXiv preprint arXiv:2011 Stt:::2011.0555: 1.05537, 2020.
[RPDB1I] J. Rabin, G. Peyré, J. Delon, and M. Bernot, Wasserstein barycenter and its application to texture mixing. In Proceedings of the International Conference on Scale Space and Variational Methods in Computer Vision (SSVM-2011) 1), pages 435-446, Gedi, Israel, May 2011.
[RTMT21] Diane Ridgeway, Mary Theofanos, Terese W Manley, and Christine Task. Chal- lenge design and lessons learned from the 2018 differential privacy challenges. NIST Technical Note 2151, 2021.
[Sma23] SmartNoise. Smartnoise sdk: Tools for differential privacy on tabular data. https: //github.com/ //ggithbb..com//p////////eaee/ee wtwpe/ppmaaatooiseeeeeeeeee 2023.
[SOT22] Theresa Stadler, Bristena Oprisanu, and Carmela Troncoso. Synthetic data-
anonymisation groundhog day. In 31st USENIX Security Symposium, pages 1451-1468, 2022.
[SSK12] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
[SXGS20] Akash Srivastava, Kai Xu, Michael U. Gutmann, and Charles Sutton. Generative ratio matching networks. In International Conference on Learning Representations, 2020.
[TKP19] Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten. Dp-cgan: Differen- tially private synthetic data and label generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[TMH+21] Yuchao Tao, Ryan McKenna, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. Benchmarking differentially private synthetic data generation algorithms. 2021.
arXiv preprint
14

---

[TTCY2I] Shun Takagi. Tsubasa Takahashi, Yang Cao, and Masatoshi Yoshikawa. P3gm: Private nigh-dimensional data release via privacy preserving phased generative model. In 2021 IEEE 37th International Conference Data Engineering (ICDE), pages 169-180. TEEE, 2021.
[TWB+ + 19] Uthaipon Tantipongpipat, Chris Waites, Digvijay Boob, Amaresh Ankit Siva, and Rachel Cummings. Differentially private mixed-type data generation for unsupervised learning. arXiv preprint arXiv:1912.03250, 1:13, 2019.
[UV20] Jonathan Ullman and Salil Vadhan. PCPs and the hardness of generating synthetic data. Journal of Cryptology, 3374)2007--2122, 2020.
[VAA 22] Giuseppe Vietri, Cedric Archambeau, Sergul Aydore, William Brown, Michael Kearns, Aaron Roth, Ankit Siva, Shuai Tang. and Steven Z Wu. Private synthetic data for multitask learning and marginal queries. Advances Neural Information Processing Systems, 15:18222-18299 2022.
[VCH+22] Margarita Vinaroz, Mohammad-Amin Charusaie, Frederik Harder, Kamil Adam- czewski, and Mi Jung Park. Hermite polynomial features for private data generation. In International Conference on Machine Learning, pages 22300-22324. PMLR, 2022.
[VFT+ 19] T. Vayer, R. Flamary, R. Tavenard, L. Chapel, and N. Courty. Sliced Gromov- Wasserstein. In Proceedings of the Annual Conference on Advances in Neural Infor- mation Processing Systems (NeurPPS-2199) Vancouver, Canada, Dec. 2019.
[VTB +20 Giuseppe Vietri, Grace Tian, Mark Bun, Thomas Steinke, and Steven Wu. New oracle- efficient algorithms for private synthetic data release. In International Conference on Machine Learning, pages 9765-9774. PMLR, 2020.
[WKV09] Qing Wang. Sanjeev R Kulkarni, and Sergio Verdú. Divergence estimation for multidi- mensional densities via k-nearest-neighhor distances. IEEE Transactions on Informa- tion Theory, 55(5):2392-24055 2009.
[WSH+23] Hao Wang, Shivehander Sudalairaj, John Henning, Kristjan Greenewald, and Akash Srivastava. Post-processing private synthetic data for improving utility on selected measures. In Conference on Neural Information Processing Systems, 2023.
[WY20] Yihong Wu and Pengkun Yang. Polynomial methods in statistical inference: Theory and practice. Foundations and Trends@ in Communications and Information Theory, 17(4):402-55) -86, 2020.
[XG Kai Xu, Georgi Ganev, Emile Joubert, Rees Davison, Olivier Van Acker, and Luke Robinson. Synthetic data generation of many-to- datasets via random graph generation. In International Conference on Learning Representations, 2022.
[XLW 18] Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou, Differentially private generative adversarial network. arXiv preprint 1rXiv:1802.06733 2018
[YAS+23] 23] Yilin Yang, Kamil Adamczewski, Danica J Sutherland, Xiaoxiao Li, and Mijung Park. Differentially private neural tangent kernels for sivaacypreserving data generation. arXiv preprint 2303::23003.168 2023.
[ZCP+ 17] Jun Zhang, Graham Cormode, Cecilia M Procopiuc, Divesh Srivastava, and Xiaokui Xiao. PrivBayes: Private data release via Bayesian networks. ACM Transactions on Database Systems (TODS), 42(4):1-41 2017.
[ZWL+211] Zhikun Zhang, Tianhao Wang, Ninghui Li, Jean Honorio, Michael Backes, Shibo He, Jiming Chen, and Yang Zhang (PrivSyn): Differentially private data synthesis. In 30th USENIX Security Symposium (USENIX Security 2/), pages 929-946, 2021.
15

---

A Privacy Analysis
Before diving into the proof of Theorem 1, we will first revisit the definition of Rényi differential privacy (RDP) [Mir17, DR16, BS16] and its connection with differential privacy (DP). Additionally, we will recall useful property of DP known privacy amplification by subsampling.
Definition 6. randomized mechanism M Raxd O satisfies (a, if for any adjacent
datasets X, X' Ruxd, we have
sup Da MM( so x~x
where Da is the Rényi-a divergence.
Proposition 2 (Prop. 3 in [Mir17]). IfM is (a,, c)-DP mechanism, also satisfies (e+ In(1/8) DP for all 8 (0, 1).
a-1 8)-
We remind readers that datasets X and X' are considered adjacent if they differ in single row, say the i-th row, such that [X, X .ll I 1 where X,, and X are the i-th row of X and X', respectively. This condition applies, for example, unbounded DP (add/remove one record) when the L2 norm of each record is upper bounded by 1.
We revisit privacy amplification by subsampling. useful technique for handling large-scale datasets that can save computational resources and memory. It proposes that the privacy guarantees of a DP mechanism be improved by randomly subsampling the private dataset before applying the DP mechanism [KLN*11 11, LQS12, 12, BBG18].
Lemma 1. Given dataset X [xf .... "" PoissonSample, independently draws Bernoulli random variables o, ~ Bern(r) for i E [m] and outputs subset [a: =1,=1, If a mechanism M is (c, 6),D, then Mo PoissonSample, satisfies ('',8')-DP where = log(1 T(e" - -1)) and 8' Tô.
We introduce some notations and a result from [GAL13] that will be used in proving the following lemma. For a matrix A E Rmxn, its determinant is denoted by |A]. The vectorization of A is defined as
(ec(A)= [a1,1. 10, ,i,
amm,n]
For two matrices A E R and B E IRXg, their Kronecker product A @ B is the pm x gn block matrix:
a1,1
a
B
A 8 A A
am,, B
am
B
We recall a result from [GALI3, Table 2]:
Da E)
a ( n²) ((1 - a: aoE 4²)
111 - [11-a{E + aΣ'
(7)
2
2(a 1)
-1 1
whenever aΣ-1 +(1 (1- (1-a) a) 0.
Next, we establish the RDP guarantees for our mechanism.
Lemma 2. Assume
and & 2a22 a) d. Let U Bdxm' and V Raxm be two random matrices whose elements are drawn independently from (N0, d-) and N N(0, respectively. Then the mechanism M(X) (U,XU + V) satisfies (a.,)) RDDD where
m'a
20
16

---

Proof of Lemma 2. Since X is not random, the mechanism MMX) maps X to (dm' + +mm variate
Gaussian distribution with mean zero and covariance
Cov voooo((XU+ (xoc + v)
veo(U*r
-)=
Ldw'
X Im
X & Im
(XX
@
Imm) +
do
Iame
Ia
X
d=
10 Im
X XX" d
d B Im's
(8)
la
X
where we define
-
Combining (7) with (8) yields
X
xx"
+ I
m'
(1-a)B- +
(a(M(()))M((())
In
2(a - 1)
B 1 B
m'
2(a [((1 a In B +aln B'I) In (11 a +BB]]
Next, we compute the determinants in the above Rényi divergence. Using the formula for determinants of block matrices, we know
B
B Ido²In.
For the linear combination, we have
(1 a)) B + a
Ia
(1 (1-a)x" fax
(1 x
(1 (1- +aXX" do
In (1 x1-aa)x++oX'Xx (Xx-)
We denote
A (1 X +aX' -
- (11 (Xx +xx) - -a - -XxX') (X)XXX) - X') +aX(X XX)T
=-a(-(X X'(X- -X)" +X(X -X) + -X- -X)x²
-a X'XX-XX" -X)" +aX(X -X)" +a(X XX)X"
= =aa- -X)(XXXXXXX
Since X and X' only differ in the i-th row, only one entry of this matrix can be nonzero, specifically,
a a² < An S 0.
Then
m'
D. (M())
[[[1 ( ((1-a) In B + aalnBB In (1 a.B aB']
2(x - 1)
m'
In Id Iml in I A
2(a 1)
m'
[I((d=
In(do² + An)]
2(a
17

---

Recall the assumption that | - a² - I yo² for E (0, d). Then using the fact that log(a) is
concave in a and has derivative 1/a for 0,
m'
D((M(())(M((X))
2( - 1) do + Aii Au
m
v 2(a 1) do + (a - a?) (a² a)
m'
2(a 1) (d) 7 (a² a)
m'
2(d 7 - 1) (a a)
m'a
20
Proof of Theorem 1. Theorem I follows directly from combining Lemma with Proposition 2.
Next, establish a privacy bound assuming the projection matrix U is deterministic. Comparing this with Theorem 1, we observe that by randomly selecting the projection matrix, we can achieve a tighter privacy bound, even if it is disclosed by the privacy mechanism.
Proposition 3. Lei X E IRuxd represents the original dataset. Let V E Ruxxmm be a random noise matrix with each element independently drawn frow ViJ ~ N((, Let E Rdxmm be deterministic matrix such that for any E [m]]
U 11111)++1
U.gh are orthonormal
The privacy mechanism Mae (X)
A XU + V satisfies
ma 2x
In(1/5)
+
DP.
Proof of Proposition 3. The privacy mechanism Mae maps nm'-variate Gaussian distribution with mean and covariance being:
mean XU + V) = vec(XU), Cov(XU + V) =o2 Inmm
Using (7) yields
Da(M(X))IM(X)) = a
vec vec(XU XU XXU) voc -X-XXU)
2
a
vec (e - -X)UU"" vex(X
20
20 (X,, X1
a
Without loss of generality, we assume
X,, - X [r, ... with |x| S 1.
Additionally, recall the assumption that for any j E [m]. U U1( Hence, -110-114411 + UR U2, I 1, leading
U tijk orthonormal.
m
MM(M(X))M( <
a
U2 <
am 20
20
As result, our privacy mechanism (a, desired DP guarantee.
omD 2
RRDP. Combining it with Proposition yields the
D
18

---

B Other Omitted Proofs
Proof of Proposition 1. The property fk.
fk (Pxl1 2 0 follows by the non-negativity of the
f-divergence in the integrand. If Px = Qx, then clearly
1
oD (PxI(Qx)
DP =0
vol(Sx(R.)) (Rd))
since the f-divergences (Po Qor are all zero by definition.
Conversely, by the nonnegativity of the f-divergence, SD SD. (Px]Qxx = 0 implies
Dp =
for almost every 0 E &xx Additionally, since f is strictly convex at 1, we have
PoTx+N QoTx+N-
As result, by the invertibility of Gaussian convolution,
Pørx Q
for almost every 0 E Sy (Rd) since f-divergence nullifies the arguments are equal. Since the distributions are equal, the moment generating functions of these projections are equal, i.e. for all t &
Ez 1e 2 Rzz-QoTx 2
i.e.
ExaA 1e 0 Ex Ex~exle (e 10xx
Since Ot is dense in Rd when Sx (R and t E Rt, we have that for any E Rd
Ex~ =xxxxne"
But these are just the moment generating functions of Px and Qx. Since the moment generating functions exist and are equal for all the densities Px. Qx must be equal.
Proof of Corollary 1. By the assumption that D, is consistent estimator of the f-divergence, b with constant implies that for any fixed w.
m
lim L(w) = m Σ Dj (P abb+00
where N N(0, IR).
In the first case, we assume Gw- (Z) ~ Px. Substituting w = w" into (9) yields
lim L(w")()))))) 1 Σ Dy D n1b-
0,
where equality with zero follows because the f-divergence nullifies its arguments are the same distribution.
For the second case, we assume that for some w" Ω
m
lim I(w*)= lim m,m,b-000
Σ Dj (2
m= m71
Since the Dy (2)N,, terms are all > 0, the fact that this limit exists implies that
lim I(w")= (Rd)) 1 Dy(P (2) 00
SD j.ko2 (Pa.. Px))
By Proposition 1, this equal zero only if Pg (Z) Px.
19

---

Dataset
#Records
#Columns
#Categorical Cols #Numerical Cols
Income
195,665
10
8
Coverage
138,554
19
17
Mobility
80,329
21
19
Employment
378,817
16
15
TravelTime
172.508
16
14
Table 2: Dataset details. In addition to the described columns, each dataset has a binary outcome column, which is used by the evaluation metric Logi tRegression.
C Details the Experimental Results
Datasets We demonstrate our method and baselines using the US Census data derived from the American Community Survey (ACS) Public Use Microdata Sample (PUMS). Using the API of the Folktables package [DHMS21], we access the 2018 California data. Additionally, Folktables package provides five prediction tasks (Income, Coverage, Mobility, Employment, TravelTime) based on a target column and a set of mixed-type features. We provide more details about these datasets in Table
Data pre- preeeeocessigg We pre- pre-process each dataset using the open-source Python library [Sma23]. normalizes numerical columns and one-hot encodes categorical columns, all within privacy- preserving way using an = 0.5. Note that the pre-processing budget is not included in the stated our results. To satisfy the bounded norm assumption our privacy analysis, we normalize the resulting dataset so that the largest row 2-norm is upper bounded by 1. This can be achieved by e.g.. normalizing each row by 1 1/Vd, where d is the number of columns. We subsample each dataset at rate 0.25 using the Poisson mechanism, which amplifies the DP guarantees according to Lemma 1. For all our experiments, we report the amplified c.
More details. For our method and SliceWass, all experiments used batch size of 128 and learning rate 2 10-5, and ran for 200 epochs. For our method and baselines, each model was trained using V100 GPU, with runtimes typically less than hours for our method (200 epochs).
Additional results. results Table 3.
We repeat our experiments with different privacy budget = 8.1 and report
20

---

Single Attribute Similarity
Pairwise Attribute Similarity
Classitier Fl Score
Dutaset
DP Mechanism
KSComp
TVComp
ContSim
CorrSim
Logireegrossion
Income
Algorithm
0.40 +0.03
0.73 +0.01
0.37 +:001
0.94 =0.01
0.35 20.17
SliceMass
0.25 +0.01
0.61 +0.01
0.27 +:000
0.98 =0.01
0.17 #0.03
DP-SGD
0.30
0.37
0.09
0.91 20006
0.15 4:030
PATE
0.19
0.40 +0.05
0.12
0.95 #0.02
0.38
MERF
0.82
0.47 4:003
0.08
0.60 #0.14
0.30
Coverage
Algorithm
0.87 +:0001
0.63
0.88 00.06
0.41 #0.07
SliceWass
+:004
0,86 1:0.01
1:0
0.87 20.08
1:0100
DP-SCD
1:0.02
0.71 +:0.03
0.42 4:0
0,52 10.11
0.27 1:0.9
PATE
1:0.0
0.51 ::0.05
1:0
0.83 10003
4:0 02
MERF
0.44 +0.15
0.52 #:0.01
0.18 +00011
0.61 +:0003
0.48 +0.10
Mobility
Algorithm
0.71 200022
0.85 :: 01
0.50 +:0.01
0.86 20.01
0.70 200.01
SliceWass
0.76 +0.04
0.85 +:0 .2
0.50 +:0.02
0.86 20.04
0.65 2:0.5
DP-SCD
0.52 +0.06
0.72 ::0 06
0.35 +:0.6
0.72 15.15
0.62 2:0 20
PATE
0.08 +0.04
0.53 #:0.03
0.22 +60022
0.83 #0.01
0.54 +0.40
MERF
0.33 +0.00
0.58 #:0.01
0.20 +t0001
0.68 =0.00
0.19 #0.08
Employment
Algorithm
0.67 +0.00
0.83 ::0 00
0.63 +:0.00
0.52 #:0.00
SliceWass
0.75 +0.00
0.81 ::0 00
0.60 +:0.00
0.49 =:0.00
DP-SGD
0.52 +0.07
0.61 #:0.05
0.33 +:0.05
0.29 #:0.44
PATE
0.47 #0.08
0.49 #:0.03
0.26 ++0.02
0.51 #:0.99
MERF
0.95 +0.02
0.65 #:0.04
0.32 ++0.00
0.65 #0.05
TravelTime
Algorithm
0.49 20.01
0.75 +00002
0.45 +:0
0.78 20.03
0.39 20.15
SliceMass
0.62
0.93 #0.01
0.46
DP-SCD
0.41
0.54 +0.00
0.22
0.82
0.11
PATE
0.39
0.14
0.88 #0.04
0.51
+:000
0.37 +:0.03
40001
0.62 20.01
0.1 20.18
Table 3: We repeat the experiment in Table with a different privacy budget
8.1.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction are consistent with the rest of the paper. They are supported by both theorems and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section 5,
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
Please refer to Appendix A and B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide experimental details in Section 4 and Appendix C for reproducibil-
ity.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: The datasets used in this paper are all publicly available. We also provide details in Section 4 and Appendix for reproducing our experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.. ec/
wwweiiisssooooiooooooooooooooo olicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: httpc//gguieee/oo/eeeeeeeeee
/ //nips. cc/
bobbmmmsssinn ssisnnolicyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Section 4 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars for our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the compute resources in Section 4 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics https /nnuurruurp...r cc cc/puullc/Bhhiccsuueeelllll
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification:
All research results presented in this paper conform with the NeurlPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Section 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We provide references to the code, data, and models used in this paper (see Section 4 and Appendix C). Their licenses be located within the references.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide details and discuss limitations about algorithm in the paper.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "LFAVrJcnHb", "reviewer": "Reviewer_NMCZ", "summary": "This paper proposes a DP generative modeling technique via f-divergence and random projection. Specifically, both real data and synthetic data are randomly projected into a lower-dimensional space, where the noise is added to the aggregation of the projected data such that the effect of individual data point is bounded. Minimizing the smoothed-slicing f-divergence between the noisy embeddings will guarantee the match of two distributions, thereby can be used to train a generative model. Compared to adding noise to gradients (e.g. DP-SGD), this method is more convenient and scalable. Experiments show that it generally outperforms prior related methods in synthesizing tabular data.", "questions": "1. Existing methods move to adversarial dual form due to \"...they often suffer from scalability issues and are not friendly to gradient-based optimization...\" Now you circumvent the adversarial training, then does the proposed method suffer from scalability issues or not friendly to gradient-based optimization?\n2. Maybe I am out of the field, but what does \"slicing and smoothing\" (line 116) exactly mean? What are the differences? And you term the proposed divergence \"smoothed-sliced f-divergence\", what does \"smoothed-sliced\" mean?\n3. In remark 1 the authors claim that [RL21] did something wrong in their derivation because the slicing matrix U is not included in the privacy mechanism. I am not familiar with [RL21], but is U independent of the dataset X? Why will U affect the privacy analysis?\n4. Line 221, \"...we can achieve a tighter privacy bound by reducing a factor of...\" How this factor is derived?\n4. The proposed method seems quite universal, i.e. should be scalable to other domains like image datasets as well, is it true?", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Looking for better alternatives to DP-SGD is an active research area to circumvent the common issues DP-SGD has in practice, thus is well-motivated and of wide interest in the related field.\n2. The method is complete, with full description and theoretical privacy analysis.\n3. Empirically, multi-dimensional evaluation metrics are considered, and the proposed method outperforms other alternatives in most metrics.", "weaknesses": "1. The proposed method is similar to the MMD-based methods (references on line 96) in the following way:\n    + MMD-based methods apply MMD as the divergence measure, since MMD(P, Q)=0 iff P=Q, similar to $D_f$(P, Q)=0 iff P=Q, thus both divergence measure can be used to train a generative model.\n    + MMD-based methods minimize the distance between kernel mean embeddings (KME) of real data and generated data, while noise is added to the KME to retain DP guarantee. In this work, both real and generated data are projected to an embedding space, and noise is injected into the embeddings to retain DP guarantee, so the idea of the overall paradigm is quite similar.\n    + KME in theory is an infinite-dimensional embedding, which is not convenient to add Gaussian noise, so those works propose to approximate KME by projecting it into a finite-dimensional space. In this work, in theory you have infinitely many directions to slice where only a finite number ($m$) of directions are randomly chosen. So both of you apply the finite-projection idea and therefore both of you will have approximation errors.\n\nIn light of the above similarity, the current discussion on the difference and association with MMD-based methods are not adequate. Also, it is more ideal to include some of those methods in the empirical comparison (Table 1), because they ran experiments on tabular dataset as well.\n\n2. For a work in DP, I did not even find sensitivity analysis, which is crucial for proving DP guarantees. Even if it is trivial it should be included for completeness.\n\n3. It looks to me that adding noise to synthetic data (eq. 2) is unnecessary, and will hurt the model utility. Can you please explain why?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "PART 1\n\nWe thank the reviewer for the thoughtful comments and for appreciating the novelty of the work!\n\n---\n**Q1. Comparison with MMD-based methods.**\n\nA1. Indeed, this is a great point! As mentioned in the general response, we have added new experiments to address this.\n\nIn Figure 1 of the attached PDF, we have considered private image generative modeling for MNIST, showing that our approach outperforms the state-of-the-art baseline MERF method for larger privacy budgets (MERF is an MMD-based method). In this experiment, note that MERF, as implemented by the authors thereof, only uses the mean embedding of the real data in each class. This allows them to use much lower noise and achieve better results for low epsilon levels, but damages performance for higher privacy budgets since their approach cannot learn the diversity of each class’s distribution (due to taking the mean only). This explains the trends we see and emphasizes the advantages of our approach that requires no such limiting preprocessing step and can retain the full diversity of the data distribution.\n\nIn Table 1 of the PDF, we also add MERF as a baseline to the tabular data experiment in the main text, showing that our proposed method (often significantly) outperforms MERF in 20 out of the 24 metrics across five datasets, including all 10 categorical metrics, 4 out of 5 classifier F1 scores, and 6 ouf of 9 numerical metrics. We used the implementation at https://github.com/ParkLabML/DP-MERF for this experiment.\n\n\n---\n**KME in theory is an infinite-dimensional embedding, which is not convenient to add Gaussian noise, so those works propose to approximate KME by projecting it into a finite-dimensional space. In this work, in theory you have infinitely many directions to slice where only a finite number (m) of directions are randomly chosen. So both of you apply the finite-projection idea and therefore both of you will have approximation errors.**\n\nNote that using a finite number of slices is a very different form of approximation than simply projecting onto a finite-dimensional space. As a result, our approach is much more amenable to theoretical guarantees. Firstly, in our setting, any errors from finite numbers of slices can be easily controlled since the slices are independent and identically distributed, while errors from truncating an infinite-dimensional space to a few dimensions is difficult to control theoretically. Secondly, note that our total projection dimension (number of slices X slice dimension) is actually fairly large, oftentimes on the order or larger than the ambient dimension of the data itself. This should imply that the loss of distributional information due to slicing is quite minimal indeed.\n\n---\n**Q2.  Sensitivity analysis**\n\nA.2. You are absolutely right. In most DP work, particularly when noise is added to a statistical query, a sensitivity analysis lemma is derived to prove the DP theorem. However, our privacy mechanism differs as it is not an additive mechanism; instead, it uses Gaussian noise to randomly project data into a lower-dimensional space and then adding noise. We established our (Renyi)-DP guarantee by directly bounding the Renyi divergence (Lemma 2 in Appendix), and leveraging a prior result that bounds traditional epsilon-delta differential privacy in terms of Renyi-DP. This strategy significantly simplifies our proof. Similar ways for proving DP theorem (without a sensitivity lemma) have been used in other privacy mechanisms, such as in the DP proof for the Johnson-Lindenstrauss transform in [BBDS12].\n\n---\n**Q3. Adding noise to synthetic data (eq. 2) is unnecessary and will hurt the model utility**\n\nA3. Adding noise to synthetic data may seem counterintuitive, but it is essential for our approach to ensure that the learned model is consistent with the true data distribution. Recall that we minimize the loss function, specifically the smoothed-slicing f-divergence (SD), between the distributions of synthetic and real data. Proposition 1 states that SD = 0 iff the real and synthetic data distributions perfectly match. This property holds only if we apply the same amount of noise to both the real and synthetic data distributions.\n\nTo illustrate this intuition, consider an example: let the real data follow a 1D normal distribution $N(0,1)$ and the synthetic data follow $N(0, \\sigma)$ with a trainable parameter $\\sigma$. If we add noise only to the real data distribution, for instance, Gaussian noise $N(0, \\sigma_{\\text{noise}})$, minimizing our loss function to zero will lead to the synthetic data having a greater variance than the real data: $\\sigma = 1 + \\sigma_{\\text{noise}} > 1$.\n\n---\n**Q4.  Does the proposed method suffer from scalability issues or not friendly to gradient-based optimization?**\n\nA.4. Our proposed method should indeed not suffer from these issues, as we circumvent the need for noisy gradients and our objective function is well-behaved and not a min-max problem. \n\n---\n**Q5. What does \"slicing and smoothing\" (line 116) exactly mean?**\n\nA5. “Slicing” refers to (randomly) projecting high-dimensional data into lower-dimensional spaces, like slicing a loaf of bread into thinner pieces. “Smoothing” refers to adding Gaussian noise, which makes the resulting distribution less peaked and more spread out.\n\nWe term the proposed divergence “smoothed-sliced f-divergence” since it (randomly) projects the original and synthetic data  distributions onto lower-dimensional spaces (i.e., slicing), followed by adding isotropic Gaussian noise (i.e., smoothing), and averaging their f-divergence over all projections.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o1LFvNRSdd", "reviewer": "Reviewer_JoQ4", "summary": "The paper proposes to add noise to randomly projected private data along with optimizing a newly proposed metric smoothed-sliced f-divergence to train generative models. Such paradigm can circumvent adding noise to gradient and enable more architecture choices. Experiment show the proposed method perform competitively with existing methods.", "questions": "1. Have the authors tested the algorithm on other type of data, such as image or text?\n2. Do the authors have explanations why the proposed approach could be better than the baseline algorithms?", "limitations": "I did not find any limitation discussion specifically for the proposed approach. Adding it would be beneficial.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed approach directly add noise to data, thus enable more data processing possibilities on privatized data.\n2. The proposed approach is shown to have consistency and perform competitively with baseline algorithms such as DP-SGD, PATE, and SliceWass.", "weaknesses": "1.I feel thee baseline algorithms are kind of weak for the setting considered in the paper (synthetic tabular data generation). I am not an expert in this specific area but SliceWass is proposed 3 years ago, DP-SGD and PATE are classical DP algorithms which have many variants improving upon them in recent years,  e.g., Adam also has DP versions. \n\n2. The algorithm is proposed for generative model training, at least it is what claimed in the title and abstract, but it is only tested on tabular data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s careful reading of our paper and thoughtful comments! \n---\n**Q1. Weak baselines**\n\nA1. We have added experimental comparisons to the state-of-the-art MERF method in the attached pdf. In Figure 1 of the attached PDF, we have considered private image generative modeling for MNIST, showing that our approach outperforms the state-of-the-art baseline MERF method for larger privacy budgets. In this experiment, note that MERF, as implemented by the authors thereof, only uses the mean embedding of the real data in each class. This allows them to use much lower noise and achieve better results for low epsilon levels, but damages performance for higher privacy budgets since their approach cannot learn the diversity of each class’s distribution (due to taking the mean only). This explains the trends we see and emphasizes the advantages of our approach that requires no such limiting preprocessing step and can retain the full diversity of the data distribution.\n\nIn Table 1 of the PDF, we also add MERF as a baseline to the tabular data experiment in the main text, showing that our proposed method (often significantly) outperforms MERF in 20 out of the 24 metrics across five datasets, including all 10 categorical metrics, 4 out of 5 classifier F1 scores, and 6 ouf of 9 numerical metrics. We used the implementation at https://github.com/ParkLabML/DP-MERF for this experiment.\n\n\n---\n**Q2.  Other types of data.**\n\nA.2 See the response to the question above, where we expand our treatment of image data (in addition to the domain-adaptation experiments in the original submission for image data). \n\n---\n**Q3. Why the proposed approach could be better than the baseline algorithms**\n\nA3.  Our intuition is that training generative models can be very unstable and requires careful design of the model architecture. Traditional approaches like DP-SGD, PATE, and their variants ensure differential privacy (DP) by modifying the training algorithm. This modification makes the model hard to converge due to gradient clipping and the noise added to the gradient, leading to potential issues like mode collapse, especially since hyperparameter tuning is challenging under DP constraints. In contrast, our algorithm separates DP from the generative model training by adding noise to low-dimensional projections of real data, which are then used to train the generative models. Although the noisy data may still affect the quality of the synthetic data, hyperparameter tuning becomes much easier, and any optimizer (e.g., SGD) can be applied effectively to optimize the generative model.\n\n---\n**Q4.  Discussions about limitations.**\n\nA.4. Thank you for highlighting this matter. In response to your concerns, we provide a discussion on the limitations and potential future directions below. We will ensure this discussion is incorporated into the revised paper.\n\nIn this paper, we introduce a generic framework for training generative models with differential privacy guarantees. We demonstrate the efficacy of our method through numerical experiments on both tabular and image data. It would be interesting to extend our framework to other types of data, such as natural language processing and time-series data. We establish an asymptotic statistical consistency guarantee for our algorithm. An interesting direction for future research would be to derive finite sample guarantees, specifically sample complexity bounds for our method. Additionally, we believe it is crucial to investigate the capabilities and limitations of private synthetic data from various perspectives. For instance, if a machine learning model trained on synthetic data exhibits algorithmic bias when deployed on real data, it is important to identify the source of this bias and correct the model.\n\n\nWe hope the above answers address your concerns, and if any questions remain, please let us know!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "noEBkGQDUW", "reviewer": "Reviewer_jo82", "summary": "The manuscript introduces a slicing privacy mechanism for training generative models without noisy gradients. This mechanism injects noise into random low-dimensional projections of private data, providing strong differential privacy guarantees. The study introduces the smoothed-sliced \n$f$-divergence and a kernel-based estimator for it, allowing the training of generative models without adversarial training. Empirical results show improved synthetic data quality compared to existing methods.", "questions": "None", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The introduction of the slicing privacy mechanism and smoothed-sliced $f$-divergence represents a novel approach to training privacy-preserving generative models, addressing the limitations of noisy gradient-based methods.\n- The paper provides strong theoretical foundations for the proposed methods, including privacy guarantees and statistical consistency of the smoothed-sliced $f$-divergence.\n- Extensive experiments demonstrate the method's effectiveness in generating high-quality synthetic data across various datasets, outperforming baseline methods.", "weaknesses": "- This is a very solid piece of work. The proposed method is simple yet effective, and I have no particular concerns or issues with it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and for appreciating the merits of the work! If there are any further questions that might lead to improving your score please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
General response:

We thank the reviewers for their thoughtful and helpful comments. Due to the suggestions of several reviewers, in the attached pdf we have included results for two new experiments showing the advantage of our methods over the state-of-the-art MMD-based MERF method. We will add both of these experiments to the revision.

Firstly, in Figure 1 of the attached PDF, we also have considered private image generative modeling for MNIST, showing that our approach outperforms the state-of-the-art baseline MERF method for larger privacy budgets. In this experiment, note that MERF, as implemented by the authors thereof, only uses the mean embedding of the real data in each class. This allows them to use much lower noise and achieve better results for low epsilon levels, but damages performance for higher privacy budgets since their approach cannot learn the diversity of each class’s distribution (due to taking the mean only). This explains the trends we see and emphasizes the advantages of our approach that requires no such limiting preprocessing step and can retain the full diversity of the data distribution.

In Table 1 of the PDF, we add MERF as a baseline to the tabular data experiment in the main text, showing that our proposed method (often significantly) outperforms MERF in 20 out of the 24 metrics across five datasets, including all 10 categorical metrics, 4 out of 5 classifier F1 scores, and 6 ouf of 9 numerical metrics. We used the implementation at https://github.com/ParkLabML/DP-MERF for this experiment.

Meta-review: The paper is on the borderline. The discussion turned out in the authors' favor and I am happy to recommend "accept".

The two main criticisms from the reviewers are:
1. (from JoQ4) Weak experimental evaluation. Why generative model for tabular data only?
2. (from NMCZ) Unclear sensitivity analysis of the differentially private mechanism.

The authors' rebuttal were mostly satisfactory in my opinion. Generative models for tabular data are still actively being researched on when we want to generate synthetic data differentially privately.  Of course, it is not wrong that the title is a bit click-baiting and the content may disappoint readers hoping to read about large language models.  It's not the authors' fault, but a disclaimer early on could be useful.

As for the second criticism, it was resolved during AC-Reviewer discussion.  The proposed mechanism and its RDP analysis appear to be correct. It is not a purely noise-adding mechanism, thus does not have the notion of a "query" nor does it require a sensitivity analysis. 

Finally, I think the main result of the work is interesting (though it is somewhat hidden in the bells and whistles of "multiple-slices" and "generative models"). I think it is worth emphasizing the following result more crisply (which I summarize in English)

> Random projection matrix results in (sometimes) tighter privacy guarantees than the alternative simple analysis that conditions on the projection matrix.  

> The key observation is that a random projection matrix symmetrizes the distribution so that for both X and X' the mean is 0.

I do have a related question:

- If you orthogonalize the iid gaussian matrix, the above main result should still work? Because at the limit of d --> inf,  Gaussian random matrix with your scaling converges to an orthonormal matrix. It makes the analysis harder, but if it works out, then the result is a lot stronger because for large \alpha, the RDP is still finite.
Decision: Accept (poster)