A Unified View of Differentially Private Deep Generative Modeling
Authors: Dingfan Chen
Raouf Kerkouche
Mario Fritz
Venue: Accepted by TMLR ()

Abstract: The availability of rich and vast data sources has greatly advanced machine learning applications in various domains. However, data with privacy concerns comes with stringent regulations that frequently prohibit data access and data sharing. Overcoming these obstacles in compliance with privacy considerations is key for technological progress in many real-world application scenarios that involve sensitive data. Differentially private (DP) data publishing provides a compelling solution, where only a sanitized form of the data is publicly released, enabling privacy-preserving downstream analysis and reproducible research in sensitive domains. In recent years, various approaches have been proposed for achieving privacy-preserving high-dimensional data generation by private training on top of deep neural networks. In this paper, we present a novel unified view that systematizes these approaches. Our view provides a joint design space for systematically deriving methods that cater to different use cases. We then discuss the strengths, limitations, and inherent correlations between different approaches, 
aiming to shed light on crucial aspects and inspire future research.
We conclude by presenting potential paths forward for the field of DP data generation, with the aim of steering the community toward making the next important steps in advancing privacy-preserving learning.

Published in Transactions on Machine Learning Research (04/2024)
A Unified View of Differentially Private Deep Generative Modeling
Dingfan Cherf
crriffan.ceennvisso..o
CISPA Helmholtz Center for Information Security
Raouf Kerkouche
rwwu/Kerkonchheeeisyya de
CISPA Helmholtz Center for Information Security
Mario Fritz
fritz
CISPA Helmholtz Center for Information Security
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id- YgmBD2c9qX
Abstract
The availability of rich and vast data sources has greatly advanced machine learning ap- plications in various domains. However, data with privacy concerns comes with stringent regulations that frequently prohibit data access and data sharing. Overcoming these ob- stacles in compliance with privacy considerations is key for technological progress in many real-world application scenarios that involve sensitive data. Differentially private (DP) data publishing provides a compelling solution, where only a sanitized form of the data publicly released, enabling privacy-preservee
downstream analysis and reproducible research in sensitive domains. In recent years, various approaches have been proposed for achieving privacy-preserving high-dimensional data generation by private training top of deep neural networks. In this paper, we present a novel unified view that systematizes these approaches. Our view provides a joint design space for systematically deriving methods that cater to different use cases. We then discuss the strengths, limitations, and inherent correlations between different approaches, aiming to shed light on crucial aspects and inspire future research, We conclude by presenting potential paths forward for the field of DP data generation, with the aim of steering the community toward making the next important steps in advancing sevaacy-prsservigg learning.
1 Introduction
Data sharing is erucial for the growth of machine learning applications across various domains. However, in many application scenarios, data sharing is prohibited due to the private nature of data (e.g., individual data from mobile devices, medical treatments, and banking records) and associated stringent regulations, such the General Data Protection Regulation (GDPR) and the American Data Privacy Protection Act (ADPPA), which largely hinders technological progress in sensitive areas. Fortunately, differentially private (DP) data publishing (Dwork, 2008: Dwork el al. 2009 Fung el. al. 2010) provides compelling solution, where only a sanitized form of the data, with rigorous privacy guarantees, is made publicly available. Such sanitized synthetic data can be leveraged as a surrogate for real data, enabling downstream statistical analysis using established analytic tools, and can be shared openly with the research community, promoting reproducible research and technological advancement sensitive domains.
Traditionally, the sanitization algorithms designed for capturing low-dimensional statistical characteristies and target at specific downstream tasks (c.g., answering linear queries (Roth & Roughgarden 2010 Hardt &
Rothblum 2010: Blum et al. 2013 Vietri et al. 2020)), which are hardly generalizable to unanticipated tasks involving high-dimensional data with complex distributions. On the other hand, the latest research, inspired by the recent successes of deep gencrative models in learning high-dimensional representations, applics deep
Coorespondence Dingfan Chen (dinffan.chenn@eiipa..e))

---

Published in Transactions on Machine Learning Research (04/2024)
generative models as the foundation of the generation algorithm. This line of approaches, as demonstrated in recent studies (Cao et al., 2021: Chen et al. 2020a Xie et. al., 2018 Yoon et al. 2019; Beaulieu-Jones et al.)
2017
Ghalebikesabi et al. 2023). have shown promising results in sanitizing high-dimensional samples for general purposes.
Towards designing models that are better compatible with the privacy target, recent research typically customizes the training objective for privacy-centric scenarios (Cao et al. 2021 Harder et al., 2021; Chen ct al. 2020a Long et al. 2021), all building on top of foundational generic generator framework. However, research is fragmented as contributions have been made in different domains, different modeling paradigms, different metric and discriminator choices, and different data modalities. So far, a unified view of private generative models is notably missing in the literature, despite its potential to consolidate the design space for systematic exploration of innovative architectures and leveraging strengths across diverse modeling frameworks.
In this paper, we pioneer in providing a comprehensive framework and a unified perspective on existing approaches for differentially private deep generative modeling. Our innovative framework, complemented by an insightful taxonomy, effectively encapsulates approaches from existing literature, categorizing them according to the intrinsic differences in their underlying privacy barriers. We thoroughly assess each category's characteristics, emphasizing crucial points relevant for privacy analysis, and discuss their inherent strengths and weaknesses, with the aim of laying a foundation that supports seamless transition into potential future research.
Moreover, we present a thorough introduction to the key concepts of DP and generative modeling. We highlight the key considcrations that should be accounted for when developing DP generative models to ensure results comparable, error-free results. Furthermore, we introduce laxonomy of existing representalive types of deep generative models, classifying them based on the distinctive privacy challenges present during DP training. This introduction aims to equip researchers and practitioners with a systematic approach for the design and implementation of future pereacy-preserving data generation techniques.
Lastly, we discuss open issues and potential future directions in the broader field of developing DP generation methods. Our objective is not limited to reviewing existing techniques, but also aims to equip readers with systematic perspective for devising new approaches or refining existing ones. This work is thoughtfully written to serve diverse audiences, with an elfort of providing practitioners with a comprehensive overview of the recent advancements, while aiding experts in reassessing existing strategies and designing innovative generative modeling.
solutions for
2 Preliminaries of Differential Privacy
Setting. In this paper, we focus on the standard central model of DP, which commonly agreed upon by all the approaches referenced herein. In this model, a trusted party or server is responsible for managing all data points, executing DP algorithms, and producing sanitized data. that conforms to privacy constraints. This sanitized data, generated from the implemented DP algorithms, can be later shared with untrusted parties or released to the public while ensuring strict privacy guarantees. It is noteworthy that although approaches based on local DP may seem to generate a form of synthotic data due to distrust in the central server and a desire to conceal private information these methods are fundamentally distinct from the ones explored in this work due to differing threat. models and the resulting privacy implications.
users typically modify their own
Definition 2.1 ((c,5)-DP Dwork 2008 ). A randomized mechanism M with range R is (c,0)-DP, if
Pr(M(D) E e e
( Pr[MM(D)) E 0 +
holds for any subset of outputs O C R and for any adjacent datasets D and D', where D and D' diller from each other with only one training example. s is the upper bound of privacy loss, and is the probability of breaching DP constraints. Smaller values of both & and translate stronger DP guarantees and better privacy protection. Typically, M refers to the training algorithm of a generative model. DP ensures that inferring the presence of an individual in the private dataset- by observing the trained generative models
2

---

Published in Transactions on Machine Learning Research (04/2024)
M(D)-is is challenging, with D being the original private dataset. This same level of guarantee also holds when the attacker observes the samples generated by the trained generative models (i.e., the sanitized dataset) due to the post-processing theorem (Theorem 2.1
Privacy notion. There are two widely used definitions for adjacent datasets existing works of DP data generation, which result in different DP notions: the "replace-one" and the aadd-r-rrmooe one" notions:
Replaco-one:
adjacent datasets are formed by replacing one data sample, Le., D' D' (z') DU {2) for some r and r'. This is sometimes referred to bounded-DP in literature.
Add-oorremove-onee: adjacent datasets are constructed by adding or removing one data sample, i.c., D' = D' {a} for some x (or vice versa).
It is crucial to understand that different notions of DP may not provide equivalent privacy guarantees even under identical (c, 6) values, potentially leading to slight differences in comparisons when algorithms are developed under varying privacy notions, a sentiment also noted in Ponomareva et al. (2023). Specifically, the "replacement" operation in the bounded-DP notion can be understood as executing two edits: removing one data point x and adding another x'. This suggests that the replace-one notion may be nested within the Sed--or-removeone one notion, and naive transformation would result in a (2z, 8)-DP algorithm under the replace-one notion from algorithm that was (e, 6)-DP under the add--orrreoove-nee notion. To minimize potential confusion and promote fair comparisons, emphasize that future researchers should clearly specify the chosen notion in their work. Moreover, encourage future research to include privacy analysis for both notions, if technically feasible.
Privacy-proservin data generation is building on top of the closedness of DP under post-processing: a generalive model is trained under a (e, 6)-D mechanism, releasing a sanitized dataset generated by the with the privacy cost
model (for conducting downstream analysis tasks) will also be bounded by (and 6).
Theorem 2.1 (Post-processing (worrk et al. 2014 ). If MM satisfies (e, 8)-D, F o M will satisfy (E, for any lata-independent function F with o denoting the composition operator.
While (e, 6)-DP provides an intuitive understanding of the mechanism's 's overall privacy guarantee, dealing with composition is more convenient under the notion of Rényi Differential Privacy (RDP). Existing approaches typically use RDP aggregate privacy costs across a series of mechanisms (such as multiple DP gradient descent steps during generative model training) and then convert the (e notion at the end (See Appendix D The formal definitions and the corresponding theorems are listed below.
Definition 2.2 (Rényi Differential Privacy (RDP) Mirooov (a, P)-PP with order a, if
2017 1 ) A randomized mechanism M is
[(
( =1 ) Pr.M((D) =
Do (M(())))M(D' a a- log B(-)
SP
holds for any adjacent datascts D and D', where (P(l)) - BogEnni ((Pp)(())()))( denotes the Rényi divergence.
Theorem 2.2 (Composition (Mironov 2017)) ). For a sequence of mechanisms M,,.,,MM s.t. Me is (a, PDD- Vi, the composition M1 o o... o Me (a. Σ. RDD..
Theorem 2.3 (From RDP to (e, 6DP (Balle al. 2020 ). If a randomized mechanism M is (),D)- then M is also (p log(a (aci(aa-)//aaa 1)/a) (log + log (a 1), 6 -DP for any 0 0< 8 < 1.
In literature, achieving DP typically involves adding calibrated random noise, with scale proportional to the sensitivity value (Definition 2.3), 3,,,, to the private dataset's associated quantity to conceal individual influence. A notable instance of this practice can be formularized as the Gaussian Mechanism, as defined below.
Definition 2.3 (Sensitivity). The (global) f,-sensitivity for function f X + Rd that outputs d-dimensional vectors is defined as:
A = max [I ((D')lp
(1)
D.D
3

---

Published in Transactions on Machine Learning Research (04/2024)
over all adjacent datasets D and D', The sensitivity characterizes the maximum influence (measured by lp norm) of one individual datapoint the function's output. When dealing with matrix and tensor outputs, the lp norm is computed over the vectors that result from flattening the matrices and tensors into vectors.
Definition 2.4 (Gaussian Mechanism (Dwork et al. 2014)). Let f: X + Rd be arbitrary d-dimensional function with ég-sensitivity 47. The Gaussian Mechanism Ma. parameterized by o, adds noise into the output, i.e,
Ma(æ)) = f(x) - +N(0,
(2)
Ma is (e,8)-DP for 22 and (a. a(1;)² 20 RRDP.
2.1 Training Deep Learning Models with DP
Additionally, we present the most prominent frameworks for training deep learning models with DP guarantees: Differentially Private Stochastic Gradient Descent (DP-SGD) in Scction 2.1.1 and Private Aggregation of Teacher Ensembles (PATE) in Section 2.1.2
2.1.1 Differenetially Private Stochastic Gradient Descent (DP-SGD)
DP-SGD (Abadi et al. 2016) is an adaptation of the standard SGD algorithm that injects calibrated random Gaussian noise into the gradients during the optimization process, which ensures DP due to the Gaussian mechanism. The algorithm consists of the following steps:
1. Compute the per-example gradients for
of training examples.
2. Clip the gradients to bound their fz-norm (i.e., (z-sensitivity) to ensure that the influence of any individual training example is limited.
3. Add Gaussian noise to the aggregated elipped gradients to introduce the required randomness for DP guarantecs.
4. Update the model parameters using the noisy gradients.
The privacy guarantees provided by DP-SGD are determined by the choice of noise multiplier (which defines the standard deviation of the Gaussian noise by multiplying it with the sensitivity), the mini-batch sampling ratio, and the total number of optimization steps. The overall privacy guarantee can be calculated using the composition rule, which accounts for the cumulative privacy loss over multiple iterations of the algorithm. By default, DP-SGD adopts the add-or-remoce-one notion, leading to a sensitivity value equal to the gradient clipping bound (see Appendix
2.1.2 Private Aggregation of Teacher Ensembles (PATE)
The PATE framework (Papernot al. 2017
2018) consists of two main components: ensemble of teacher models and student model. The training process begins with the partitioning of sensitive data into multiple disjoint subsets. Each subset is then used to train a teacher model independently (and non-privately), limiting the effect of each individual training sample to influence only one teacher model. To train DP student model, a public dataset with similar characteristics to the sensitive data is used. During the training process, the student model queries the ensemble of teacher models for predictions on the public dataset. The teacher models' predictions are then aggregatod using DP voting mechanism, which adds noise to the aggrogated votes to ensure privacy. The student model subsequently learns from the noisy aggregated predictions, leveraging the collective knowledge of the teacher models while preserving the privacy of the original training data.
The sensitivity of PATE is measured as the maximum change in label counts for teacher models' predictions
between neighboring datasets. Given m teacher models, label classes, the counts for class j is defined by the number of teachers that assign class j to query input ã, i.c., na( = : [m], fz) = jl for j E [c], where denotes the i-th teacher model. Changing single data point (whether by replacing, adding, removing) will at most affect data partition and, consequently, the prediction for one teacher trained
4

---

Published in Transactions on Machine Learning Research (04/2024)
the altered partition, increasing the counts by 1 for one class and decreasing the counts by 1 for another class. This results in global sensitivity equal to 4 (m......)) = v2 for both the replace-one and add-or-remose-one notion (see Appendix Cp. To reduce privacy consumption, PATE is associated with a data-dependent privacy accountant method to exploit the fact that when teachers have large agreement, the privacy cost is usually much smaller than the data-independent bound would suggest. Moreover, Papernot et al. (2018) suggest private threshold checking for queries to only use teacher predictions with high consensus for training the student model. Notably, to obtain comparable results to approaches with data-independent privacy costs, extra sanitization via smooth sensitivity analysis is required.
2.2 Important Notes for Deploying DP Models
The development of DP models necessitate thorough examination to ensure their correctness for providing a fair comparison of research progress and maintaining public trust in DP methodologies. We present below a series of critical questions that serve as fundamental sanity checks when developing DP models, This enables researchers to rapidly identify and rule out approaches that are incompatible with DP, thereby optimizing their rescarch efforts towards innovation in this domain.
What will be released to the public and accessible to potential adversaries? The most critical question is to determine which components (e.g, model modules, data statistios, intermediate results, etc.) will be made public and, as a result, could be accessible to potential adversaries. This corresponds to the assumed threat model and establishes the essential concept of privacy barrier, which scparates components accessible lo potential attackers from those that are not.
All components within the attacker-acoessible domaïn must be provided with DP guarantees. One common oversight is neglecting certain data-related intermediate statistics utilized during the model's training phase. These statistics might constitute only a minor aspect of the entire process, or their existence might be implicit, given that they are incorporated into other quantities. Nevertheless, failing to implement DP sanitization for these aspects can undermine the intended DP protection for the outcomes, e.g., the trained model may no longer adhere to DP standards.
For instance, when pre-processing is required for the usage of DP model, an additional privacy budget should be allocated for exposing related statistics such as the dataset's 's mean and standard deviation (Trmmer & Boneh 2021 From a research standpoint, innovations may involve carefully designing DP mechanisms that apply DP constraints only to components accessible by attackers, while other components can be trained or computed non-privately to maintain high utility. A concrete example includes training a discriminator con-privately and withholding it by the model owner in deploying DP generative adversarial norssrks (see Section 4.3) while only privatizing the generator's training and releasing it to the public with a dedicated DP mechanism.
What is the adopted privacy notion and gramularity? While DP assorts that an algorithm's 's output remains largely unchanged when a single database entry is modified, the definition of a "single entry" can vary considerably (reflecting the concept of granularity), and the way to modify the single entry can also be different (embodying the privacy notion). Thus, the claims of DP necessitate an unambiguous declaration of the sense and level at which privacy is being promised. As discussed in the previous section, the distinction in privacy notion is universally crucial in the design of DP mechanisms. On the other hand, the granularity becomes particularly relevant when handling data modalities that exhibit relatively less structural representations, such as graphs and text. For instance, training DP (generalive) language models that provide guarantees at different levels (tokens, sentences, or documents) will lead to substantial differences in the complexity and the application scenarios.
What constitutes the sensitivity analysis? Sensitivity analysis demands rigorous attention, focusing on two primary aspects. The first consideration calls for a clear statement of the sensitivity type in use, e.g,, global, local, and smooth sensitivity. Notably, techniques predicated on local and smooth sensitivity are generally not directly comparable to those depending on the global sensitivity. Second, determining the sensitivity bound during the training of generative model that consists of more than one trainable module may be challenging, as discussed in Section 4.4 which necessitates a meticulous analysis to ensure the correctness of the privacy cost computation.
5

---

Published in Transactions on Machine Learning Research (04/2024)
Measurement
Synthetic data
data D
Generator
compare
Créc
L
update
Figure 1: Overview of Training pipeline of generative models. (Blue arrow: forward pass; Red arrow: backward pass; Dashed arrows indicate optional processes that may not be present in all generative models.)
3 Preliminaries of Generative Models
In this section, we present comprehensive overview of representative generative models, with the aim to develop clear understanding of the essential operations required to achieve DP across different types of generalive models, as well as to demonstrate the fundamenial differences in their compatibility with private training.
3.1 Overview & Taxonomy
Given real data samples æ from a dataset of interest, the goal of a generative model is to learn and capture the characterislics of true underlying distribution p(æ) and subsequently allows the model generate new samples from the learned distribution. At high-level of abstrractoon, the training pipeline of generative models can be depicted as the diagram in Figure ! The "Measurement" block in the diagram summarizes the general process of comparing the synthetic and real data distributions using a "critic", which yields a loss term C that quantifies the similarity between the two. This loss term then acts as the training objective for the generator, with the update signal computed and then backpropagated to adjust the generator's parameters and improve its ability to generate realistic samples.
Furthermore, the diagram outlines two optional processes (indicated by dashed arrows), that are involved some gencrative models but not all. The first optional process involves guiding the training of the generator by feeding (quantities derived from) real data as inputs, which enables the explicit maximum likelihood computation and categorizes the models into two types: implicit density and explicit density. The second optional process involves updating the critic to better capture the underlying structure of the data and more accurately reflect the similarity between the distributions. This distinction highlights the usage of either static (dala-independent) or learnable (dola-dependent) features for the critic function within implicit density models.
We present a taxonomy of existing representative types of generative models whose private training has been realized in literature in Figure g We examine the following tiers in the taxonomy trees that exert significant. influence on the application scenarios and the design of corresponding private training algorithms:
Explicil vs. Implicil Density Models
Learnable vs. Static Critics
Distribution-wiss vs. Point-wise Optimization
Tructable vs. Approrimale Density
Explicit vs. Implicit Density Models. Existing generative models can be divided into two main calegories: explicil densily models define an explicit density function Pmodet 0), while implicit densily models learn mapping that generates samples by transforming easy-to-sample random variable, without explicitly defining density function.
These distinctions in modeling design result in different paradigms during the training phase, particularly in how real data samples are used (or accessed) in the process. Explicit density models typically use real data samples as inputs the generator and also for measurement (as demonstrated in Figure I. thereby enabling the tractable computation or approximation of the data likelihood objective. contrast, implicit density models necessitate real data samples solely for the purpose of distribution comparison measurements.

---

Published in Transactions on Machine Learning Research (04/2024)
Generative
Explicit Density
Implicit Density
Tractable Denaity
Approximate Density
Leernable Critic
Static Orttic
Flow-based Model
VAE Diffusion Model
GAN, EBM
Distribution Matching
Autoregressive Model
Point- Point-wise Generation)
Distribution-wise
Optimization
Figure Overview of different deep generative models.
This distinction demarcates potential privacy barriers for these two types of models during DP model training. In the context of implicit models, it is sufficient to privatize the single access point to the real data (Section 4.1 4.3 However, when dealing with training private explicit density models, it becomes essential to apply DP mechanisms that take both access points into account.
Learnable vs. Static Critics. The Training of generative models necessitales a "critic" to assess the distance between the real and generated distributions, which then builds up the training objectives for optimizing the generator. Specifically for implicit density models, the use of different types of critics could potentially influence the placement of privacy barrier when training DP models (Section 4.2
Within this framework, the critics may exist in two primary forms, namely learnable and static (data- independent) variants. The distinction between the two lies in whether the critic itself is parameterized function that undergoes updates during the training of generative models (learnable), or a data-independent function that remains static during the training process (static).
We do not further differentiate for explicit density models as they typically employ simple, data-independent critic such as l and &g losses. Meanwhile, in contrast to implicit models, varying the critics in explicit models typically does not alter the privacy barrier in DP training. This is due to the constraint imposed by multiple access to real data in training of explicit models, which restricts the flcxibility in positioning the privacy barriers.
Distribution-uise vs. Point-wise Optimization. Generative models are designed to be stochastic and capable of producing a distribution of data. This is achieved by supplying the generator with random inputs (i.e., latent variables), stochastically drawn from a simple distribution, such as the standard Gaussian. The optimization process generally proceeds through mini- bini-batches, essentially serving as point-wise approximations. Through substantial number of update steps that involve various random latent variable inputs, the model is trained to generalize over new random variables during the generation phase, enabling a smooth transition from a pisnt-wise approximation to the distribution-uui objective.
However, certain contexts may not necessitate the stochasticity nature in these models. Instead, there might be an intentional focus on generating a small set of representative samples, a notion that resonates with the "coreset"" concept. This could involve optimizing the model over limited, fixed set of random inputs rather than the entire domain. We label this as poinl-wise oplimizalion to distinguish it from the default. distribution--uia optimization used in training conventional generative models.
Recent studics have revealed intriguing advantages of merging insights from both these strategics, particularly in the realm of private learning. For instance, the point-wise optimization method exhibits remarkable compatibility with private learning primarily arises from the fact that point-wise optimization is generally less challenging comparison the distribution-uus training that requires generalization, which generally improves model convergence, and consequently enhances privacy. However, this point-wist approach has its limitations. Unlike distribution-we wise training, it does not inherently support generalization over new latent.
7

---

Published in Transactions on Machine Learning Research (04/2024)
Measurement Pa523
Messunement
Generator
Synthetic
Generator
Synthatic
S
17
Gols)
Stase
-
P
(a) GAN
(b) Distribution matching
Encoder (p(a)
Forward Diffusion
Decoder (vlela)
Synthelic data
Real data
Reverse Diflusion
Synthetic data
Measurement
Real data
R.RO
Aol
S
ELBO
D
(c) VAE
(d)
Diffusion models
Flow
f+(x)
Inwerse 5 (4)
Synthetic data
Real data 2
Autoregressive lnn Inlett
Synthetic data
Real data
MI
C
8
NLLL
P
(e) Flow
(f)
Autoregressive
Figure 3: Diagram illustrating training process generative models. Blue arrow: forward pass; Red arrow: backward pass.
code inputs. This may restrict the stochastic sampling of new synthetic samples during inference. As a result, there is trado-off between the flexibility of use in downstream applications and improved privacy guarantces.
We do not expressly differentiate between potential optimization strategies for explicit density models within our taxonomy in Figure 2 such distinction is not obvious in the context of explicit density models. In these models, the latent space is typically formulated through transformation of the distribution within the data space. This transformation process in turn complicates the control of stochasticity throughout the training phase and diminishes the applicability of point-wise optimization.
Tractable vs. Approzimate Density, For models defining explicit density, a key distinguishing factor that shows practical relevance pertains to whether they allow exact likelihood computations. These models broadly be categorized into two types: tractable density and approzimate density models. The classification primarily stems from the model structural designs, which either enable tractable density inference or fall within the realm of approximate density.
Existing studics have demonstrated encouraging results when conducting DP training on both types of models. Intrguungly, the DP training mechanisms appear to exhibit minor distinctions when applied to these two different categories. On an optimistic note, such results implies that it might be feasible attain tractable likelihood computations with DP guarantce without considcrable effort. However, it remains unclear to whether the difference in model designs will systematically influence their compatibility with DP training.
3.2 Representative Models
We provide an illustration of the operational flow of representative generative models in Figure g As demonstrated, existing representative generative models can be effectively encapsulated within our unified framework shown in Figure ! We proceed to briefly discuss the key characteristics of each type of generative models and their relation to potential implementations for DP training in this subsection.

---

Published in Transactions on Machine Learning Research (04/2024)
3.2.1 Implicit Density Models
As a canonical example of implicit density model, Generative Adversarial Network (GAN) (Goodfellow jet al. 2014) employs a generator, Go (parametrized by 0), to learn the data distribution with the aid of a discriminator D& (parametrized by Φ) trained jointly in an adversarial manner, obviating the need for explicit density definition. The generator's functionality is enabled by inputting random latent variables, 2, drawn from simple distributions such a standard Gaussian, and mapping these random inputs to the data space. Concurrently, the discriminator is provided with both synthetic and real samples and its training objective is to differentiate between the two. Throughout the training process, the generator and the discriminator compete and evolve, enabling the generator to create realistic samples that can deceive the discriminator, while the discriminator enhances its ability to distinguish between real and fake samples. The original GAN training objective can be interpreted as optimizing the generator to produce synthetic data that minimizes the Jensen-Shannon (JS) divergence between the synthetic and real data distributions. This idea, has been expanded in various GAN training objective extensions explored in the literature. For instance, variants have been proposed based generalizations to any f-divergence (Nowozin et al. 2016), Wasserstein distance (Arjovsky et al. 2017 Guirajani et al., 2017p, maximum mean discrepancy (MMD) (Binkowski et al. 2018
Li et al. 2017). and Sinkhorn distance (Genevay et al. 2018).
Of particular interest to DP training is the observation that many of these divergence metrics can be approximated without requiring the training of discriminator network. This has led to recent research in private generative models, which use static function as the critic instead of discriminator network. While such approaches might fall short in standard (non-private) generative modeling due to a lower expressive power compared to using learnable critic (Uhat is adaptable to large data with diverse properties), they are highly competitive in DP training, as a static critic can effectively speed up convergence, thereby improving privacy guarantecs.
In the case of implicit density models, the generator's 's interaction with the privale dataset is typically indirect (only via the backward pass), meaning that there exists no direct link between the data source, as illustrated in the accompanying diagrams (Figure 3). This configuration presents opportunity to strategically position the privacy barrier anywhere along the backpropagation path where the generator retrieves signals from the real data, facilitating an improved signal-to-noise ratio or simplified implementation. A more comprehensive
understanding is presented in Section 4.1 4.3
3.2.2 Explicit Density Models
Several prominent explicit density models have been developed literature, each with distinct characteristics:
The Variational Autoencoder (VAE) (Kingma & Welling 2014] is trained to maximize the Evidence Lower Bound (ELBO), a lower bound of the log-likelihood, which typically simplifies to 6 losses on the data sample and its reconstruction under standard Laplacian/ Gaaussian noise modeling assumptions. The model comprises trainable encoder and decoder modules. Encoding is conducted through the encoder gø, which maps observed data to its corresponding latent variables, denoted latent variables are typically smaller than the data dimension d, embodying the concept of an information bottleneck (Tishyy et al. 1999. Shwartz-Ziv & Tishby| (017)). The decoder module is responsible for data reconstruction or generation, i.c., z "3, æ. Additionally, VAE imposes regularization on the latent distributions to match the pre-defined prior, thereby enabling the generation of valid novel samples during inference.
2$ z.
The dimensions of these
2019 Ho et al. 2020) operate similarly Diffusion models (Sohl-Dickstein et al. 2015 Song & Ermon to VAEs in terms of maximizing the ELBO. However, instead of using a trainable encoder to map data to latent variables, diffusion models transform the data iteratively through a linear Gaussian operation, represented as æ 9_.. 3 x;-1 4 æ, % ær. This procedure causes the latent variables at the final step ør to form a standard Gaussian distribution and maintain the same dimensionality as the data. The generation process is executed by reversing the diffusion operation, which mcans itcratively applying po(2)_1|zz) (x)) for all time steps t [T]. The trainable component of diffusion models resides in the reverse diffusion process, while the forward process is pre-defined and does not require training.
9

---

Published in Transactions on Machine Learning Research (04/2024)
Kingma & Dhariwal 2018), in contrast, minimize the Negative Log-Iikelihood (NI.I) directly. Uniquely, flow-based models employ the same invertible model
Klow-based models (Rezende & Mohamed, 2015
for both encoding (a Ls z) and generation (z 2 æ), by executing either the flow or its inverse. Due to the inverlibility demanded by the model construction, the dimensions of the latent variables z are identical lo those of the data.
Autoregressive models [Tarochelle Murray 201 Oord et al. 2016 Van Den Oord al. 2016 Van den Oord al. 2016), as another instance of model with tractable density, are also designed minimize the NLL. Unlike some other models, they accomplish this without the need for explicit latent variables encoding mechanism. Instead, these models utilize partially observed data, denoted as #15-1, where each sample is regarded as a high-dimeeniional vector with observations up to the (i 1)th element. The model is then trained to predict potential values for the subsequent element, Z;. Data generation is conducted through an iterative autoregressive process, where elements each data vector predicted one-by-one, starting from initial seeds. This can be represented as Zo 2 P Iisi-1 29 T " 44 wiid- The component subject to training is the autoregressive model itself. Its parameters, denoted by 0, are optimized to best predict the next elements in the sequence based on previously observed values.
As illustrated in Figure 3 all these models require real data derived quantities (such as latent variables) imputs to the generator during the training phase. This necessitates significant difference in the DP training of these models compared to implicit density models, which only need indirect data access through the backward pass. In the context of typical explicit density models, DP constraints must be accounted for, given the access to real data in both the forward and backward pass. This typically rosults in privacy barricrs being directly integrated into the update process of the generator module, as further discussed in Section 4.4
3.2.3 Extensions
Our diagram has been consciously designed encompass future developments, including potential hybrid variants of generative models. It facilitates systematic analysis of the modifications required to transition the original training pipeline to privacy preserving one. Specifically, to train DP variant of such model, one could follow the following steps: (1) Illustrate the model components and information flows using diagrams analogous to those shown in Figure 3 (2) Determine the component(s) that will be provided with DP guarantees, taking into account practical use requirements and a feasible privacy-utility trade-off. (3) Establish the privacy barrier ensure the privacy of the targeted component, which will later be made accessible for potential threat exposure. This step should consider all access paths between the larget component and the data source. (4) Calculate and bound the sensitivity. (5) Implement the DP mechanism and calculate the accumulated privacy cost of the entire training process.
4 Taxonomy
Accompanied by a comprehensive diagram encapsulating the complete spectrum potential design choices for deep generative models, we put forth a classification system for current DP generative methods. This system predicated on the positioning of the privacy barrier within the diagram (Figure []. Specifically, for explanatory purposes, consider the key components within diagram (the Generator, Synthetic data, Measurement, and Real data), resulting in following options for positioning the privacy barrier:
B1: Between Real data and Measurement
B2: Within Measurement
B3: Between Measurement and Synthetic data
B4: Within Generator
B1 through B4 are introduced soquentially, demonstrating the systematic transition of the privacy barrier from the real data source towards the generator end. The data- cata-processing theorem 2.1 ensures that the DP guarantee is upheld as long the data is "sanitized" through DP mechanism prior exposure to potential
10

---

Published in Transactions on Machine Learning Research (04/2024)
adversaries. In this context, if DP training algorithm safeguards against threats introduced by B1, then it also provides the same protective guarantee against attackers defined by B2 through B4.
The generator end typically represents the smallest unit necessary for preserving the full functionality of the model, implying that the privacy barrier cannot be shifted further without compromising the operational capabilities of the generative model. Moreover, we reserve a more detailed discussion on the threat model (privacy barrier) integrated within the adopted DP mechanism (not specifically relevant to generative models) for later sections, where individual approaches will be introduced.
4.1 B1: Between Real Data and Measurement
Threat Model. Establishing a privacy barrier between the Real data and the Measurement entails using a DP mechanism to directly sanitize the data (features), thereby obtaining statistics that characterize the real data distribution for subsequent operations like computing the loss L as Measurement that serves as the training objective for the generator. This approach provides protection against attackers who might gain access to the sanitized data features or any resultant statistics derived from the sanitized features, such as the loss measured on the sanitized data, any gradient vectors for updating the generator, and the generator's model parameters.
General Formulation. Methods within this category typically adopt the distribution matching framework (illustrated in Figure B(b]p which aims to minimize the statistical distance between real and synthetic
data distributions (Harder et al. 2021 Rakotomamoniy & Liva 2021; Vinaroz et al. 2022 This distance is assossed with a static, unlcarnable function, typically applying a data-independent feature extraction function & to project the data samples into a lower-dimensional embedding space and subsequently calculating the (Euclidean) distance between the resulting embeddings of real and synthetic data. The generator is optimized reduce the disparity between the mcan embeddings of synthetic and real data, which can be interpreted as minimizing the maximum mean discrepancy (MMD) between the real and synthetic data distributions [Binkowski al. 2018 Li et al. 2017)-
During DP training of these models, data points e; feature vectors (xx;) are first clipped normalized (by norm) to ensure bounded sensitivity. Subsequently, random noise is injected into the mean features derived from the real samples, e.g,, via Gaussian mechanism (Definition 2.4 The objectives can be formulated as follows:
DD Σ
S
Non-private:
min 0
(ax)
1
Σ
2 (Go (Go()))
min j - īss 0
(3)
D
S
DP: min ji ïs 2 with üD = uD NO0043 0²1)
(4)
with Po = | synthetic data, respectivcly. Mcanwhile, fD denotes the DP-sanitized mcan real data embedding with A being the sensitivity value that characterizes the influence of each real data point on the mean embedding. A visual illustration can be found in Figure 4
1 E121 (z;)) and ğs = |s E (Z)()))) representing the mean features of the real and
Representative Methods. While all mothods in this category adhere the same general formulation, they primarily diverge in their construction of the feature extraction function v and the objective function that forms the training loss L for the generator. DP-Merf (Harder et al. 2021) employs the MMD minimization approach, optimizing generator to minimize the difference between synthetic and real data embeddings, using random Fourier features (Rahimi & Recht] 2007) for the embedding function tb. DP-SWD (Rakotomamonjy & Liva 2021) instead employs random projections u E @d-1 for feature extraction. Specifically, DP-SWD uniformly samples & random directions for data projection, thereby enabling tractable computation of one-dimensional Wasserstein distances along each projection direction. The Sliced Wasserstein Distance (SWD) (Rabin et al. 22012) Bonneel et al.) (2015), which is determined as the mean of onc-dimensional Wasserstein distances over Da-sanitized projections, serves as the training objective for the generator. Similar
11

---

Published in Transactions on Machine Learning Research (04/2024)
Synthetic data
Generator Go
(z(z)
Embedding
(b)((z))
Mean
Measurement
C
publicly accessible wegeeeaeeeeeeeeeeeeeee
Privacy barrier
publicly inaccessible
Embedding
a
Meen
M
Real data
DP sanitization
data-independent
sensitive
DP
Figure 4: Diagram illustrating the general training procedure of methods under B1.
to DP-Merf, PEARL (Liew et al. 2021 employs the Fourier transform as the feature extraction function while offering an alternative interpretation of describing the data distribution using the characteristic function with the characteristic function distance as the objective. Furthermore, PEARL proposes learning re- weighting function for the embedding features, placing greater emphasis the discriminative features, in order to enhance the expressiveness of the plain Fourier features employed in the DP-Merl approach.
Recent research efforts have primarily focused on identifying informative features that can efficiently capture the underlying characteristics of the data distribution. Specifically, DP-HP (Vinaroz et al. 2022 employs Hermite polynomials as the feature embedding function. This choice embedding function reduces the required feature dimension, which consequently decreases the effective sensitivity of the data mcan embodding and leads to an improved signal-tonoise ratio in the DP training. Harder et al. (2022) further propose utilizing feature extraction layers from pre-trained classification networks that capture general concepts learned on large-scale public datasets. Additionally, DP-NTK (Yang et al.. 2023) introduces the use of the Neural Tangent Kernel (NTK) to represent data, resulting in the gradient of the neural network function serving as the feature map, i.e, (x)) = Vol (2:
Privacy Analysis. The privacy analysis for methods in this category involves computing the sensitivity and applying the privacy analysis of associated noise mechanisms, such as the Gaussian mechanism (Definition 2.4 The sensitivity represents the maximum effect of an individual data point the mean embedding:
2 max ④uD - PD 12 D,D
Σ (x,)
D Σ (aaa
55)
D
In existing literature, the replace-one privacy notion is commonly used to compute the sensitivity value A?, resulting in upper bound of 2 when the feature vector by construction has norm equal to or normalized with maximum norm of 1, i.e., [|)(()) S 1. Deriving the sensitivity value for the add-or-remove-one notion is slightly more technically involved, but applying existing techniques used for the replace-one notion leads to a conservative bound of πDî+I (See Appendix). This implies two things: first, the sensitivity value decreases inversely proportional to the size of the dataset, showing the beneficial effect of the "mean" operation over large datasets, which smooths out individual effects through population aggregation. Second, there is minor difference in the computed sensitivity between the two privacy notions: प:म versus T.. This means that the current comparison results hold with negligible effect when the dataset size is sufficiently large. While achieving a lighter bound for the sensitivity value is possible with the add-or-remove-one privacy notion, it may require additional assumptions.
2
In contrast to other studies that compute the (worst-case) global scnsitivity (Definition 2.3 the sensitivity in DP-SWD represents a form of expected value, accompanied by a sufficiently small failure probability. This efficiently harnesses the characteristics of random projections to achieve a tight sensitivity bound, but requires careful comparison to other mothods. When combining this scnsitivity definition with mechanisms that offer (e, 8DD (i.e., the relaxed DP notion), the final privacy guarantee will be weaker than (e, 8), due to the additional failure probability derived from the sensitivity itself.
12

---

Published in Transactions on Machine Learning Research (04/2024)
Privacy barrier
publicly accessible
publicly inaccessible
Synthetic data
Measurement
Generator Go
Discriminator
2
Ga
D
9G
M
gD
DP
data-independent sensitive
DP
data
Figure 5: Diagram illustrating the training pipeline of DP-GAN with a (vertical) privacy barrier of type B2 as shown.
Generator Ge Go(z)
Synthetic
Measurement
2
D Dstu
L
publicly accessible
Privacy barrier
www. www.://www...................... publicly inaccessible
D1
Ltaa
Ltea.
...
M
D k
Ltea.
Real data
DP sanitization
data-independent sensitive
DP
Diagram illustrating the training pipeline of PATE-GAN with (horizontal) privacy barrier of type B2 as shown. Lytu and Lies denote the student and teacher training losses respectively, while Detu is the student discriminator and Di,..., D3 represent the teacher discriminators.
Figure 6:
Analysis, Insights, Implications. Methods under this category present several strengths. Firstly, the "mean" operation adopted during the extraction of descriptive feature embeddings significantly reduces the impact of each individual. This leads to lower sensitivity value that scales in inverse proportion to the number of data points being aggregated through the "mean" operation. As result, strong privacy guarantee can be ensured with less randomness required from the DP mechanism. Moreover, they are straightforward to implement, typically necessit ating just one instance of sanilization on the computed mean feature (known as "one-shot sanitization") throughout the training process, which further saves the privacy consumption in comparison to iterative methods. These methods also converge quickly and can yield acceptable results even under low privacy budget, given the ease of fitting the static target, i.e., the noisy mean.
Nevertheless, they come with certain drawbacks. The static feature might not be sufficiently discriminative informative, lacking the expressiveness found in methods that employ trainable models critics. Furthermore, the "mean" operation could potentially induce unintended mode collapse in the generated distributions, trading off generation diversity for privacy protection. This situation warrants attention in future works, particularly in optimizing the trade-off between the expressiveness of the feature extraction method in the critic and the privacy cost of achieving such expressiveness. A promising direction could be exploit knowledge from public non-sensitive data and/or pre-trained models that better describe data without. compromising the privacy of the sensitive data.
4.2 B2: Within Measurement
Threat Model. The previous calegory focuses on a static, sanitized statistical summary, derived from a inttadnndeenddent function, as a replacement for real data when training generative models. However, learnable functions that are able to adapt to diverse data distribution may offer superior expressive power. In this regard, logical strategy is incorporate DP into the measurement process, particularly by training a DP critic. This privacy barrier sits "within Measurement" and safeguards against adversaries with access to
13

---

Published in Transactions on Machine Learning Research (04/2024)
the critic and subsequent quantities, including information flows to the generator. If gradient sanitization techniques like DP-SGD are employed for updating the critic, the DP mechanism further protects against. attacks targeting all intermediate gradients w.r.t. the critic's parameters during the training phase.
General Formulation. Methods this calegory follow two main principles: Firstly, they use learnable critic (feature extraction function) that dynamically adapts to the private datasct, necessitating a boundary on the potential privacy leakage of such critic. Secondly, the generator is prohibited from accessing private real data directly, its access limited to indirect interaction through the backward pass. This ensures the generator's update signals are fully derived from the learnable critic. As such, developing a DP critic is sufficient to assure DP for the generator module (and the entire model) for privacy-preservigg generation. GAN models (depicted in Figure (aa] meet these criteria and serve as a foundationa framework that most. existing DP methods in this category generally conform to.
Representative Methods. The implementation of the privacy barrier within the Neasurement block is exemplified in DP-GAN Zhang et al. 2018 Xie et al. 2018 and concurrent studies (Beaulieu-Jones et al.
2017 Tristcyyn & Faltings 2018 Alzantot & Srivastava 2019 Xu et al.) 2019 Torkzadeehmaaani et al. 2019 Frigerio et al. 2019). In this context, the discriminator, acting as the learnable critic model, is trained via DP-SGD (Section 2.1.1. The privacy of the generator is ensured by the post-processing theorem. As per the public timestamp of paper releases, this approach can be traced back to Beaulleu-Jones et al. (0017), who proposed training an ACGAN (Auxiliary Classifier GAN) (Odena et al. 2017) in a DP manner to conditionally generate samples for downstream analysis tasks on medical data. The training pipeline can be formalized as follows, with the illustration shown in Figure
9b VoL(Go D&)
(Discriminator gradient)
(6)
96 VeL(Go, D&)
(Generator gradient)
(7)
9D Mo,clg (2) clip(g). (0
C) N00,
(Apply DP sanitization)
(8)
8 (1-1) (11) -1I 9D
(()
(Discriminator update)
(9)
0(++1 o( G 9G
(t)
(Generator update)
(10)
The generator Go and discriminator D& are parameterized by 0 and o. respectively, with ne and no denoting their learning rates. Ma,.c refers to the Gaussian mechanism in DP-SGD, with representing the noise scale and C indicating the gradient clipping bound. Although have omitted the sample index in the above
equations for the sake of brevily, it should be noted that the clipping function in Equation g is expected to take per-example gradients as inputs, adhering to the standard procedure of DP-SGD (Section 2.1.1 Specifically, it suffices to apply the sanitization only to the gradients that depend on the real data samples, including indirect usage of real samples, such as through gradient penalty terms (Gulrajani et al.] 2017
Unlike DP-GAN that employs DP-SGD for training the DP discriminator, PATE-GAN (Yoon et al. 2019 leverages the PATE framework (Section 2.1.2 to train its DP (student) discriminator. PATE-GAN comprises three main components that are jointly trained throughout the process: multiple (non- (non-private) teacher discriminators, a DP student discriminator, and a DP generator. Similar to the original PATE framework, PATE-GAN starts by partitioning the real dataset into disjoint subsets, which subsequently serve to train the teacher discriminators independently. In each training iteration, PATE-GAN follows a sequence of steps: (1) independently updating the teacher discriminators using minc-batch samples from real data partitions and synthetic samples drawn from the generator; (2) qucrying the tcacher discriminators with a set of synthetic samples; (3) the teacher discriminators then engage in voting process on the real/fake predictions for the synthetic samples they have received, and apply DP noise to the results of the vote; (4) training the student discriminator with the query synthetic samples as input and the DP aggregation of teacher predictions as the label; (5) finally, jointly updating the generator and the student discriminator, with the generator querying the student discriminator with new synthetic samples and obtaining update gradient signals from the DP student discriminator. A visual illustration is presented in Figure G
While the discriminator in the GAN framework aims to distinguish between two distributions, recent research uncovered intriguing results when the learnable critic is designed to target specific downstream tasks, such
14

---

Published in Transactions on Machine Learning Research (04/2024)
classification. Specifically, Private-Set [Chen et al. 2022a) employs a classification network as learnable feature extractor, which is trained with DP-SGD. This learnable feature extractor, combined with the alignment in the gradients serving as the critic, encourages the synthetic data to emulate the training trajectories of the real data during the training process within a classification network, making the synthotic data useful for training downstream classifiers and safe for public release due to the DP guaranlees embedded within the measurement process.
Privacy Analysis. Methods in this category inherit the privacy notion and sensitivity computation from their respective framework for training the DP critic (See Section 2.1.1 2.1.2), while also inheriting the need for careful consideration regarding the application of data-dependent privacy analysis or adherence to privacy nolion constraints to emsure comparable results. For methods grounded by DP-SGD, this results in a noticeable disparity between the replace-one and onn DP notions, as illustrated by the doubled sensitivity value when transitioning from the default add-or-removv-one to the replace-onc notion, i.c., C' versus 2C with C denoting the gradient clipping bound. Consequently, a doubled noise scale is required to achieve ostensibly identical privacy guarantee, inevitably resulting in utility degradation and unfavorable comparison outcomes under the replace-one notion.
A nalysis, Insights, Implications. While this training paradigm enjoys several advantages, such as ease of implementation and representative features for characterizing the difference between distributions, several challenges persist when applying such a paradigm in practice. Firstly, the joint training of a generator alongside a critic, which typically necessitates an adversarial approach, is inherently unstable due to the difficulty in maintaining equilibrium between these two components. This instability can be further amplified by the incorporation of gradient clipping and noise addition operations introduced by DP-SGD, or the additional fitting process involved in transferring knowledge from the teacher discriminators to the student one through the PATE framework. Moreover, the DP training of the critic often impedes its convergence, resulting in a sub-optimal critic that may not effectively guide the generator.
Recent studies have investigated various strategies to alleviate these challenges, particularly in the context of GANs, These include warm-starting the GAN discriminator by pre-training on public data (Zhang al.) 2018), dynamically adjusting the gradient clipping bounds during the training process (Zhang et al.) 2018), re- ba-balanne balaacinig the discriminator and generalor updates to restore parily to discriminator weakened by DP noise Bie et a 2023), and exploiting public pre-trained GANs while restricting private modeling to the latent space (Chen et al. 2021). In the Private-Set (Chen et al. 2022a) framework that optimizes for downstream classification task, it is reported that optimizing the generator in a point-wise manner (as discussed in Section 3) or directly optimizing the synthetic set instead of the generator model can empirically lead to faster convergence and preferable when strong privacy guarantee required. In this regard, we anticipate promising outcomes from the future development of new variants of DP-compliant training pipelines and objectives that offer improved convergence and, consequently, enhanced privacy guarantees.
4.3 B3: Between Measurement and Synthetic Data
Threat Model. In response to challenges associated with training the DP critic (Section 4.2 recent studies have proposed shifting the privacy focus from the Measurement to the sanitization of the intermediate signal that backpropagates to update the generator, i.e., between Measurement and Synthetic data. The goal is to preserve the critic's s training stability and its utility for accurately comparing synthetic and real data, thereby guiding the generator's training effectively. This strategy ensures privacy when revealing sanitized intermediate gradients exchanged between the generator and the critic during the backward pass, as well as guarantees DP for the generator, which is updated with sanitized gradients. However, this scheme does not provide privacy guarantees for the release of the critics, since their training is conducted non-privately.
Similar to the case outlined in Section 4.2 the backbone generative models for this category are typically implicit density models. This restriction is in place as these models do not invoke direct interaction between the real data and the generator during the forward pass, which means that sanitizing the intermediate signals transmitted between the Measurement and Synthetic data is sufficient for ensuring
General Formulation.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Privacy barrier
publicly accessible publicly inaccessible
Synthetic data
Measurement
Generator Ge
Critic D Φ
2
Go
L
9G
M
up 9G
9D
DP sanitization
L Real data
data-independent
sensitive
DPD
The diagram illustrates the general training process of methods incorporating the privacy barrier B3. In the figure, 96 and 9G denote the upstream gradient (referenced as 9c sanitized variant, respectively. Note that variations exist in the formulation of critics and their corresponding training paradigins.
Figure 7:
upstream in Equation [2) and its
privacy protection. Methods in this category adhere to the gradient sanitization scheme, which introduces DP perturbation into the gradients communicated between the critic and generator during the backward pass. This can be formulated as follows:
9
= -cctt)(c(O(0)))) -oGGo(z)
(11)
9G M (o((t) JgGo(z)
(12)
upatronm
flocal
Here, LG represents the generator's loss (originating from a critic), and M denotes a potential DP sanitization mechanism on 9G be considered the gradient of the objective with respect to the current synthetic samples. It is important to note that the second term (Jlecl), ,, i.e., the local generator Jacobian, is computed independently of training data and thus docs not require sanitization. The generator is subsequently updated with the DP sanitized _1 Meanwhile, the critic, if learnable, updated normally (non-privately).
upstresm the gradient information backpropagating from the critic to the generator. This can
gradient, i... ((+11) ( I|G 9G A visual illustration is presented in Figure
Representative Methods. Existing methods explored various choices for the critic and different DP mechanisms to sanitize the upstream gradients 9G upstream GS-WGAN (Chen et al. 2020a) adopts the Gaussian mechanism for sanitization and capitalizes on the inherent bounding of the gradient norm. This follows from the Lipschitz property when employing the Wasserstein distance with gradient penalty (Arjovsky et al. ct al. gradients and allows iult teacher discrininator models to vole these discretive ized gradient values. The DP noisy argmax is then transferred to the generator. DataLens (Wang et al. 2021) further improves the signal-to-noise ratio in the PATE sanitization by employing top-K dimension compression.
2017 Gulrajani et al.] 2017) as the objective when training GAN. In contrast, G-PATE (Long
2021
incorporates the PATE framework as its sanitization mochanism. This approach discretizes the
In a different vein, DP-Sinkhorn (Cao et al. 2021) presents compelling results using a nonparametric critic. Specifically, -i-SSinhoon estimates the Sinkhorn divergence grounded on & and l losses in the data space, adhering to the distribution matching generative framework as depicted in Figure 3(b] This use of data-independent critic contributes stability to the training process and capitalizes the privacy enhancement. brought by subsampling.
Privacy Analysis. The privacy analysis for this method category largely aligns with the established unit sanitization mechanisms, denoted as M, which function on upstream gradients 90 upstream Nevertheless, specific attention necessary given that these intermediate gradients do not directly originate from real data samples. This scenario noticeably influences the sensitivity computation, defined formally by:
2 = max agstremm f(g/ upstream
D.D [ff
)||
(13)
16

---

shifting the privacy Published in Transactions on Machine Learning Research (04/2024)
In this equation, f encapsulates the operations required to set bounds on the sensitivity and to render the associated sanitization mechanism applicable. 9G upetreermm and 9G upstream symbolize the intermediate upstream gradients originating from neighboring datasets DD and D' respectively. Specifically, f performs distinct roles according to the method employod: For GS- ES-N and DP-Sinkhorn, f significs the operation of norm clipping; In G-PATE, I encompasses the processes of dimension reduction and gradient discretization, and the computation of teacher voting histograms basod on these discretized gradients; In the context of DataLens, rather than employing random projection and discretization as in G-PATE, f adopts top-k stochastic sign quantization of the gradients. Subsequent to this operation, the teacher voting histograms are also calculated.
A direct application of the triangle incquality reveals that A equals 2C' (with C' representing the gradient clipping bound) in both GS-WGAN and DP-Sinkhorn for both the replace-one and add-or-remove-one notions, while C is further guaranteed to be 1 in GS-WGAN by the nature of the adopted Wasserstein objective. This notably different from the substantial disparity between the two privacy notions in the standard DP-SGD framework. In G-PATE, the voting histogram diverges by a maximum of 2 entries for cach gradient dimension, which are processed independently via DP aggregation. As for the DataLens approch, the change of one data point will at most reverse all the signs of the top-k elements of gradients originated from one teacher model, leading to A = 2/k (See Appendix for details).
Notably, each Typically, the total privacy cost. is calculated based on the RDP accountant (Theorem 2.2 syuthetic sample in a mini-batch constitutes one execution of the sanitization mechanism for the DP-SGD framework, or one query in the PATE framework. In other words, performing an update step with mini-batch of syuthetic samples on the generator can be regarded as a composition of batch size times its unit sanitization mechanism.
Analysis, Insights, Implications. Compared to previous categories (Section 4.1 4.2).
barrier away from the Measurement process itself offers several benefits. These include: (1) the flexibility to employ a powerful critic, theroby effectively guiding the generator towards capturing the characteristics of the data distribution; (2) seamless support for different privacy notions (as discussed in privacy analysis above); (3) practically simpler to properly bound the sensitivity. This can be achieved by exploiting the intrinsic propertics of the objective (Chen et al. 2020a), or through the usage of the PATE framework (Long) et al. 2021 Wang et al., 2021). This is particularly beneficial when compared to the previous scenario of learnable crities that typically necessitate a laborious and fragile hyperparameter search for a reasonable gradient clipping bound.
However, the increased expressive capacity comes with the trade-off of relatively high privacy consumption. The accumulation of privacy cost across iterations is notably faster in this scenario than in standard DP-SGD training of single model: each DP update on the generator in this category equates to a batch size number of calls to the Gaussian mechanism, possibly without the advantage of subsampling, as detailed in the preceding privacy analysis section. This markedly contrasts with the standard DP-SGD training on single discriminator, as mentioned in the previous category (refer to Section 4.2 where cach individual DP gradient update equates to a single execution of the (subsampled) Gaussian mechanism.
Fortunately, this drawback has been partially mitigated through the use of data-dependent privacy analysis (as demonstrated in PaTE-based methods like G-PATE and DataLens) that provides analytically tighter results that lead to stronger DP guarantees, or dala-independent critic (as in DP-Sinkhorn) that offers smooth compatibility with subsampling and better convergence. Looking forward, we anticipate further developments from refining this training paradigm, particularly through the utilization of strong backbone discriminators (and generators) trained on external non-private data, thereby optimizing privacy consumption.
4.4 B4: Within Generator
Threat Model. DP can be directly integrated into the training or deployment of a generator, the minimal uiit within the generative models pipeline essential for maintaining the full generation functionality for future use. Generally, the privacy barrier safeguards against attackers who have access to the trained generator model while a more fine- ane-grained distinguishment between the type of access (e.g., white-box or black-box) may be required depending on the application scenarios and the adopted DP mechanism. If the gradient.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Privacy barrier publicly accessible ; publicly inaccessible
Real
Synthetic data
Encoder l 2 x
Generator Po a z)
2
ELBO
L
9
M 9
9o
M DP sanitization
go
Measurement
sensitive
DPP
DP sanitization
Diagram showcasing the DP training of a VAE (Section 3.2.2 This representation is also applicable to other DP model training scenarios conforming to privacy barrier B4, e.g., by replacing the trainable encoder with non-trainable module.
Figure 8:
sanitization scheme is adopted, can protect against adversaries who can access the white-box generator (and possibly other trainable components subject to DP sanitization) and the intermediate sanitized gradients during the whole training process.
General Formulation. In this context, the training pipeline can be generally simplified to the standard process of training DP classification models. This process, excmplified by the commonly used DP-SGD framework, entails bounding sensitivity through gradient clipping and subsequently injecting randomness into the generator's gradients. In contrast to calegory B3, where the upstream gradient 9G
upstreemm undergoes
sanitization, in this case, is the final generator gradient 9G (refer to Equation 11] that being sanitized. This results in difference equivalent the multiplicator of the local generator Jacobian (refer to Equation 12 Special attention should be paid when implementing DP-SGD here, as additional model components (e.g., the encoder in a VAE) alongside the generator could compromise the transparency of the privacy analysis. It is crucial to ensure that the gradient clipping operation is executed accurately to effectively limit each individual real sample's influence on the generator. The presence of an additional model component may disperse individual effects across multiple gradients within a mini-batch, rendering standard per-example gradient clipping inadequate (refer to the discussion in the privacy analysis below). Moreover, optimize model utility, it is necessary to precisely define the scope of gradient clipping and perturbation to ensure that the implementation does not introduce unnecessary noise exceeding the desired privacy guarantec.
Representative Methods. Existing works have realized such privacy barrier for various types of generative models, particularly those within the explicit density category. Examples include DP Normalizing
Flow (Waites Cummings)
2021 Jiang & Sun 2023 DP VAE (Chen et al. 2018 Acs al. 2018
Abay
et al. 2019 |
Takagi et al.
2021 Pfitzner & Arnrich 2022), DP Diffusion models (Dockhorn et al. 2022
Ghalebikesabl et al.J 2023
Lyu al.) 2023), and DP training of language models (McMahan al. 2018
Li
et al. 2022 Mattern et al. 2022 Yue al. 2023) which collectively illustrate the extensive potential of DP
generators across numerous applications such natural language generation, density estimation, high-qualic image generation, training downstream models, and model selection. In particular, Ghalebikesabi et al. 2023
highlighted that certain training techniques advantageous for DP classification models (De et al. 2022) such
as pre-training, utilization of large batch sizes, and augmentation multiplicity (oort et al. 2021 De et al. 2022), also show effectiveness when applied to training DP generators in diffusion models. Furthermore, the work by Jiang & Sun (2023) underscores the potential efficacy of training DP Flow model within a compressed, dower-dimensional latent space. This strategy not only circumvents the substantial computational demands (Rombach et al. 2022 but also synergizes well with DP protocols, given the direct correlation between the DP noise-to-signal-rati and the modcl's dimensionality.
Privacy Analysis. The privacy analysis follows from the adopted DP mechanism for training the generators, similar the standard case of training a DP classifier. key consideration lies the correct implementation and analysis of the privacy cost when the models comprise multiple trainable components, such as the encoder and decoder in the VAE. In such cases, simply incorporating the DP-SGD into the generator module
18

---

Published in Transactions on Machine Learning Research (04/2024)
and conducting standard privacy accountant is inappropriate. This is due to the fact that each training example's influence is assimilated into the encoder's parameters. Consequently, every training example, even those absent from the current mini-batch, can affect all latent variables (which serve as inputs to the generator/ docoder) in each iteration, rendering the per-cxample gradient clipping itself insufficient for bounding the sensitivily. A proper implementation would require either enforcing DP also on the encoder (i.e,, applying DP-SGD on both the encoder and decoder) or factoring this into the privacy cost computation (i.e., the DP-SGD step on the decoder should be counted as full batch Gaussian mechanism instead of a subsampled one). Moreover, in situations where each sample in mini-batch is used more than once, such their use over multiple time steps when training diffusion models, the cost must be accounted for every such occurrence. To deal with this, one can refer to the multiplicity technique (Fort et al. 2021 De et al. 2022; Ghalebikesabi et al. 2023 Dockhorn et al. 2022), which averages all gradients resulting from each unique training sample before clipping them.
Analysis, Insights, Implications. Methods in this category are generally easy to implement, particularly for models with only generator as the learnable component. This reduces training to the standard classification cases, demonstrating significant potential and achieving state-of-the-arr generation quality when adapted to the latest generative modeling techniques (Dockhorn et al. 2022 Ghalebilesabi et al.) 2023 However, this privacy barrier setting may not be fully compatible with models containing multiple trainable components. The reason for this lies in the potential integration of training samples' effects into the parameters of components other than the generator (e.g,, the encoder in VAEs, the discriminator in GANs), which substantially complicates the implementation of DP mechanisms and may lead to unexpectedly high privacy consumption. Moreover, DP methods are bounded by the expressive capability of the underlying generative model. Particularly in this category, which predominantly relies explicit density models, the usage of simple critics (like static & or & loss functions) tends to restrict the capture of fine details, often delivering less desirable outcomes compared to trainable critics. For instance, VAËs have commonly produced blurrier images, whereas GANs pionccred the production of high-rcsolution photorcalistic gencrations. While recent advancements in explicit density models have siguificantly improved their capabilities, particularly through innovative designs that enable training on extensive datasets, there is a potential limitation concerning their practical utility. This limitation primarily arises from the substantial need for sensitive training data, which is essential to achieve satisfactory performance level with the resulting DP model in real-world applications. Looking forward, we envision future advancemnnt on balancing the data efficiency and generation performance could largely improve the practicability of the DP methods under this category.
5 Discussion
5.1
Connection to Related Fields
While the data generation methods investigated in this work mostly designed to capture the entire data distribution for general purposes, intriguing results are observed when the generator is intentionally guided towards enhancing its downstream utility for specific target tasks such as training neural network classifiers (Chen et al., 2022a) and answering linear queries (Liu et al.] 2021]. This can be achieved by employing objectives tailored for downstream tasks, rather than relying solely on general distribution divergence measures. If downstream tasks can be executed on specific set of samples and do not require a complete understanding of the distribution, problem complexity can be further reduced by directly optimizing the synthetic samples instead of the generative models. This strategy, which trade-off the generality of general-purpost generative modeling for downstream utility, might be particularly beneficial considering the high complexity inherent to DP generation. Morcover, such framework naturally aligns with broader fields such as coreset generation, private query release, privale Bayesian inference. In these scenarios, a set of synthetic data can be optimized to resemble real data for specific tasks (Wang et al. 2018 Bachem et al., (207), substitute real data for answering queries to conserve the privacy budget under DP (Hardt & Rothblum, 2010 Hardt et al. 2012)) or support privacy-preserving computation of the posterior distribution (Manousakas
et al.
2020 Savitsky al. 2022).
19

---

Published in Transactions on Machine Learning Research (04/2024)
5.2 Relation to Other Summary Papers
Several related summary papers complement our work by focusing on different aspects. For instance, Tao
et al. 2021) benchmark multiple DP models for tabular data; Fan (2020) and Cai et al. (2021) discuss early DP GANs; Jordon et al. (2022) and De Cristofaro (2023) provide high-level overviews of DP synthetic data generation for non-experl audiences; Hu et al. (2023) covers broad classes of DP data generalion methods
without focusing on the technical part of deep generative modeling; Lastly, Ponomareva et al. (2023) offer a
comprchensive summary of developing and deploying general DP MI. models, supplementing our focus on the
technical aspects of DP generative modeling.
5.3 Challenges and Future Directions
Public Knowledge. A promising future direction which holds significant practical relevance is the exploitation of public data/k /knowledge in training DP generative models. Recent studies have demonstrated promising improvements in DP generation introduced by leveraging public data (Chen et al. 2021 Liu et al,
2021; Harder et al. 2022; Lyu et al. 2023) and reported high-quality generation (Ghalebikesabi et al. 2023 Lin et al., 2023) with the aid of such resources. A prevalent method for leveraging public knowledge involves utilizing large foundation models, initially pre-trained on public datasets, and subsequently fine-tuned to align with private data distributions for various applications. This approach is particularly relevant in the field of natural language processing (NLP), where the widespread availability of foundation models and the typically significant semantic overlap between public and private data renders DP fine-tuning relatively effective (Li et al. 2022). Additionally, the rapid growth of efficient fine-tuning techniques also show great potential for facilitating DP learning (Yu et al. 2021 Duan et al.) 2023). While these advancements are particularly notable in the NLP domain, exploring the specific benefits and most effective strategies for applying these techniques to other data modalities is a topic that warrants further research. Furthermore, challenges that. are generally associated with private learning on public data (Tramèr et al. 2022) call for further investigation. In particular, the unique difficulties specific to generative modeling, such as small tolerance for distribution shift between the public and private data distributions, warrant additional exploration.
Task-specific Generation. There exists a principled trade-off between the flexibility offered by general- purpose generative modeling and the utility of task-specific data generation. In particular, capturing complete high-dimensional data distribution is a difficult task. This task becomes even harder when considering the privacy constraints, thus making the models highly data-demanding and almost impossible for DP model to achieve reasonable performance in practice. It has also been recently questioned to what exlend a perf-performing general-purpse DP generative model can be realized at all (Stadler et al.) 2022 Stadler & Troncoso| 2022). While it is difficult to predict how these tradc-off develop in the future, task-specific (or lask-guided) data generation can greatly relax the objectives, leading to real-world useful DP synthetic data (see examples discussed in Section 5. On the other hand, such task-specific generation is particularly advantageous for scenarios where the synthetic data intentionally designed to be useful only for specific (benign) tasks, thereby preventing potential unauthorized data misuse.
Conditional Generation. While the formulas presented throughout Section ! are illustrated through unconditional generation for simplicity and clarity, in practice, DP generation is typically executed in a conditional manner, whereby samples are generated given specific input conditions. Although implementing conditional generation is technically straightforward for all generative network backbones (Mirza & Osindero 2014 Sohn et al. 20155: Odena et al. 2017 Winkler et al., 2019), it might necessitate additional consideration with respect to the privacy analysis. For instance, when modeling the class-conditional data feature distribution, an additional privacy budget may be allocated to learn the class label occurrence ratio for addressing class imbalance (Harder et al. 2021). contrasting with other methodologics that typically employ a dalla-independent uniform class-label distribution. Moreover, certain situations necessitate meticulous investigation into privacy implications and performance. Firstly, when the training process employs conditional (o.g., per-label class) sampling, additional consideration for privacy cost imperative, as this contradicts the requirements of random sub-sampling incorporated in standard privacy cost computations. Secondly, some generative modules may integrate such conditional information in non-trivial ways (e.g., being embedded into
20

---

Published in Transactions on Machine Learning Research (04/2024)
the module parameters beyond mere gradients (Karras et al. 2020 This integration can mean that the conditional input might no longer be protected under DP guarantees via a vanilla DP sanitization scheme. These scenarios necessitate further exploration to ensure the reliability of privacy protections and to facilitate the development of more effective utility-preserving DP generative models.
Federated Learning. DP data generation models have also shown promising potential in applications
related to federated training AAgensteii et al. 2019 Xin al. 2020 Zhang et al. 2021 Triasteyn & Faltings, 2020), facilitating tasks such privacy-preserving data inspection and debugging that were previously infeasible due to privacy constraints. Specifically, Augenstein et al. (2019) incorporated DP-SGD into the training of a GAN in & federated setting, where each client maintains a local GAN model and communicates the gradients to the server during each communication round, with the gradients being sanitized under DP noise. Moreover, Chen et al. (2020a) illustrated that the privacy barrier B3 (Section (.3) is seamlessly compatible with the federated training sctting. In this context, only the upstream gradient (Equation 12) needs to be communicated, offering additional benefits such improved communication efficiency. More recently, task-specific DP generation has proven particularly advantageous in alleviating non-iid challenges and enhancing convergence speed for federated learning (Xiong et al. 2023 Wang al.) 2023). Although these approaches might still require a substantial amount of client local data and computational resources, the future development of efficient. algorithms is anticipated to yield fruitful outcomes.
Evaluation and Auditing. Evaluating generative models has historically posed a significant challenge Luccie et al., 2018), ; and the same holds true for DP generation methods. While evaluating them based on specific downstream tasks has been a common approach in existing literature, it has become evident that relying solely on single metric may be inadequate. This limitation arises from the general lack of alignment among various aspects, including downstream utility, statistical properties, and visual appearance (Alaa et al.) 2022; Stadler & Troncoso 2022; Chen et al., 2022a Ganev et al., 2022). Consequently, there arises a need for future invostigations into comprehensive metrics that consider mixed objectives to more effectively address a wide range of polential practical applications.
Furthermore, assessing the privacy guarantees of DP generators against real-world attacks (i.e., "audit- ing" (Jagiclski et al., 2020 Nasr et al. )) and quantifying the privacy risk associated with synthetic dala (Stadler& & Troncoso, 2022 Houssiau el al. 2022 presents a particularly intricate challenge for generative models. This complexity primarily arises from two key factors. Firstly, the measurement of privacy risks often conflicts with the primary objective of maximuum likelihood, which aims to precisely fit the training data. While achieving an exact alignment with the training data aligns with training objectives, it raises a debatable question about compromising privacy protection. Deciding whether an exact match should be regarded as a privacy breach in such cases remains a matter of debate. Secondly, generative models typically exhibit low sensitivity to privacy attacks (Hayes et al. 2017 Chen et al. 2020b), which diminishes the informativeness of computed auditing scores. These challenges highlight the need for dedicated design tailored to the auditing of DP generative models.
6 Conclusion
In summarry we introduce unified view coupled with a novel taxonomy that effectively characterizes existing approaches in DP deep generative modeling. Our taxonomy, which encompasses critical aspects such as threat models, general formulation, detailed descriptions, privacy analysis, as well as insights and broader implications, provides a consolidated platform for systematically exploring potential innovative mothodologies while leveraging the strengths of existing techniques. Furthermore, we present a comprehensive introduction to the core principles of DP and generative modeling, accompanied by substantial insights and discussions regarding essential considerations for future rescarch in this arca.
Acknowledgements
This work is partially funded by the Helmholtz Association within the project "Privatsphäären-cchitzzee Analytik in der Medizin" (PrivateAIM, 01772316G), the BMBF with the project "Reprasentative, synthetische
21

---

Published in Transactions on Machine Learning Research (04/2024)
Gesundheitsdaten mit starken Privatspharengarantien" (PriSyn, 16KISA029K) and ELSA European Lighthouse on Secure and Safe AT, funded by the European Union under grant agreement No. 101070617. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or European Commission. Neither the Europcan Union nor the European Commission can be held responsible for them. Dinglan Chen acknowledges the funding from the Qualcomm Innovalion Fellowship Europe. We acknowledge Matthias Reisser for providing valuable comments on this work.
References
Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang, Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2016.
Nazmiye Ceren Abay, Yan Zhou, Murat Kantarcioglu, Bhavani Thuraisingham, and Latanya Sweeney. Privacy preserving synthetic data release using deep learning. In Machine Learning and Knowledge Discovery in Dalabases: European Conference, ECML PKDD 2018. Springer, 2019.
Gergely Acs, Luca Melis, Claude Castelluccia, and Emiliano De Cristofaro. Differentially privale mixture of generative neural networks. IEEE Transactions on Knowledge and Data Engineering, 31(6), 2018
Ahmed Alaa, Boris Van Breugel, Evgeny $ Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-lovel metrics for evaluating and auditing generative models. In International Conference on Machine Learning (ICML). PMLR, 2022.
Moustafa Alzantot and Mani Srivastava. Differential Privacy Synthetic Data Generation using WGANs, 2019.
URL https://githuu. www./ooooo/nnnneeeeeeee htwp:///iii.c.clecooraaaaaaaaaar
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning (ICML). PMLR, 2017.
Scan Augenstein, H Brendan McMahan, Daniel Ramage, Swaroop Ramaswamy, Peter Kairouz, Mingqing Chen, Rajiv Mathews, and Blaise Aguera Arcas. Generative models for effective ml on private, recentralized datasets. In International Conference on Learning Representations (ICLR), 2019.
Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXiv: 1xxx:1108.000 701.7003..00 06476, 2017.
Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis testing interpretations and renyi differential privacy. In International Conference on Artificial Intellggence and Statistics, pp. 2496-2506. PMLR, 2020.
Brett K Beaulieu-Jones, Zhiwei Steven Wu, Chris Williams, and Casey Greene, Srrrucypreseevinnggeee generative deep neural networks support clinical data sharing. biorxiv. DOI, 2017.
Alex Bie, Gautam Kamath, and Guojun Zhang. Private gans, revisited. arXiv preprint arXic:2302.02936, February 2023.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. Tn International Conference on Learning Representations (TCLR), 2018.
Avrim Blum, Katrina Ligett, and Aaron Roth. A learning theory approach to noninteractive database privacy. Journal of the ACM (JACM), 2013.
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasserstein barycenters measures. Journal of Mathematical Imaging and Vision, 51, 2015.
Zhipeng Cai, Zuobin Xiong, Honghui Xu, Peng Wang, Wei Li, and Yi Pan. Generative adversarial networks: A survey toward private and secure applications. ACM Computing Surveys (CSUR), 54(6):1-33 2021.
22

---

Published in Transactions on Machine Learning Research (04/2024)
Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, and Karsten Kreis. Don't generate me: Training differentially private generative models with sinkhorn divergence. Advonces in Neural Information Processing Systems (NewrIPS), 2021.
Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. Gs-wgan: A gradient-sanitizee approach for learning differentially private generators. In Neural Information Processing Systems (NeurIPS). 2020a.
Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: taxonomy membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Securily (CCS), 2020b.
Dinglan Chen, Raouf Kerkouche, and Mario Fritz. Privale set generation with discriminative information. In Advances in Neural Information Processing Systems (NeurIPS), 2022a.
Dongjie Chen, Sen-ching Samson Cheung, Chen-Nee Chuah, and Sally Ozonolf. Differentially private generalive adversarial networks with model inversion. In 2021 IEEE International Workshop on Information Forensics and Security (WIFS). IEEE, 2021.
Jia-We Wei Chen, Chia-Mu Yu, Ching-Chia Kao, Tzai-Wei Pang, and Chun-Shien Lu. Dpgen: Differentially private generative energy-guided network for natural image synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition (CVPR), 2022b.
Qingrong Chen, Chong Xiang, Minhui Xue, Bo Li, Nikita Borisov, Dali Kaarfar, and Haojin Zhu. Differentially private data generative models. arXic preprint arXiv: 1att.18122.2 2018.
Soham De, Leonard Berrada, Jamie Hayes, Samuel Smith, and Borja Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.138 13650, 2022.
Emiliano De Cristofaro. What is synthetic data? the good, the bad, and the ugly. arXiv preprint WrXiu:233.012200 2023.
Tim Dockhorn, Tianshi Cao, Arash Vahdat, and Karsten Kreis. Differentially private diffusion models. arXiv preprint ar arXi::2210.099 09929, 2022.
Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: Differentially private prompt learning for large language models. arXiv preprint urxi:23305. 0...29955.1.,, 2023.
Cynthia Dwork. Differential privacy: A survey of results. In International Conference on Theory and Applications of Models of Computation (TAMC). Springer, 2008.
Cynthia Dwork, Moni Naor, Omer Reingold, Guy N Rothblum, and Salil Vadhan. On the complexity of differentially private data release: efficient algorithms and hardness results. In Proceedings of the forty-first anmual ACM symposium on Theory of computing (STOC), 2009.
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations of differential privacy. Foundations and Trendst in Theorelical Compuler Science, 9(3 4), 2014.
Liyue Fan. A survey of differentially private generative adversarial networks. In The AAAI Workshop on Privacy-Preseruvs Artificial Intelligence, pp. 8, 2020.
Stanislav Fort, Andrew Brock, Razvan Pascanu, Soham De, and Samuel L Smith. Drawing multiple augmen- tation samples per image during training efficiently decreases test error. arXiv preprint arXiv:2105.1833 2021.
Lorenzo Frigerio, Anderson Santana de Oliveira, Laurent Gomez, and Patrick Duverger. Differentially private generative adversarial networks for time series, continuous, and discrete open data. In IFIP International Conference on ICT Systems Security and Privacy Protection. Springer, 2019.
Benjamin CM Fung, Ke Wang, Rui Chen, and Philip S Yu. Privacy-preservin data publishing: A survey of recent. developments. ACM Compuling Surveys (Csur), 2010.
23

---

Published in Transactions on Machine Learning Research (04/2024)
Georgi Ganev, Bristena Oprisamu, and Emiliano De Cristofaro. Robin hood and matthew effects: Differential privacy has disparate impact on synthetic data. In International Conference on Machine Learning (ICL.).. PMLR, 2022.
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn divergences. In International Conference on Artificial Intelligence and Statistics. PMLR, 2018.
Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Ira Ktena, Robert Stanforth, Jamie Hayes, Soham De, Samuel L Smith, Olivia Wiles, and Borja Balle. Differentially private diffusion models generate useful synthetic images. arXiv preprint arXiv:200. 2023.
lan Goodfellow, Jean Pouget-Abadiee Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems (Ne,rIPS), 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems (NewrIPS), 30, 2017.
Frederik Harder, Kamil Adamezewski, and Mijung Park. Dp-merf: Differentially private mean embeddings with randomfeatures for practical privacy-rreservng data generation. In International conference on artificial intelligence and statistics (AISTAT). PMLR, 2021.
Fredrik Harder, Milad Jalali Asadabadi, Danica J Sutherland, and Mijung Park. Differentially private data generation needs better features. Xrviv preprint arXiv:2205.12900 2022.
Moritz Hardt and Guy N Rothblum. A multiplicative weights mechanism for privacy-preservin data analysis. In IEEE 51st Annual Symposium on Foundations of Computer Science (FOCS), 2010.
Moritz Hardt, Katrina Ligett, and Frank McSherry. A simple and practical algorithm for differentially private data release. Advances in neural information processing systems (NeurIPS), 25, 2012.
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. arXiv preprint arXiv: 706.07633, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Florimond Houssiau, James Jordon, Samuel N Cohen, Owen Daniel, Andrew Elliott, James Geddes, Callum Mole, Camila Rangel-Smith, and Lukasz Szpruch. Tapas: a toolbox for adversarial privacy auditing of synthetic data. arxie preprint orXhv:2211.06550, 2022.
Yuzhong Hu, Fan Wu, Qinbin Li, Yunhui Long, Gonzalo Munilla Garrido, Chang Ge, Bolin Ding, David Forsyth, Bo Li, and Dawn Song. Sok: Privacy-preserrve data synthesis. arXiv preprinl ar xxXim22307.02000 2023.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Dihong Jiang and Sun Sun. Dp-Iflow: Differentially private latent flow for scalable sensitive image generation. ICML DeployableGenenntivccl Workshop, 2023.
Dihong Jiang. Guojun Zhang, Mahdi Karami, Xi Chen, Yunfeng Shao, and Yaoliang Yu. Dp2-vae: Differentially private pre-trained variational autoencoders. arXiv preprint arXiv:2208. 03409, 2022.
James Jordon, Lukasz Szpruch, Florimond Houssian, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N Cohen, and Adrian Weller. Synthetic data-what, why and how? arXiv preprint 2XXx:2255.0225 2022.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition (CVPR), 2020.
24

---

Published in Transactions on Machine Learning Research (04/2024)
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations (TCLR), 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Advances in neural information processing systems (NeurIPS), 31, 2018.
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Proceedings of the fourleenth international conference on artificial intelligence and statislies. JMLR Workshop and Conference Proceedings, 2011.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems (NeurIPS), 30, 2017.
Xuechen Li, Florian Tramer, Percy Liang and Tatsunori Hashimoto. Large language models can be strong diffcrentially private learners. In International Conference on Learning Representations (ICLR), 2022.
Seng Pei Liew, Tsubasa Takahashi, and Michihíko Ueno. Pearl: Data synthesis via private embeddings and adversarial reconstruction learning. arXiv preprinl Xrx: 04590, 2021.
Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, Harsha Nori, and Sergey Yekhanin. Differentially private synthetic data via foundation model apis 1: Images. arXie preprint arX:::2305. 15560, 2023.
Terrance Liu, Giuseppe Victri, and Steven Z Wu. Iterative methods for private synthetic data: Unifying framework and new methods. Advances in Neural Information Processing Systems (NewrIPS), 34, 2021.
Yumhui Long, Boxin Wang, Zhuolin Yang, Bhavya Kailkhura, Aston Zhang, Carl Gunter, and Bo Li. G-pate: Scalable differentially private data generator via private aggregation teacher discriminators. Advances in Newral Information Processing Systems (NeurIPS), 34, 2021.
Mario Lucie, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal?
a large-scale study. Advances in neural information processing systems (NeurIPS), 31, 2018.
Saiyue Lyu, Margarita Vinaroz, Michael F Liu, and Mijung Park. Differentially private latent diffusion models. arXiv preprint XrXiv:2005. 15759, 2023.
Dionysis Manousakas, Zuheng Xu, Cecilia Mascolo, and Trevor Campbell. Bayesian pseudocoresets. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, and Mrinmaya Sachan. Differen- tially private language models for secure data sharing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations (ICLR), 2018
Tya Mironov. Rényi differential privacy. In IEFE 30th Computer Security Foundations Symposium (CSF), 2017.
Lya Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism. arXiv preprint 19X.i 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. orXio preprint urXim:1411.198 2014.
Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tramèr, Matthew Jagielski, Nicholas Carlini, and Andreas Terzis. Tight auditing of differentially private machine learning. ar preprint (rXXi:2302.07956 2023.
25

---

Published in Transactions on Machine Learning Research (04/2024)
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems (NeurIPS), 29, 2016.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning (ICML). PMLR, 2017.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arv preprint Xrxi:11609.03. 03499, 2016.
Nicolas Papernot, Martín Abadi, Ülfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. In International Conference on Learning Representations (ICLR), 2017.
Nicolas Papernot, Shuang Song, llya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with pate. In International Conference on Learning Representations (ICLR), 2018.
Bjarne Pfitzner and Bert Arnrich. Dpd-fvae: Synthetic data generation using federated variational autoen- coders with differentially-privaal decoder. arXiv preprint 2rXiv:2211.11500 2022.
Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradecp Thakurta. How to dp-fy ml: practical guide to machine learning with differential privacy. arXiv preprint ar Xix:22033..m Xiw:2303.00654, 2023.
Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and its application to texture mixing. In Scale Space and Variational Methods in Computer Vision: Third International Conference, SSVM 2011. Springer, 2012.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems (NeurIPS), 20, 2007.
Alain Rakotomamonjy and Ralaivola Liva. Differentially private sliced wasserstein distance. In International Conference on Machine Learning (ICML). PMLR, 2021.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference machine learning (ICML). PMLR, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. reshutosolution image synthesis with latent diffusson models. In Proceedings of the IEEE/CVF conference on computer vision and pallern recoguition (CVPR), 2022.
Aaron Roth and Tim Roughgarden. Interactive privacy via the median mechanism. In Proceedings of the forby-second ACM symposium on Theory of computing (STOC), 2010.
Terrance Savitsky, Matthew Williams, and Jingchen Hu. Bayesian pseudo posterior mechanism under asymptotic differential privacy. J. Mach. Learn. Res. (JMLR), 23, 2022.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprinl Xrxi 00810, 2017.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML). PMLR, 2015.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems (NeurIPS), 28, 2015.
26

---

Published in Transactions on Machine Learning Research (04/2024)
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Aduances in neural informalion processing systems (NewrIPS), 32, 2019.
Theresa Stadler and Carmela Troncoso. Why the search for failing, Nature Computational Science, 2(4):208-211) 2022.
data sharing mechanism is
Theresa Stadler, Bristena Oprisanu, and Carmela Troncoso. Synthetic data anonymisation groundhog day. In USENIX Security Symposium, 2022.
Shun Takagi, Tsubasa Takahashi, Yang Cao, and Masatoshi Yoshikawa. P3gm: Private high-dimensionss data
release via privacy preserving phased generative model. In International Conference on Data Engineering (ICDE). IEEE, 2021.
Uthaipon Tao Tantipongpipat, Chris Waites, Digvijay Boob, Amaresh Ankit Siva, and Rachel Cummings. Differentially private synthetic mixed-lype data generation for unsupervised learning. Intelligent Decision Technologies, 15(4), 2021.
Yuchao Tao, Ryan McKenna, Michael Hay, Ashwin Machanavajlhala, and Gerome Miklau. Benchmarking differentially private synthetic data generation algorithms. arXiv preprint arXin:2112 12.0281,,,99,,,, 2021.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information boutlneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, 1999.
Reihaneh Torkzadehmmhanni, Peter Kairouz, and Benedict Paten. Dp-ogan: Differentially private synthetic data and label generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.
Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In International Conference on Learning Representations (ICLR), 2021.
Florian Tramèr, Gautam Kamath, and Nicholas Carlini. Considerations for differentially private learning with large-scale public pretraining. ar preprinl arXiv:2212.064 064700, 2022.
Aleksei Triastcyn and Boi Faltings. Generating artificial data for private deep learning. ar preprint SFf:::1800001448 2018.
Aleksei Triastcyn and Boi Faltings. Federated generative privacy. IEEE Intelligent Systems, 35(4):50 57, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neurul information processing systems (NeurIPS), 29, 2016.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukeuoglu. Pixel recurrent neural networks. In International conference on machine learning (ICML). PMTR, 2016.
Giuseppe Vietri, Grace Tian, Mark Bun, Thomas Steinke, and Steven Wu. New oracle-efficient algorithms for private synthetic data release. In International Conference on Machine Learning (ICML). PMLR, 2020.
Margarita Vinaroz, Mohammad-Amin Charusaie, Frederik Harder, Kamil Adamczewski, and Mi Jung Park. Hermite polynomial features for private data generation. Tn International Conference on Machine Learning (ICML). PMLR, 2022.
Chris Waites and Rachel Cummings. Differentially private normalizing flows for privacy-preserving density estimation. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021.
Boxin Wang, Fan Wu, Yunhui Long, Luka Rimanic, Ce Zhang, and Bo Li. Datalens: Scalable privacy preserving training via gradient compression and aggregation. In Proceedings of the 2021 ACM SIGSAC Conference Computer and Communications Security (CCS), 2021.
Hui-Po Wang, Dingfan Chen, Raouf Kerkouche, and Mario Fritz. Fedlap-dp: Federated learning by sharing differentially private loss approximations. Xi preprint arXi::2302.0100 2023.
27

---

Published in Transactions on Machine Learning Research (04/2024)
Tongzhou Wang. Jun- Yan Zhu, Antonio Torralba, and Alexci A Efros. Dataset distillation. arXiv preprint 1XX.:181111095, 2018.
Christina Winkler, Daniel Worrall, Emiel Hoogeboom, and Max Welling. Learning likelihoods with conditional normalizing flows. arXiv preprint arXiv: 91 200042, 2019.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint 1802: 067,06699, 2018.
Bangzhou Xin, Wei Yang, Yangyang Geng, Sheng Chen, Shaowei Wang, and Liusheng Huang- Private fl-gan: Differential privacy synthetic data generation based on federated learning, In JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020. doi: 10.1 /t//://CASSP007..222000.00
Yuanhao Xiong, Ruochen Wang, Minhao Cheng, Felix Yu, and Cho-Jui Hsich. Feddm: Iterative distribution matching for communicaiion-elliieni federated learning. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Chugui Xu, Ju Ren, Deyu Zhang, Yaoxue Zhang, Zhan Qin, and Kui Ren. Ganobfuscator: Mitigating information leakage under gan via differential privacy. IEEE Transactions Information Forensics and Securily, 14(9), 2019.
Yilin Yang, Kamil Adamczewski, Danica Sutherland, Xiaoxiao Li, and Mijung Park. Difterentially private neural tangent kernels for
data generation. arXiv preprint orXiv:2303 016687, 2023.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. PATE-GAN: Generaling synthetic data with differential privacy guarantees. In International Conference on Learning Representations (ICLR), 2019.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language 2021.
models. arXiv preprint arXiv:2110.0653
Xiang Yuc, Huscyin A Inan, Xucchen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim. Synthetic text generation with differential privacy: A simple and practical recipe. In The 61 Annual Meeting Of The Association For Computational Länguistics, 2023.
Longling Zhang, Bochen Shen, Ahmed Barnawi, Shan Xi, Neeraj Kumar, and Yi Wu. Feddpgan: federated differentially private generative adversarial networks framework for the detection of covid-19 pneumonia. Information Systems Frontiers, 23(())1403-144) 2021.
Xinyang Zhang, Shouling Ji, and Ting Wang. Differentially private releasing via deep generative model (technical report). arXi preprint arXiv: 1011:1080,,1194, 2018.
28

---

Published in Transactions on Machine Learning Research (04/2024)
Appendix
A
Summary of Existing Works
Approach
Privncy barrier
Privacy
Senuitivity
Generative
DP
Code
notion
type
framewwork
framework
DP-Merf Harder aL.
2021
B1
Replace-one
Global
Distribution mutching
Gaussion
DP-SWD
Liva 2021
B1
Replace-one
Smooth
Distribution matching
Gaussian
Replace-one
Global
Distribution mustching
DP-HP
mnator
2122
B1
Replace-oue
Globel
Distribution matching
Gaussian
DP-GEN
(Ceon et
21226
BB1
Energy based model
Harder
B1
Replace-oue
Globel
DistribuLion matching
Gaussiau
DP-NIK
Yang eE ml
2022
B1
Replace-one
Global
Distribution matching
Gaussian
DPSDA [Li oi M)
X123
B1
Addd-or-remove-oo
Globel
Difuusoon
Gausslan
SPRINT- pan
(Heat Jims-Jomes
2017
B2
Global
GAN
DP-SGD
dp-GAN
Zhang et nu. 2118
B2
Add ondnnvv--
rennove-one
Global
GAN
DP.SGD
DPGAN
al. [2TN
Triaseyn
B2
GAN
empirical DP
CANE-AAN Yoom et
2119
B2
Both
Local
GAN
PATE
Srivasiava
venove-one
Global
Xn at 219
B22
Add cww.oor--reeeeeeee one
Global
GAN
DP-SGD
DP-8eR
cocaaeemmabaui et al 2019
B2
Global
GAN
DP-SGD
Frigerio
B32
Add-on
Global
GAN
DP-SGD
DPSI CCitnn et al
2021
B2
aid-oor-emoove-ou
Global
GAN
DF-SGD
DPantoGAANN Tannppotoppput
remoove-oue
Global
Private-Sel (C Cee
A7224
B2
Add-or-renove-oue
Globel
Distribution matching
DP-SGD
Bie et al. 22/z3]
B2
Global
GAN
DP-SGD
GS:WGAN [Chen
B3
Both
Global
GAN
DESGD
G.PATE Long of al. 21
F3
Both
Tocal
GAN
PATE
DuraLeas Waang of n.. 20121
B33
Both
Glaba//Local
GAN
PATE
DP-Sinkhor
B3
Both
Globel
Distribution mutching
DP-SGI
DP-CM Ars 61 m 2018
F14
Add-or Add-or-remov--om
Globel
VAE
DP-SGID
DP-VaeGM CCee el
2018
B4
nedeorr-emowe-one
Globel
VAE.
DP-SGD
DP-SYN Abay
20119
B4
hid-or-remoov--oee
Globel
AR"
DP-SGD
P3GM Takagl es ml
202
B4
aid-oo-remove-oe
Globel
VAE
DP-SGD
DP-NF
Chamminng
ald-or-remoww-one
Global
DP-SCD
DP'-VAB Jining al
B4
Add-on Add-or-remove-one
Globel
VAE
DP-SGD
DPDM Dorkhorn ct 2023
B4
aid-orr-eonow-ee
Clobal
Diffusion
DP-SCD
Ghalebile
B4
Add-or-
Globel
Diffusion
DP-SGI
DISLDM
Lyn
1
B4
ord-or or remnove-one
Global
Diffission
DP-SGD
Jaan &
Add-or-reoow--om
Flow
DP-SGD
Table 1: Table summary of existing works most relevant to our contributions. The shaded area corresponds
approaches that require public data features.
https:
attps: httpss//ggtaabbcc///rkkkooo////
https: htttae//gghaubb..o//gggll///////
https: /ttpa///////aaa..o//////////aaaa
attps: http:://tthbb..c///ccooooosssst
https///jithbb.coom/rree http:///ghab..coo//zeeeal///111
10
https:
11
12
13
14
attps: https: /tt
15
16
/ttp:://gttaub.coon http:////////baoooooogaaaaaaaaaa
17
attps: /ttp:////graab.cc////aooooo/r//e
https
bttps:
25
https///tthubb.co///yyyaa/////
28
29

---

Published in Transactions on Machine Learning Research (04/2024)
Note that the DP training of language models typically falls within DP generative modeling and can be categorized as the "within generator" category in our taxonomy. However, the training strategies adopted are more akin to the DP training of general (discriminative) models, which is not the primary focus of our work. Hence, these approaches are not exhaustively mentioned in the table above.
B Additional Notes on Potential Methods with Privacy Barrier B1
In the DP deep generative modeling literature, existing approaches with privacy barrier between Real data and Measurement (Section 4.1 typically release sanitized features a condensed and aggregated form. In this sense, recent approaches, which may deviate from the general "mean embedding" formulation (as shown Equation but still publish a sanitized statistical summary of the private dataset, such as DPSDA (Lin et al. 2023 fall into this category. Specifically, DPSDA sanitizes count histogram that summarizes the distribution of real data and employs it as measurement to refine the synthetic data distribution, thereby rendering it more similar to the real private data distribution.
However, one might wonder if it is feasible to release a DP database in the original form of the real data, prior to the training of a generative model. A positive example of this idea can found in the Small Database Mochanism (SmalIDB) in the context of private query relcase, introduced Section 4.1 of Dwork al. (2014). This mechanism outputs sanitized database in the same form the original data, by selecting the database (from all possible sets of the data universe) via the exponential mechanism with utility function of the negative error to the query release problem (difference in the query answer the synthetic versus the real database). However, as the name suggests, the use of such an algorithm is largely limited to small (low-dimensional) datasets. This is mainly due to the exponential growth of the data universe with dimensionality, which drastically increases the computational burden and undermines the accuracy guarantees.
While DP-GEN (Chen et al. 2022b) attempted to apply similar idea to deep generative models, the output space of their generation method only supports (has non-zero probability) combinations of its input private dataset (See detailed proofs in Appendix B of Dockhorn et al. (2022)), ) instead of the entire data universe. This invalidates their claimed privacy guarantee, and the performance of a proper implementation of such a "direct database release" approach on high-dimensional data remains unclear.
C Additional Sensitivity Analysis
C.1 Privacy barrier B1
Sensitivity of DP-Merf (Harder et al., 2021 and the General Formulation in Section 4.1
It can /. where m = |D] represents the size of
be clearly seen that the fz-sensitivity for the replace-one notion is 2 the private dataset, as demonstrated in the original paper. Subsequently, we proceed to derive a conservative bound for the sensitivity value in the DP-Merf method under the add-or-remove-one DP notion, which can be generalized other approaches within the same category (Section 4.1. including Liew al. (2021): Vinaroz
et al.
2022 Harder et al. (2022): Yang et al. 2023 For the add-one case, we let D and assume,
30

---

Published in Transactions on Machine Learning Research (04/2024)
without loss of generality, that D' U (mm+1) J and x; æ; for all i 1,...
m+1
A max
(ax))
1
Σ
<(x;)))
D,D'
m
m
f=1
1=1
1
max
(0(2)
m++1) 11) + A)
A
#m+1:A A
m
m
2
1
max
A
a
(m + 1)m
m +
1
A
max
+ max
(m + 1)m
2
x' m-1
m +
1
m
(m + 1)m
17
+
m + 1
where A = Ex=1 o(x;) for brevity. The inequalities follow from the triangle inequality and the fact that (12(-))l2 =1
Similarly, for the remove-one case, we let m = [D], D' {mm] = and at = a, for all i 1, ... m
1 Σ
Σ Maxi /|2
A² = m18
((xr))
D.D'
1=1
1
(A + (( 1
max Em.A
m71
m
1
max
A
o (
Rm
(m-
m
1
max A
A
+ max
2 Imm)
(m 1)m
2
2m
m7
1
2
(m 1)
+
(m 1)m
with A= Em
m- ((x:). The inequalities follow from the triangle inequality and the fact that [lo(-))l2 =
Sensitivity of DP-SWD [akootomamonjy & Liva 2021). The sensitivity is calculated as the maximum difference over two embeddings, determined after performing random projections on two neighboring datasets. The 'replace-one' notion is adopted to simplify the analysis. With probability of at least 1 - 6, it can be shown that:
XU -XU < w(k
with w(k, = + 1nn++3 2 242 J. Here X' denote data matrices in R DD for neighboring datasets 0)
D, D' under the bounded-DP notion, while U E ædxt represents the random projection matrix with each column independently drawn from gd-1, Additionally, it is ensured that | X!,lla < 1 for all i by pre- rreppoccessing the dataset, making each sample record have unit norm. To prove the desired result, the sensitivity first transformed into a summation of l i.i.d random variables following the beta distribution B(1/2, (d - 1)/2), which then allows the application of Bernstein's is inequality to establish concentration bounds for the summation. For more detailed proof, please refer to Appendix 8.1-8.2 in Rakotomamonjy & Liva (201)) 1.
Sensitivity of DPSDA (Lin et al., 2023). The core component of DPSDA is the method of constructing a nearest neighbors histogram that describes the real data distribution while providing DP guarantees (refer Algorithm 2 in Lin et al. (2023)). Specifically, for every real sample æ, in the private datasct DD the algorithm identifies its nearest synthetic counterparts and constructs a histogram. This histogram represents the frequency of each existing synthetic sample 8 being the closest to the real samples. Given synthetic
31

---

Published in Transactions on Machine Learning Research (04/2024)
dataset consisting of samples (sx FE= and let [D]:
h2 = i [m],j arg min d(x;; 8k)] for = I,..,
kE[n]
where h = (I,,,.., hp) builds up the histogram with each hy reflecting the number real samples for which the corresponding synthetic sample 8j is the nearest neighbor, based on the distance metric d. Subsequently, DP Gaussian noise added to the histogram for providing privacy guarantees: h h + o0,
For the add-or-remove-one notion, D' U [ = D (or D' = (uffmm) Let s, be the closest synthetic sample to 2m and h, ht represent the histograms on D and DD' respectively. The fg-sensitivity then given by:
assume that w.l.o.g. the neighboring datasets D, satisly
D,D A max [1,1, ... hm) (1/ .. N2)ll2
max II(O.... 0, h2 n1 .,...... 0)
hg.ht
DU D with For the replace-one notion, we define neighboring datasets D, D' to satisfy D' U {am}
Um + z'm. The fz-sensitivity is defined by:
A max [I( ... (....,,,))).. Am)) - (16, ... NnDll2
D,D
max by hi.
[100 ..., 0, h1 - 1t.. ..., 0, hs - M4, 0,..,,.0) 0)
V 12 -2- V2
where 8y and sy are the closet syuthetic samples to and a'm respectively, while w.l.o.g. j < k.
C.2 Privacy barrier B2
The sensitivity analysis for methods in this category inherits the approach used in the DP-SGD and the PATE framework, which is presented below.
Sensitivity of DP-SGD (Section 2.1.1 . The main component the DP-SGD algorithm can be formalized as follows:
Clip: (x,)) g max (1. [I9(())))2))
1 Σ (x:)) + +N(0, Add noise: ge + B
where (x,) Vo, denotes the gradient on sample at iteration t, C represents the clipping bound, B is the batch size, is the noise scale, and the summation is taken over all samples in the batch. The sensitivity in DP-SGD is computed as:
2 mnax
g.. - (x()))
D,D
For the add-or-remove-one DP notion, let D', D only differ in the existence of æ/, i.c., D' = DDU (z/), it is easy to see that
2 = max
||g (x)) c
For the replace-one DP notion, w.l.o.g. let U [ai} = D U {z.}. thus
2
max
/ (m:)) g.(a S 2C
.i..;
32

---

Published in Transactions on Machine Learning Research (04/2024)
due to the triangle inequality.
Sensitivity of PATE (Section 2.1.2). Given m teachers, e possible label classes and an input vector æ, the "votes" of teachers that assign class j to a query input z is denoted as:
m j [m)) f((
5 for j ,,,,
with Jj denotes the i-th teacher model. And the histogram of the teachers' vote histogram is:
nxx) (M1, me) E N
As each training data sample only influences single teacher due to the disjoint partitioning, changing one data sample in the training dataset- Sheaseetwwhethertt whether it's removal, addition, replacement will at most alter the votes (by 1) for two classes, denoted here as classes and j, on any possible query sample æ. Let the vote histograms resulting from neighboring datasets D, D' (M1.".., mee) and (ni,,-. "nn') respectively, the global sensitivity can be represented as:
willlllltttwl
A = max (111, ... ,n)) - (n1, ... n'))l1
D,D
max (l.,, O, - -n²; 0, 0, n'z. ..... 0)]l
Mt !/
max (4 n²l max nj:m
|nj n'yl 2
A
max (100, .... 0, -n', 0, ...., 0, nj nlj, .,.......000 0)
my
(n: - 2 + (ng - n2)² s v2
max MamR
This holds for all possible query samples a.
The f1- and fz-sensitivities calibrate the two variants of noise mechanisms used PATE: the Gaussian NoisyMax (GNMax) and the max-of-Laplacian (LNMax). The GNMax defined as:
PATE, = arg max(ny - jE(d]
and the LNMax as:
PATE, = arg (nx + Lap(1/7))
sE(c]
C.3 Privacy barrier B3
Sensitivity of GS- CS-WGAN (Chen et al.) 2020a) and DP-Sinkhorn (Cao et al. 2021) The sensitivity for both GS- WGAN and DP-Sinkhorn can be derived via triangle inequality:
A max [f D,D'
upstream f(g'
upstremm
l|l
< max [[ upstream ||2 + max D
([ D
upstream I||
2C
with ff denoting the gradient clipping operation and C the clipping bound. Notably, no matter which privacy notion is used, both terms (maxp [ upsatrem )|| and maxp [g upstream [||)) are upper-bounded by the gradient clipping bound C.
33

---

Published in Transactions on Machine Learning Research (04/2024)
Sensitivity of DataLens (Wang et al., 2021). Given teachers, the d-dimensional gradients yielded from each teacher after applying top-k sign quantization take the following form (refer to Algorithm 2 in Wang et al. (0211)):
9. E (0,1, with 11 = and 11:112 = vk
In other words, 9i contains exactly non-zero elements, with the non-zero elements taking values of either -1, depending on the sign of the original upstream gradient.
Consider gradient sets (á:) and (0 which originate from neighboring datasets D and D' respectively. As the influence of each data point is limited to single teacher model, these gradient sets differ by at most one element. Without loss of generality, let's assume they diverge in the i-th element. The ég-sensitivity is then computed as follows:
TR
m
A maxx D,D/
Σ i=1
9
g1
1=1
max
|
9 ll
9.8,
S || la + 111l12 2
C.4
Privacy barrier B4
The sensitivity analysis for methods in this category adheres to the DP-SGD framework. While special considerations may be required to ensure the implementation correctly adheres to this framework, these considerations typically do not alter the sensitivity analysis itself.
D
Additional Background on Privacy Cost Accumulation
Theorem 2.2 [presented in Section provides a straightforward method for calculating the aggregated privacy cost. when composing multiple (potentially heterogeneous) DP mechanisms. In this section, we present more details regarding determining the accumulated privacy cost over multiple executions of sampled Gaussion mechanisms (Definition D.I..
Definition D.1 (Sampled Gaussian Mechanism (SGM) Abbai et al. 2016 Mironov et al. 2019 Let f be
an arbitrary function mapping subsets of 'D to Rd. The sampled Gaussian mechanism (SGM) parametrized with the sampling rate q and the noise multiplier > 0 is defined
SG - f (+x= x E D is sampled with probability a}) 00,
where each element of D sampled independently random with probability q without replacement.
The sampled Gaussian mechanism consists of adding i.dd Gaussian noise with zero and variance o² to each coordinate of the true output of f, i.e., SG injects random vectors from multivariate isotropic Gaussian distribution (000 La) and into the true output, where Ia is written as I if unambiguous in the given context.
Theorem D.1. Mironov et al. 2019 Let SG2. be the sampled Gaussian mechanism for some function f with A < 1 for any adjacent D, D' under the add--or-remove-one notion. Then SG satisfies (o, if
s D. (N. + (N((1,03)
and ps DDD D. (1- +N ||N(0,03)) ||
Theorem D.1 reduce the problem of proving the RDP bound for SG,, to a simple special case of mixture of one-dimensional Gaussians.
34

---

Published in Transactions on Machine Learning Research (04/2024)
Theorem D.2. Mironov et al. 2019 Let SG be the sampled Gaussian mechanism for some function I and under the assumption 4z < for any adjacent D, D' under the add-or-remove-one notion. Tet Mo denote the pdf of N0,,0²), in denote the pdf of N(1,o²), and let μ be the mixture of two Gaussians μ (1 - (1 + qui. Then SG satisfies (a, p)-RDP if
1
log (max Ac Ba))
a
where
Aa E - - [[(=))(((()
Ba E [==aa(H(())/m((2))
Theorem D.2 states that applying SGM function of sensitivity (Equation 2.3 at most 1 satisfies (a, p)- RDP if a-I log,max(A,, B. Thus, analyzing RDP properties of SGM is equivalent to upper bounding Aa and B..
Corollary D.1. (Mironov et al. 2019 Aa 2 Ba for any 1.
This allows reformulation of the RDP bound as
log Aa
The Aa be calculated for range of values using the numerically stable computation approach presented in Section 3.3 of Mironov et al. (2019), which is implemented in standard DP packages such Opacus P7 and Tensorflo--privacy 28 Then, the smallest Aa (tightest bound) is used to upper bound and later the RDP privacy cost is converted to (e, 6)-D via. Theorem 2.3 Notably, this approach generalizes previous results such as moment accountant
(Abadi et al. 2016 (See Table in Mironov et al. (019) for a summary).
27 httpe://opscue.a/
35


Reviews:
[{"review_id": "2MXx3CeO8x", "reviewer": "Reviewer_bWxG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The exposition is clear and thorough. I consider this a strong review paper. The novelty is limited, but I agree that the proposed framework and taxonomy is a strong contribution.\n\nI recommend this paper for acceptance to TMLR. I would consider this paper an solid resource for both academic researchers, industry practioners, and grad students.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a unified view of differential privacy for generative models. It is a synthesis paper, that aptly summarizes about 10 years of methods in the intersection of DP and Gen AI into a coherent mathematical framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would recommend that the authors add the following to items to improve this work: A discussion on the similarities and distinctions when applying DP as a in fine-tuning a generative model compared to in pre-training. With generative models now serving as foundation models, it would be helpful to understand the differences along this dimension.\n\nFurthermore, the authors should probably be a bit more thorough concerning DP fine-tuning of transformer-based LLMs. This is an extremely timely topic, as these models are becoming increasingly pervasive.\n\nI do not considering this \"critical\", rather, strongly recommended.", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LUA4KFb6KN", "reviewer": "Reviewer_UFro", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The visualizations, I think they are all very intuitive, stimulating and to the point. I loved them. They are also pretty accurate and thorough. \n2. The flow of the paper and the categorization: I think the way the paper is written will benefit both experts and non-expert, which is not always the case for survey papers! \n\n\nWeaknesses:\n1. Although the paper claims to unify all DP-synthesis approaches, and to work on different modalities, the work from the NLP literature is missing. See the requested changes for a full list. In general, I think a discussion of modalities and different challenges for each one is needed. \n\n2. The paper is more like a good tutorial, than an SoK paper, as in it doesn't really fully help the reader understand when to use what method. What is next in this field? \n\n3. I think the within metric categorization is redundant. See below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a systemization of the knowledge surrounding differentially private data-synthesis. It starts off by providing primers on differential privacy and generative models, and then moves on to categorizing different DP-synthesis methods based on where the privacy barrier is placed. They provide four such groups: barrier between measure and real data, barrier within measure, barrier between measure and synthetic data and barrier within generator. They then discuss the work that falls under each category.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Missing all NLP related synthesis references: The paper is missing a discussion on synthesized data in language, in particular the following papers:\n\n[1] Mattern, Justus, et al. \"Differentially Private Language Models for Secure Data Sharing.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n[2] Xiang Yue, Huseyin Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim. 2023. Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)\n[3]  Fatemehsadat Mireshghallah, Yu Su, Tatsunori Hashimoto, Jason Eisner, and Richard Shin. 2023. Privacy-Preserving Domain Adaptation of Semantic Parsers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)\n\nI believe all of these methods would fall under the 'within generator' category. \n\n\n2. The within measure category seems a bit unintuitive for me. Barrier is usually not within a thing, it is between componenets. I think both methods (PATE-GAN and DP-GAN) would be between real-data and measure. Because for PATE, it's actually the noisy output that is fed to the measure, so the noise is not in the measure. Similar case for GAN.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7oDXqOMOFr", "reviewer": "Reviewer_Lvy8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n1. This paper is a well-written review paper for existing techniques of DP generative models. All key concepts are well introduced with high-quality figure illustrations. The organization and presentation is friendly to broader audiences\n2. The paper provides systematic categorizations of existing techniques under a unified framework, clearly discussing the setting, strength, and weakness of different methods\n3. The paper clearly identifies open questions in the field for future developments\n\n## Weakness\n\nAs the motivation of DP generative model is to enable downstream statistical analysis by sharing a sanitized form of the data, the tradeoff between privacy and utility naturally come into play. Such tradeoff is an important factor for end users to select a proper DP method for their need. Though the paper made extensive discussion on the privacy guarantee of different methods, the implication on utility is not discussed. There also lacks a comprehensive discussion on potential downstream task that can be benefited from the DP generative models. Adding the discussion on downstream task and the utility tradeoff would be especially helpful for practitioners aiming to utilize the DP model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a comprehensive review on existing approaches for differentially private deep generative modeling from a unified prespective. Different methods are systematically introduced, categorized, and analyzed on the strength and weakness. Open issues and potential future issues in the field of DP generation methods are also discussed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness part", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission is recommended for a survey certification on the basis of the content and recommendation of the reviewers.
Decision: Accept (poster)