Title: DIFFERENTIALLY PRIVATE MIXED-TYPE DATA GENERATION FOR UNSUPERVISED LEARNING

Abstract: In this work we introduce the DP-auto-GAN framework for synthetic data generation, which combines the low dimensional representation of autoencoders with the flexibility of Generative Adversarial Networks (GANs). This framework can be used to take in raw sensitive data, and privately train a model for generating synthetic data that will satisfy the same statistical properties as the original data. This learned model can be used to generate arbitrary amounts of publicly available synthetic data, which can then be freely shared due to the post-processing guarantees of differential privacy. Our framework is applicable to unlabeled mixed-type data, that may include binary, categorical, and real-valued data. We implement this framework on both unlabeled binary data (MIMIC-III) and unlabeled mixedtype data (ADULT). We also introduce new metrics for evaluating the quality of synthetic mixed-type data, particularly in unsupervised settings.

1. INTRODUCTION:

As data storage and analysis are becoming more cost effective, and data become more complex and unstructured, there is a growing need for sharing large datasets for research and learning purposes. This is in stark contrast to the previous statistical model where a data curator would hold datasets and answer queries from (potentially external) analysts. Sharing entire datasets allows analysts the freedom to perform their analyses in-house with their own devices and toolkits, without having to pre-specify the analyses they wish to perform. However, datasets are often proprietary or sensitive, and cannot be shared directly. This motivates the need for synthetic data generation, where a new dataset is created that shares the same statistical properties as the original data. These data may not be of a single type: all binary, all categorial, or all real-valued; instead they may be of mixed-types, containing data of multiple types in a single dataset. These data may also be unlabeled, requiring techniques for unsupervised learning, which is typically a more challenging task than supervised learning on labeled data.

Privacy challenges naturally arise when sharing highly sensitive datasets about individuals. Ad hoc anonymization techniques have repeatedly led to severe privacy violations when sharing "anonymized" datasets. Notable examples include the Netflix Challenge (Narayanan & Shmatikov, 2008) , the AOL Search Logs (Barbaro & Zeller, 2006) , and Massachusetts State Health data (Ohm, 2010) , where linkage attacks to publicly available auxiliary datasets were used to reidentify individuals in the dataset. Even deep learning model have been shown to inadvertently memoize sensitive personal information such as Social Security Numbers during training (Carlini et al., 2019) .

Differential privacy (DP) (Dwork et al., 2006) (formally defined in Section 2) has become the de facto gold standard of privacy in the computer science literature. Informally, it bounds the amount the extent to which an algorithm can depend on a single datapoint in its training set. This guarantee ensures that any differentially privately learned models do not overfit to individuals in the database, and therefore cannot reveal sensitive information about individuals. It is an information theoretic notion that does not rely on any assumptions of an adversary's computational power or auxiliary knowledge. Furthermore, it has been shown empirically that training machine learning models with differential privacy protects against membership inference and model inversion attacks (Triastcyn & Faltings, 2018; Carlini et al., 2019) . Differentially private algorithms have been deployed at large scale in practice by organizations such as Apple, Google, Microsoft, Uber, and the U.S. Census Bureau.

Much of the prior work on differentially private synthetic data generation has been either theoretical algorithms for highly structured classes of queries (Blum et al., 2008; Hardt & Rothblum, 2010) or based on deep generative models such as Generative Adversarial Networks (GANs) or autoencoders. These architectures have been primarily designed for either all-binary or all-real-valued datasets, and have focused on the supervised setting, where datapoints are labelled.

In this work we introduce the DP-auto-GAN framework, which combines the low dimensional representation of autoencoders with the flexibility of GANs. This framework can be used to take in raw sensitive data, and privately train a model for generating synthetic data that should satisfy the same statistical properties as the original data. This learned model can be used to generate arbitrary amounts of publicly available synthetic data, which can then be freely shared due to the post-processing guarantees of differential privacy. We implement this framework on both unlabeled binary data (for comparison with previous work) and unlabeled mixed-type data. We also introduce new metrics for evaluating the quality of synthetic mixed-type data, particularly in unsupervised settings.

1.1. OUR CONTRIBUTIONS:

In this work, we provide three main contributions: a new algorithmic framework for privately generating synthetic data, new evaluation metrics for measuring the quality of synthetic data in unsupervised settings, and empirical evaluations of our algorithmic framework using our new metrics, as well as standard metrics.

Algorithmic Framework. We propose a new data generation architecture which combines the versatility of an autoencoder (Kingma & Welling, 2013) with the recent success of GANs on complex data. Our model extends previous autoencoder-based DP data generation (Abay et al., 2018; Chen et al., 2018) by removing an assumption that the distribution of the latent space follows a mixture of Gaussian distribution. Instead, we incorporate GANs into the autoencoder framework so that the generator must learn the true latent distribution against the discriminator. We describe the composition analysis of differential privacy when the training consists of optimizing both autoencoders and GANs (with different noise parameters). Furthermore, in this analysis we halve the noise injected into autoencoder from all existing works while provably maintaining the same mathematical privacy guarantee.

Unsupervised-Learning Evaluation Metric of Synthetic Data. We define several new metrics that evaluate the performance of synthetic data compared to the original data when the data is of mixedtype. Previous metrics in the literature are applicable only to all-binary or all-real-valued datasets. Our new metrics generalize the previously used metrics (Choi et al., 2017; Xie et al., 2018 ) from allbinary data to mixed-type by training various learning models to predict each feature from the rest of the data in order to assess correlation between features. In additional, our metrics do not require a particular feature to be specified as a label, and therefore do not assume a supervised-learning nature of the data, as in much of the previous work does (Papernot et al., 2017; 2018; Jordon et al., 2018) .

Empirical Results. We empirically comepare the performance of our algorithmic framework on the MIMIC-III medical dataset (Johnson et al., 2016) and UCI ADULT Census dataset (Dua & Graff, 2017) using previously studied metrics in literature (Frigerio et al., 2019; Xie et al., 2018) . Our experiments show that our algorithms perform better, and allow significantly improved values with ≈ 1, compared to prior work (Xie et al., 2018) with ≈ 200. We evaluate our synthetic data using new quantitative and qualitative metrics, confirming that the performance of our algorithm remains high even for small values of , corresponding to strong privacy guarantees. Our code is made publicly available for future use and research.

1.2. RELATED WORK ON DIFFERENTIALLY PRIVATE DATA GENERATION:

Early work on differentially private synthetic data generation was focused primarily on theoretical algorithms for solving the query release problem of privately and accurately answering a large class of pre-specified queries on a given database. It was discovered that generating synthetic data on which the queries could be evaluated allowed for better privacy composition than simply answering all the queries directly (Blum et al., 2008; Hardt & Rothblum, 2010; Hardt et al., 2012; Gaboardi et al., 2014) . Bayesian inference has also been used for differentially private data generation (Zhang et al., 2017; Ping et al., 2017) by estimating the correlation between features. See Surendra & Mohan (2017) for a survey of techniques used in private synthetic data generation through 2016.

In 2016, Abadi et al. (2016) introduced a framework for training deep learning models with differential privacy. Non-convex optimization, which is required when training deep models, can be made differentially private by adding a Gaussian noise to a clipped (norm-bounded) gradient in each training step. Abadi et al. (2016) also introduced the moment accountant privacy analysis for private stochastic gradient descent, which provided much tighter Gaussian-based privacy composition and allowed for significant improvements in accuracy over previously used composition techniques, such as advanced composition Dwork et al. (2010) . The moment account was later defined in terms of Renyi Differential Privacy (RDP) (Mironov, 2017) , which is a slight variant of differential privacy designed for easy composition, particularly for differentially private stochastic gradient descent (DP-SGD). Much of the work that followed on private data generation used deep (neural-networkbased) generative models to generate synthetic data, and can be broadly categorized into two types: autoencoder-based and GAN-based. Our algorithmic framework is the first to combine both DP GANs and autoencoders into one framework.

Differentially Private Autoencoder-Based Models. A variational autoencoder (VaE) (Kingma & Welling, 2013 ) is a generative model that compresses high-dimensional data to a smaller space called latent space. The compression is commonly achieved through deep models and can be differentially privately trained (Chen et al., 2018; Acs et al., 2018) . VaE makes the (often unrealistic) assumption that the latent distribution is Gaussian. Acs et al. (2018) uses Restricted Boltzmann machine (RBM) to learn the latent Gaussian distribution, and Abay et al. ( 2018) uses expectation maximization to learn a Gaussian mixture. Our work extends this line of work by additionally incorporating the generative model GANs which have also been shown to be successful in learning latent distributions.

Differentially Private GANs. GANs are a generative model proposed by Goodfellow et al. (2014) that have been shown success in generating several different types of data (Mogren, 2016; Saito et al., 2017; Salimans et al., 2016; Jang et al., 2016; Kusner & Hernández-Lobato, 2016; Wang et al., 2018) . As with other deep models, GANs can be trained privately using the aforementioned private stochastic gradient descent (formally introduced in Section 2.1). Additional related work, including variants of the DP GAN framework, optimization techniques to improve the performance of DP GANs, and Table 4 summarizing these works can be found in Appendix D.

Differentially Private Generation of Mixed-Type Data. Next we describe the three most relevant recent works on privately generating synthetic data of mixed type. Abay et al. (2018) consider the problem of generating mixed-type labeled data with k possible labels. Their algorithm, DP-SYN, partitions the dataset into k sets based on the labels and trains a DP autoencoder on each partition. Then a DP expectation maximization (DP-EM) algorithm of Park et al. (2017) is used to learn the distribution in the latent space of encoded data of the given label-class. The main workhorse, DM-EM algorithm, is designed and analyzed for Gaussian mixture models and more general factor analysis models. Chen et al. (2018) works in the same setting, but replaces the DP auto-encoder and DP-EM with a DP variational auto-encoders (DP-VaE). Their algorithm assumes that the mapping from real data to the Gaussian distribution can be efficiently learned by the encoder. Finally, Frigerio et al. (2019) used a Wasserstein GAN (WGAN) to generate differentially private mixed-type synthetic data. This type of GAN uses a Wasserstein-distance-based loss function in training. Their algorithmic framework privatized the WGAN using DP-SGD, similar to the previous approaches for image datasets (Zhang et al., 2018; Xie et al., 2018) . The methodology of Frigerio et al. (2019) for generating mixed-type synthetic data involved two main ingredients: changing discrete (categorical) data to binary data using one-hot encoding, and adding an output softmax layer to the WGAN generator for every discrete variable.

Our framework is distinct from these three approaches. We use a differentially private auto-encoder which, unlike DP-VaE of Chen et al. (2018) , does not require mapping data to a Gaussian distribution. This allows us to reduce the dimension of the problem handled by the WGAN, hence escaping the issues of high-dimensionality from the one-hot encoding of Frigerio et al. (2019) . We also use DP-GAN, replacing DP-EM in Abay et al. (2018) , for learning distributions in the latent encoded space.

Evaluation Metrics for Synthetic Data. Various evaluation metrics have been considered in the literature to quantify the quality of the synthetic data (see Charest (2011) for a survey). The metrics can be broadly categorized into two groups: supervised and unsupervised. Supervised evaluation metrics are used when there are clear distinctions between features and labels of the dataset, e.g., for healthcare applications, a person's disease status is a natural label. In these settings, a predictive model is typically trained on the synthetic data, and its accuracy is measured with respect to the real (test) dataset. Unsupervised evaluation metrics are used when no feature of the data can be decisively termed as a label. Recently proposed metrics include dimension-wise probability for binary data (Choi et al., 2017) , which compares the marginal distribution of real and synthetic data on each individual feature, and dimension-wise prediction which measures how closely synthetic data captures relationships between features in the real data. This metric was proposed for binary data, and we extend it here to mixed-type data. Recently, NIST (2019) used a 3-way marginal evaluation metric which used three random features of the real and synthetic datasets to compute the total variation distance as a statistical score. See Appendix D for more details on both categories of metrics, including Table 1 which summarizes the metrics' applicability to various data types.

2. PRELIMINARIES ON DIFFERENTIAL PRIVACY:

In the setting of differential privacy, a dataset X consists of m individuals' sensitive information, and two datasets are neighbors if one can be obtained from the other by the addition or deletion of one datapoint. Differential privacy requires that an algorithm produce similar outputs on neighboring datasets, thus ensuring that the output does not overfit to its input dataset, and that the algorithm learns from the population but not from the individuals. Definition 1 (Differential privacy (Dwork et al., 2006) ). For , δ > 0, an algorithm M is ( , δ)differentially private if for any pair of neighboring databases X, X and any subset S ⊆ Range(M),

Pr[M(X) ∈ S] ≤ e • Pr[M(X ) ∈ S] + δ.

A smaller value of implies stronger privacy guarantees (as the constraint above binds more tightly), but usually corresponds with decreased accuracy, relative to non-private algorithms or the same algorithm run with a larger value of . Differential privacy is typically achieved by adding random noise that scales with the sensitivity of the computation being performed, which is the maximum change in the output value that can be caused by changing a single entry. Differential privacy has strong composition guarantees, meaning that the privacy parameters degrade gracefully as additional algorithms are run on the same dataset. It also has a post-processing guarantee, meaning that any function of a differentially private output will maintain the same privacy guarantees.

2.1. DIFFERENTIALLY PRIVATE STOCHASTIC GRADIENT DESCENT (DP-SGD):

The DP-SGD framework (given formally in Algorithm 5 in Appendix D.1) is generically applicable for private non-convex optimization. In our proposed model, we use this framework to train the autoencoder and GAN.

Training deep learning models reduces to minimizing some (empirical) loss function f (X; θ) :=

1 m m i=1 f (x i ; θ) on a dataset X = {x i ∈ R n } m i=1 .

Typically f is a nonconvex function, and a common method to minimize f is by iteratively performing stochastic gradient descent (SGD). To make SGD private, Abadi et al. (2016) proposed to is to first clip the gradient of each sample to ensure bounded 2 -norm, and then add multivariate Gaussian noise to the gradient. The clipping reduces the scale of noise that must be added to preserve differential privacy. The noisy-clippedgradient is then used in the update step instead of the true gradient. Further details of this procedure are deferred to Appendix D.1.

A variant notion of differential privacy, known as Renyi Differential Privacy (RDP) (Mironov, 2017) , that is often used to analyze privacy for DP-SGD. A randomized mechanism M is (α, )-RDP if for all neighboring databases X, X that differ in at most one entry,

RDP (α) := D α (M(X)||M(X )) ≤ , where D α (P ||Q) := 1 α-1 log E x∼X P (x) Q(x) α

is the Renyi divergence or Renyi entropy of order α between two distributions P and Q. Renyi divergence is better tailored to tightly capture the privacy loss from the Gaussian mechanism that is used in DG-SGD, and is a common analysis tool for DP-SGD literature. To compute the final ( , δ)-differential privacy parameters from iterative runs of DP-SGD, one must first compute the subsampled Renyi Divergence, then compose privacy under RDP, and then convert the RDP guarantee into DP. Further details of this process are given in Appendix D.2.

3. ALGORITHMIC FRAMEWORK:

The overview of our algorithmic framework DP-auto-GAN is shown in Figure 1 , and the full details are given in Algorithm 1. The algorithm takes in m raw data points, and pre-processes these points into m vectors x 1 , . . . , x m ∈ R n to be read by DP-auto-GAN, where usually n is very large. For example, categorical data may be pre-processed using one-hot encoding, or text may be converted into numerical values. Similarly, the output of DP-auto-GAN can be post-processed from R n back to the data's original form. We assume that this pre-and post-processing can done based on public knowledge, such as possible categories for qualitative features and reasonable bounds on quantitative features, and therefore does not require privacy.

Within the DP-auto-GAN, there are two main components: the autoencoder and the GAN. The autoencoder serves to reduce the dimensionality of the data before it is fed into the GAN. The GAN consists of a generator that takes in noise z sampled from distribution Z and produces G w (z) ∈ R d , and a discriminator D y (•) : R n → {0, 1}. Because of the autoencoder, the generator only needs to synthesize data based on the latent distribution R d , which is a much easier task than synthesizing in the original high-dimensional space R n . Both components of our architecture, as well as our algorithm's overall privacy guarantee, are described in the remainder of this section. 

3.1. AUTOENCODER TRAINING:

The autoencoder consists of the encoder En φ (•) : R n → R d and decoder De θ (•) : R d → R n parametrized by edge weights φ, θ, respectively. The architecture of the autoencoder assumes that high-dimensional data x i ∈ R n can be represented compactly in low-dimensional space R d , also called latent space. The encoder En φ is trained to find such low-dimensional representations. We also need the decoder, De θ , to map this point En φ (x i ) in the latent space back to x i . A measure of the information preserved in this process is the error between the decoder's image and the original x i . Thus a good autoencoder should minimize the distance dist(De θ (En φ (x i )), x i ) for each datapoint x i and the appropriate distance function dist. Our autoencoder uses binary cross entropy loss:

dist(x, y) = - n j=1 y (j) log(x (j) ) - n j=1 (1 -y (j) ) log(1 -x (j) ) (where x (j) is the jth coordinate of x ∈ R n ).

This also motivates the definition of a (true) loss function E x∼Z X [dist(De θ (En φ (x i )), x i )] when data are drawn independently from an underlying distribution Z X . The corresponding empirical loss function when we have an access to sample

{x i } m i=1 is L auto (φ, θ) := m i=1 dist(De θ (En φ (x i )), x i ).

(1)

The task of finding a good autoencoder reduces to optimizing φ and θ to yield small empirical loss as in Equation 1.

We minimize Equation 1 privately using DP-SGD (described in Section 2.1). Our approach differs from previous work on private training of autoencoders (Chen et al., 2018; Acs et al., 2018; Abay et al., 2018) by not adding noise to the decoder during DP-SGD, whereas previous work adds noise to both the encoder and decoder. This improves performance by reducing the noise injected into the model by half, while still maintaining the same privacy guarantee (see Proposition 3). The full description of our autoencoder training is given in Algorithm 2 in the appendix. In our DP-auto-GAN framework, the autoencoder is trained first until completion, and is then fixed for the second phase of training GAN.

3.2. GAN TRAINING:

A GAN consists of the generator G w and discriminator D y : R n → {0, 1}, parameterized respectively by edge weights w and y. The aim of the generator G w is to synthesize (fake) data similar to the real dataset, while the aim of discriminator is to determine whether an input x i is from the generator's synthesized data (and assigning label D y (x i ) = 0) or is real data (and assigning label D y (x i ) = 1). The generator is seeded with a random noise z ∼ Z that contains no information about real dataset, such as a multivariate Gaussian vector, and aims to generate a distribution G w (z) that is hard for D y is distinguish from the real data. Hence, the generator wants to minimize the probability that D y makes a correct guess, E z∼Z [1 -D y (G w (z))]. At the same time, the discriminator wants to maximize its probability of correct guess when the data is fake

E z∼Z [1-D y (G w (z))]

and when the data is real

E x∼Z X [D y (x)].

We generalize the output of D y to a continuous range [0, 1], with the value indicating the confidence that a sample is real. We use the zero-sum objective for the discriminator and generator proposed by Arjovsky et al. (2017) and motivated by the Wasserstein distance of two distributions. Although their proposed Wasserstein objective cannot be computed exactly, it can be approximated by optimizing the objective:

EQUATION

We optimize Equation 2 privately using the DP-SGD framework described in Section 2.1. We differ from prior work on DP GANs in that our generator 

G w (•) outputs data G w (z) in latent space

3.3. PRIVACY ACCOUNTING:

Our autoencoder and GAN are trained privately by adding noise to the encoder and discriminator. Since the generator only accesses data through the discriminator's (privatized) output, then the trained parameters of generator are also private by post-processing guarantees of differential privacy. Finally, we release the privatized decoder and generator, together with generator's noise distribution Z and post-processing procedure, both of which are assumed to be public knowledge.

The privacy accounting is therefore required for the two parts that access real data X: training the autoencoder and the discriminator. In each training procedure, we apply the RDP accountant (described in Section 2.1 and Appendix D.2) to analyze privacy of the DP-SGD training algorithm, to compute final ( , δ)-DP bound. Our application of the RDP accountant diverges from the previous literature in two main ways.

First, we do not add noise to decoder during the autoencoder training, which is contrary to prior work that adds noise to both the encoder and decoder. Our approach of not adding noise to the decoder does not affect the algorithms' overall privacy guarantees. This claim is stated formally in the following corollary, which follows immediately from Propositions 3 and 4.

Corollary 2. Suppose autoencoder in DP-auto-GAN is trained with RDP privacy RDP auto (•) and the discriminator in DP-auto-GAN is trained with RDP privacy RDP D (•), then DP-auto-GAN is RDP with values RDP auto (•)+RDP D (•).

Second, the privacy analysis must account for two phases of training, usually with different privacy parameters (due to different batch sampling rates, noise, and number of iterations). One obvious solution is to calculate the desired ( , δ)-DP parameter obtained from each phase and compose them to obtain ( 1 + 2 , δ 1 + δ 2 )-DP using basic composition of differential privacy (Dwork et al., 2006) . However, we can obtain a tighter privacy bound by composing the privacy at the Renyi Divergence level before translating Renyi Divergence into ( , δ)-DP. In other words, we first apply Proposition 4 to compute RDP(•) of two-phase training before applying Proposition 5 to translate RDP into DP. This is the approach highlighted in Corollary 2. In practice, this reduces the privacy parameter by about 30%.

4. EVALUATION METRICS:

In this section, we discuss the evaluation metrics that we use in the experiments (described in Section 5) to empirically measure the quality of the synthetic data. Some of these metrics have been used in the literature, while many are novel contributions in this work. The evaluation metrics are summarized in Table 1 ; our contributions are in bold.

For the first two metrics described below, the dataset should be partitioned into a training set R ∈ R m1×n and testing set T ∈ R m2×n , where m = m 1 + m 2 is the total number of samples the real data, and n is the number of features in the data. After training the DP-auto-GAN, we use it to create a synthetic dataset S ∈ R m3×n , for sufficiently large m 3 .

Dimension-wise probability. This metric is used when the entire dataset is binary, and it serves as a basic sanity check to verify whether DP-auto-GAN has correctly learned the marginal distribution of each feature. Specifically, it compares the proportion of 1's (which can be thought of as estimators of Bernoulli success probability) in each feature of the training set R and synthetic dataset S.

Dimension-wise prediction. This metric evaluates whether DP-auto-GAN has correctly learned the relationships between features. For the k-th feature of training set R and synthetic dataset S, we choose y R k ∈ R m1 and y S k ∈ R m3 as labels of a classification or regression task based on the type of that feature, and the remaining features R -k and S -k are used for prediction. We train either a classification or regression model and measure their goodness of fit based on the model's accuracy using AUROC, F 1 or R 2 scores, which are formally defined in Appendix C.

We also propose following novel evaluation metrics. For more details, we refer the reader to Appendix D.5.

1-way feature marginal. This metric works as a sanity check for real features. We compute histograms for the feature interest of both real and synthetic data. The quality of the synthetic data with respect to this metric can be evaluated qualitatively through visual comparison of the histograms on real and synthetic data. This can be extended to k-way feature marginals and made into a quantitative measure by adding a distance measure between the histograms. 

Yes:

Yes Yes 2-way PCA marginal. This metric generalizes the 3-way marginal score used in NIST (2019). In particular, we compute principle components of the original data and evaluate a projection operator for first two principle components. Let us denote P ∈ R n×2 as the projection matrix such that R = RP is the projection on first two principle components of R. Then we evaluate projection of synthetic data S = SP and scatterplot 2-D points in R and S for visual evaluation. For quantitative evaluation, we also compute Wasserstein distance between R and S. In the simulations described in Section 5, we used Wasserstein distance since we optimize for the WGAN objective, but any distributional divergence metric can be used. This approach can also be extended to k-way marginals by making the projection matrix P ∈ R n×2 for the first k principle components.

Distributional distance. In this metric, we first compute the Wasserstein distance W 2 (R, S) between the entire real and synthetic datasets R, S. The Wasserstein score is then defined as

W score (R, S) := 1 - W2(R,S) max x,y∈X ||x-y|| 2 2 ,

where the Wasserstein distance is normalized by the maximum distance possible of two datapoints in data universe X . To compute the Wasserstein score on k-way marginal PCA projection P , we normalize the score with additional term √ v, where v is the explained variance of P :

W score ( R, S, P ) := 1 - W2( R, S) √ v max x,y∈X ||x-y|| 2 2 .

5. EXPERIMENTS:

In this section we present details of our datasets and show empirical results of our experiments.

Throughout our experiments, we fix δ = 10 -5 for training DP-auto-GAN and show results for different values of including = ∞ (i.e., non-private GAN) which serves as a benchmark. We also compare our results with existing works in the literature where relevant. Details of hyper-parameters and architecture can be found in the appendix. The code of our implementation is available at https://github.com/DPautoGAN/DPautoGAN. The line y = x corresponds to the ideal performance.

5.2. MIXED DATA:

Second, we consider the ADULT dataset Dua & Graff (2017) which is an extract of the U.S. Census and contains information about working adults. This dataset has 14 features out of which 10 features are categorical and four are real-valued. Figure 4 shows the dimension-wise prediction plot of DPauto-GAN on this dataset. For categorical features (represented by blue points and a single green point), we use random forest classifier in order to compare our result with Frigerio et al. (2019) . For real-valued features (represented by red points), we used a lasso regression model. The green point corresponds to the salary feature of the data, which is real-valued but treated as binary, based on the condition > $50k, which is similarly used as a binary label in Frigerio et al. (2019) . We use F 1 score as our classification accuracy measure for categorical features in in Figure 4 , and we use R 2 score as our regression accuracy for real-valued features. The F 1 score is preferred over AUROC score for the ADULT dataset because it has many non-binary features where AUROC cannot be used. Each point in Figure 4 corresponds to one feature, and the x and y coordinates respectively show the accuracy score on the real data and the synthetic data.

Similar to the MIMIC-III dataset, we see that for large values of , points are scattered close to y = x line, and as gets smaller, points gradually shift downward implying, that accuracy of synthetic data deceases with stronger privacy guarantees. For the salary feature, we also compute accuracy scores for comparison with Frigerio et al. (2019) . In Table 2 , we report the accuracy of each synthetic dataset as well as benchmark accuracy. We see that our accuracy guarantees are higher than those of Frigerio et al. (2019) with smaller values.

Note that in the ADULT dataset, we have four real-valued features but we only plot one red point in Figure 4 , corresponding to the age feature. The other real-valued features (capital gain, capital loss, and hours worked per week) were not included because even on the real data, we were not able to find a regression model with good fit (as measured by R 2 score) for these features in terms of the other features. To check whether we learned the distribution correctly for these features, we plot 1-way feature marginal histogram on each of them. See Figure 7 in Appendix C. It can be seen that DP-auto-GAN identifies the distribution in those features.

In order to understand combined performance of all features, we use two metrics. First, we show the qualitative results from 2-way PCA marginal score in Figure 5 A close qualitative inspection 3 . 1: architecture input: Private dataset D ∈ X m where X is the set of (raw) data universe, preprocessed data dimension n, latent space dimension d, preprocessing function P re : X → R n , post-processing function P ost : R n → X , encoder architecture En φ : R n → R d parameterized by φ, decoder architecture De θ : R d → R n parameterized by θ, generator's noise distribution Z on sample space Ω(Z), generator architecture G w : Ω(Z) → R d parameterized by w, discriminator architecture D y : R n → {0, 1}. 2: autoencoder training parameters: Learning rate η 1 , number of iteration rounds (or optimization steps) T 1 , loss function L auto , optimization method optim auto batch sampling rate q 1 (for the batch expectation size b 1 = q 1 m), clipping norm C 1 , noise multiplier ψ 1 , microbatch size r 1 3: generator training parameters: Learning rate η 2 , batch size b 2 , loss function L G , optimization method optim G , number of generator iteration rounds (or optimization steps) T 2 4: discriminator training parameters: Learning rate η 3 , number of discriminator iterations per generator step t D , loss function L D , optimization method optim D , batch sampling rate q 3 (for the batch expectation size b 3 = q 3 m), clipping norm C 3 , noise multiplier ψ 3 , microbatch size r 3 5: privacy parameter δ > 0 6: procedure DPAUTOGAN for j = 1 . . . k do Both g j φ , g j θ can be computed in one backpropagation 8:

g j φ , g j θ ← ∇ φ (L auto (De θ (En φ (B j )), B j )), ∇ θ (L auto (De θ (En φ (B j )), B j ) 9: g φ ← 1 k k j=1 g j φ 10: g θ ← 1 k k j=1 CLIP(g j φ , C 1 ) + N (0, C 2 1 ψ 2 1 I) 11:

(φ, θ) ← optim auto (φ, θ, g φ , g θ , η 1 ) 

{z i } r i=1 ∼ Z r 9: B ← {De(G w (z i ))} r i=1

10:

g j ← ∇ y (L D (B j , B , D y ))

In the case of WGAN, 

L D (B j , B , D y ) := 1 r b∈Bj D y (b) - 1 r b ∈B D y (b ) 11: g ← 1 k k j=1 CLIP(g j , C 3 ) + N (0, C 2 3 ψ 2 3 I)

{z i } b2 i=1 ∼ Z b2 5: B ← {De(G w (z i ))} b2 i=1 6: g ← ∇ w (L G (B , D y ))

In the case of WGAN,

L G (B , D y ) := - 1 b 2 b ∈B D y (b ) 7:

w ← optim G (w, g, η 2 )

(2): Linear(in-feature=256, out-feature=1, bias=True) ) )

C.2 ADDITIONAL MIMIC-III EMPIRICAL RESULTS

Figure 3 showed dimension-wise prediction plot for different values of . As one can see, for = ∞, many points are concentrated along the lower side of line y = x which is the ideal performance. This shows that AUROC score of the real dataset is marginally higher than that of synthetic dataset. For = 3.11865 and = 1.27655, there is a gradual shift downwards compared to line y = x with larger variance in the plotted points. This means that AUROC scores of real and synthetic data shows more difference for smaller values of . For = 0.94145, which shows the same trend, one can also see that number of datapoints plotted have reduced significantly. This is since many features in synthetic data have very high proportion of 0, so logistic regression classifier trained on these features uniformly outputs 0 on the hold-out test dataset T . In such cases, AUROC score outputs 1/2 by default and as such, does not have any meaning. Hence we drop those features from the plot.

Below we show the full plots of dimension-wise prediction for MIMIC-III dataset. The autoencoder was trained via Adam with Beta 1 = 0.9, Beta 2 = 0.999, and a learning rate of 0.005 for 20,000 minibatches of size 64 and a microbatch size of 1. The L2 clipping norm was selected to be the median L2 norm observed in a non-private training loop, equal to 0.012. The noise multiplier was then calibrated to achieve the desired privacy guarantee.

The GAN was composed of two neural networks, the generator and the discriminator. The generator used a ResNet architecture, adding the output of each block to the output of the following block. It was trained via RMSProp with alpha = 0.99 with a learning rate of 0.005. The discriminator was a simple feed-forward neural network with LeakyReLU hidden activation functions, also trained via RMSProp with alpha = 0.99. The L2 clipping norm of the discriminator was set to 0.022. The pair was trained on 15,000 minibatches of size 128 and a microbatch size of 1, with 15 updates to the discriminator per 1 update to the generator. Again, the noise multiplier was then calibrated to achieve the desired privacy guarantee.

A serialization of the model architectures used in the experiment can be found below. 

g ← 1 k k i=1 CLIP(∇ θ f (X Bi , θ), C) + N (0, C 2 ψ 2 I) 11: θ ← OPTIM(θ, g, η)

Performance improvements. In general, the descent step can be performed using other optimization methods-such as Adam or RMSProp-in a private manner, by replacing the gradient value with g in each step. Also, one does not need to clip the individual gradients, but can instead clip the gradient of a group of datapoints, called a microbatch (McMahan & Andrew, 2018) . Mathematically, the batch B is partitioned into microbatches B 1 , . . . , B k each of size r, and the gradient clipping is performed on the average of each microbatch:

g ← 1 k k i=1 CLIP(∇ θ f (X Bi , θ), C) + N (0, C 2 ψ 2 I)

Standard DP-SGD corresponds to setting r = 1, but setting higher values of r (while holding |B| fixed) significantly decreases the runtime and reduces the accuracy, and does not impact privacy significantly for large dataset. Other clipping strategies have also been suggested. We refer readers to McMahan & Andrew (2018) for more details of clipping and other optimization strategies.

The improved privacy analysis by Abadi et al. (2016) (which has been implemented in Google (2018) and is widely used in practice) obtains a tighter privacy bound when data are subsampled, as in SGD. This analysis requires independently sampling each datapoint with a fixed probability q in each step.

D.2 CONVERTING RENYI DP TO DP:

To compute the final ( , δ)-differential privacy parameters from iterative runs of DP-SGD, there are three key steps.

Step 1: Subsampled Renyi Divergence. Given sampling rate q and noise multiplier ψ, one can obtain RDP(•) values as a function of α ≥ 1 for one run of DP-SGD (Mironov, 2017) . We denote this function by RDP T =1 (•), which will depend on q and ψ.

Step 2: Composition of RDP. When DP-SGD is run iteratively, we can compose the Renyi privacy parameter across all runs using the following proposition. Proposition 4 ( (Mironov, 2017) ). If M 1 , M 2 respectively satisfy (α, 1 ), (α, 2 )-RDP for α ≥ 1, then the composition of two mechanisms M 2 (M 1 (X)) satisfies (α, 1 + 2 )-RDP.

Hence, we can compute RDP(•) values for T iterations of DP-SGD as RDP-ACCOUNT(T, q , ψ) := T • RDP T =1 (•).

Step 3: Conversion to ( , δ)-DP. After obtaining the final RDP(•) function, any (α, )-RDP guarantee can be converted into ( , δ)-DP.

Proposition 5 ( (Mironov, 2017) ). If M satisfies (α, )-RDP for α > 1, then for all δ > 0, M satisfies ( + log 1/δ α-1 , δ)-DP.

Since the privacy parameter of RDP is also a function of α, this last step involves optimizing for the α that achieves smallest privacy parameter in Proposition 5.

D.3 DIFFERENTIALLY PRIVATE GAN ARCHITECTURES:

Training deep learning models reduces to minimizing some (empirical) loss function f (X; θ) :=

1 m m i=1 f (x i ; θ) on a dataset X = {x i ∈ R n } m i=1 .

Typically f is a nonconvex function, and a common method to minimize f is by iteratively performing stochastic gradient descent (SGD):

EQUATION

The size of B is typically fixed as a moderate number to ensure quick computation of gradient, while maintaining that 1 |B| i∈B ∇f (x i , θ) is a good estimate of true gradient ∇ θ f (X; θ). In the setting of differential privacy, X is a dataset of m individual's sensitive information, and two datasets are neighbors if one can be obtained from another by the addition or deletion of one datapoint. To make SGD private, a standard method proposed by Abadi et al. (2016) is to first clip the gradient of each sample to ensure the 2 -norm is at most C:

CLIP(x, C) := x • min (1, C/||x|| 2 ) .

Then a multivariate Gaussian noise parametrized by noise multiplier ψ is added before taking an average across the batch, leading to noisy-clipped-averaged gradient estimate g:

g ← 1 |B| i∈B CLIP(∇ θ f (x i , θ), C) + N (0, C 2 ψ 2 I)

The quantity g is now private and can be used for the descent step θ ← θ-η•g in place of equation 4.

Variants of DP GANs have been used for synthetic data generation, including the Wasserstein GAN (WGAN) (Arjovsky et al., 2017; Gulrajani et al., 2017) and DP-WGAN (Alzantot & Srivastava, 2019; Triastcyn & Faltings, 2018 ) that use a Wasserstein-distance-based loss function in training (Arjovsky et al., 2017; Gulrajani et al., 2017; Alzantot & Srivastava, 2019; Triastcyn & Faltings, 2018) ; the conditional GAN (CGAN) (Mirza & Osindero, 2014) and DP-CGAN (Torkzadehmahani et al., 2019) that operate in a supervised (labeled) setting and use labels as auxiliary information in training; and Private Aggregation of Teacher Ensembles (PATE) (Papernot et al., 2017; 2018) for the semi-supervised setting of multi-label classification when some unlabelled public data are available (or PATEGAN (Jordon et al., 2018) when no public data are available). Our work focuses on unsupervised setting where data are unlabeled, and no (relevant) labeled public data are available.

Existing works in differentially private synthetic data generation can be summarized in Table 4 .

D.4 DIFFERENTIALLY PRIVATE TRAINING OF DEEP MODELS:

There are numerous works on optimizing the performance of differentially private GANs, including data partitioning (either by class of labels in supervised setting or a private algorithm) (Yu et al., 2019; Papernot et al., 2017; 2018; Jordon et al., 2018; Abay et al., 2018; Acs et al., 2018; Chen et al., 2018) ; reducing the number of parameters in deep models (McMahan et al., 2017) ; changing the norm clipping for the gradient in DP-SGD during training (McMahan et al., 2017; van der Veen et al., 2018; Thakkar et al., 2019) ; changing parameters of the Gaussian noise used during training (Yu et al., 2019) ; and using publicly available data to pre-train the private model with a warm start (Zhang et al., 2018; McMahan et al., 2017) . Clipping gradients per-layer of models (McMahan & Andrew, 2018; McMahan et al., 2017) and per-dynamic parameter grouping (Zhang et al., 2018) are also proposed. Additional details for some of these optimization approaches are given below. (Hardt & Rothblum, 2010) , MWEM Hardt et al. (2012 ), DualQuery Gaboardi et al. (2014) , DataSynthesizer (Ping et al., 2017 ), PriBayes (Zhang et al., 2017) Batch Sampling Three ways are known to sample a batch from data in each optimization step. The three methods are described in McMahan & Andrew (2018) . We also summarize here for the completeness of DP-SGD background.

The first is to sample each individual's data with a fixed probability independently. This sampling procedure is one used in analysis of subsampled moment account in Abadi et al. (2016) ; McMahan & Andrew (2018) and subsampled RDP composition in Mironov (2017) . This RDP composition is publicly available at Tensorflow Privacy (Google, 2018) . We implement this sampling procedure and use Tensorflow Privacy to account Renyi Divergence during the training.

Another sampling policy is to sample uniformly at random a subset of fixed size of all datapoints. This achieves a different RDP guarantee from the first one, but the analysis of this sampling has been done in Wang et al. (2019) .

Finally, a common subsampling procedure is to shuffle the data at random, and take a fixed-size batch in the order of the shuffling without replacement. The process is repeated after a pass over all datapoints (an epoch). Though this batch sampling is most common in practice, no subsampled privacy composition is known in this case.

Hyperparameter Tuning Training a deep learning models involves hyperparameter tuning to find good architecture and optimization parameters. This process is private and privacy budget must be accounted for. Abadi et al. (2016) accounts for hyperparameter search using the work of Gupta et al. (2010) . Beaulieu-Jones et al. (2019) uses Report Noisy Max Dwork & Roth (2014) to private select a model with top performance when a model evaluation metric is known. Some works are done to account for selecting high-performance models without losing much privacy (Chaudhuri & Vinterbo, 2013; Liu & Talwar, 2019) . In our experimental work, we omit the privacy accounting of hyperparameter search as this is not the focus fof our contribution (new algorithmic framework using RDP subsampled composition for privacy analysis), following most literatures in differentially private synthetic data generation.

D.5 EVALUATION METRICS FOR SYNTHETIC DATA:

Now we review the evaluation schemes for measuring quality of synthetic data. Various evaluation metrics have been considered in the literature to quantify the quality of the synthetic data (Charest, 2011) . Broadly, evaluation metrics can be divided into two major categories: supervised and un-

:



Figures and Tables:

Figure 1: The summary of our DP-auto-GAN algorithmic framework. Pre-and post-processing (in black) are assumed to be public knowledge. Encoder and generator (in green) are trained without noise injection, whereas decoder and discriminator (in yellow) are trained with noise. The four red arrows indicate how data are forwarded for each training: autoencoder training, generator training, and discriminator training. After training, the generator and decoder (but not encoder) are released to the public to generate synthetic data.

R d which needs to be decoded to De(G w (z)) before being fed into the discriminator D y (z). The gradient ∇ w G w is obtained by backpropagation through one more component En(•). Hence, the training of generator remains totally private because the additional component En(•) is fixed and never accesses the private data. The full description of our GAN training is given in Algorithm 4 in the appendix. At the end of the two-phase training (including autoencoder and GAN), the noise distribution Z, trained generator G w (•), and trained decoder De(•) are released to the public. The public can then generate synthetic data by sampling z ∼ Z to obtain a synthesized datapoint En(G y (z)) repeatedly to obtain synthetic dataset of any desired size.

Figure 2: Dimension-wise probability scatterplots for different values of . For each point in the plot represents one of the 1071 features in MIMIC-III dataset. The x and y coordinates of each point are the proportion of 1 in real and synthetic datasets of a feature, respectively.

Figure 3: Dimension-wise prediction scatterplots for different values of . Each point represents one of 1071 features in MIMIC-III dataset. For each point, the x and y coordinates represent the AUROC score of a logistic regression classifier trained on real and synthetic datasets, respectively. The line y = x corresponds to the ideal performance.

Figure 4: Dimension-wise prediction scatterplot for different values of . Each point represents one of 14 features in the ADULT dataset. Blue points and single green points correspond to categorical features, and are plotted according to F 1 score. Red points correspond to real-valued features, and we plot R 2 score. For each point, x and y coordinate represents relevant score evaluated on real and synthetic datasets, respectively.

Figure 5: Scatterplot of projection of dataset on first two principle component of the real dataset

φ, θ, w, y for En φ , De θ , G w , D y Phase 1: autoencoder training 9: for t = 1 . . . T 1 do 10: DPTRAIN AUTO (X, En, De, autoencoder training parameters) Phase 2: GAN training 11: for t = 1 . . . T 2 do 12: for j = 1 . . . t D do (privately) train D y for t D iterations 13: DPTRAIN DISCRIMINATOR (X, Z, G, De, D, discriminator training parameters) 14: TRAIN GENERATOR (Z, G, De, D, generator training parameters) Privacy accounting 15: RDP auto (•) ← RDP-ACCOUNT(T 1 , q 1 , ψ 1 , r 1 ) 16: RDP D (•) ← RDP-ACCOUNT(T 2 • t D , q 3 , ψ 3 , r 3 ) 17: ←GET-EPS(RDP auto (•) + RDP D (•)) 18: return model (G w , De θ ), privacy ( , δ) Algorithm 2 DPTRAIN AUTO (X, En φ , De θ , training parameters) 1: training parameter input: Learning rate η 1 , number of iteration rounds (or optimization steps)T 1 , loss function L auto , optimization method optim auto batch sampling rate q 1 (for the batch expectation size b 1 = q 1 m), clipping norm C 1 , noise multiplier ψ 1 , microbatch size r 1 2: goal: train one step of autoencoder (En φ , De θ ) B into B 1 , . . . , B k each of size r (ignoring the dividend)

DPTRAIN DISCRIMINATOR (X, Z, G w , De θ , D y , training parameters) 1: training parameter input: Learning rate η 3 , number of discriminator iterations per generator step t D , loss function L D , optimization method optim D , batch sampling rate q 3 (for the batch expectation size b 3 = q 3 m), clipping norm C 3 , noise multiplier ψ 3 , microbatch size r 3 2: goal: train one step of discriminator D y 3: procedure DPTRAIN DISCRIMINATOR 4: B ← SAMPLEBATCH(X, q 3 ) 5: Partition B into B 1 , . . . , B k each of size r (ignoring the dividend)

← optim D (y, g, η 3 ) Algorithm 4 TRAIN GENERATOR (Z, G w , De θ , D y , generator training parameters) 1: training parameter input: Learning rate η 2 , batch size b 2 , loss function L G , optimization method optim G , number of generator iteration rounds (or optimization steps) T 2 2: goal: train one step of generator G w 3: procedure TRAIN GENERATOR 4:

Figure 6: Full plots of dimension-wise prediction for mimic dataset

bias=True)   (1): LeakyReLU(negative-slope=0.2) (2): Linear(in-feature=60, out-feature=15, bias=True) (3): LeakyReLU(negative-slope=0.2) ) (decoder): Sequential(D ADDITIONAL BACKGROUND AND RELATED WORK D.1 DETAILS OF DP-SGDThe DP-SGD framework (given formally in Algorithm 5) is generically applicable to private nonconvex optimization.Algorithm 5 DP-SGD (one iteration step) 1: parameter input: Data X = {x i } m i=1 , deep learning model parameter θ, learning rate η, loss function f , optimization method OPTIM, batch sampling rate q (for the batch expectation size b = qm), clipping norm C, noise multiplier ψ, microbatch size r 2: goal: differentially privately train one step of the model parametrized by θ with optim 3: procedure DPx i to B with probability q return B 8:Partition B into B 1 , . . . , B k each of size r (ignoring the dividend)

Summary of evaluation metrics in DP synthetic data generation. We list applicability of each metric to each of the data type. Parts in bold are our new contributions. Evaluation methods with asterisk * are predictive-model-specific, and their applicability therefore depends on types of data that the chosen predictive model is appropriate for. Methods with asterisks ** are equipped with any any distributional distance of choice such as Wasserstein distance.

Table content: <table><tr><td>TYPES</td><td>EVALUATION METHODS</td><td/><td/><td>DATA TYPES</td></tr><tr><td/><td/><td colspan="2">Binary</td><td colspan="2">Categorical Regression</td></tr><tr><td>Supervised</td><td>Abay et al., 2018; Frigerio et al., Label prediction* (Chen et al., 2018;</td><td>Yes</td><td/><td>Yes</td><td>Yes</td></tr><tr><td/><td>2019)</td><td/><td/><td/></tr><tr><td/><td>Predictive model ranking* (Jordon</td><td>Yes</td><td/><td>Yes</td><td>Yes</td></tr><tr><td/><td>et al., 2018)</td><td/><td/><td/></tr><tr><td>Unsupervised,</td><td>Dimension-wise prediction plot*</td><td colspan="2">Yes (Choi</td><td>Yes</td><td>Yes</td></tr><tr><td>prediction-based</td><td/><td>et</td><td>al.</td><td/></tr><tr><td/><td/><td>(2017),</td><td/><td/></tr><tr><td/><td/><td>ours)</td><td/><td/></tr><tr><td>Unsupervised,</td><td>Dimension-wise probability plot</td><td>Yes</td><td/><td>No</td><td>No</td></tr><tr><td>distributional-</td><td>(Choi et al., 2017)</td><td/><td/><td/></tr><tr><td>distance-based</td><td>3-way feature marginal, total varia-</td><td>Yes</td><td/><td>Yes</td><td>Yes</td></tr><tr><td/><td>tion distance (NIST, 2019)</td><td/><td/><td/></tr><tr><td/><td>k-way feature marginal**</td><td>Yes</td><td/><td>Yes</td><td>Yes</td></tr><tr><td/><td>k-way PCA marginal**</td><td>Yes</td><td/><td>Yes</td><td>Yes</td></tr><tr><td/><td>Distributional distance**</td><td>Yes</td><td/><td>Yes</td><td>Yes</td></tr><tr><td>Unsupervised,</td><td colspan="2">1-way feature marginal (histogram) Yes</td><td/><td>Yes</td><td>Yes</td></tr><tr><td>qualitative</td><td>2-way PCA marginal (data visual-</td><td/><td/><td/></tr><tr><td/><td>ization)</td><td/><td/><td/></tr></table>

Accuracy scores of prediction on salary feature evaluated on different values.

Table content: <table><tr><td>value</td><td>Real dataset</td><td>∞</td><td>7</td><td>3</td><td>1.5</td><td>1</td><td>0.8</td></tr><tr><td>Accuracy (ours)</td><td>86.63%</td><td>79.18%</td><td/><td/><td colspan="3">77.86% 76.92% 77.7%</td></tr><tr><td>Accuracy (Frigerio et al. (2019))</td><td>77.2%</td><td colspan="3">76.7% 76.0% 75.3%</td><td/><td/><td/></tr></table>

Wasserstein distance scores on 2-way PCA marginal and on whole dataset, for different 's.

Table content: <table><tr><td>Method</td><td/><td colspan="2">2-way PCA score Whole-data score</td></tr><tr><td colspan="2">DP-auto-GAN 1.5</td><td>44.36%</td><td>60.84%</td></tr><tr><td>DP-auto-GAN</td><td>1</td><td>41.17%</td><td>60.53%</td></tr><tr><td colspan="2">DP-auto-GAN 0.8</td><td>19.25%</td><td>60.51%</td></tr></table>

Algorithmic frameworks for differentially private synthetic data generation. Our new algorithmic framework (in bold) is the first to combine both DPGAN and autoencoder into one framework by using GAN to learn generative model in latent space.

Table content: <table><tr><td>Types</td><td>Main architecture</td><td>Algorithmic framework Variants</td></tr><tr><td>Deep generative models</td><td>DPGAN (Abadi et al., 2016)</td><td>PATEGAN (Jordon et al., 2018) DP Wasserstein GAN (Alzantot &amp; Srivastava, 2019) DP Conditional GAN (Torkzadehmahani et al., 2019) Gumbel-softmax for categorical data (Frigerio</td></tr><tr><td/><td/><td>et al., 2019)</td></tr><tr><td/><td/><td>DP-VaE (Chen et al., 2018; Acs et al., 2018)</td></tr><tr><td/><td>Autoencoder</td><td>RBM generative models in latent space (Acs et al.,</td></tr><tr><td/><td/><td>2018)</td></tr><tr><td/><td/><td>Mixture of Gaussian model in latent space (Abay</td></tr><tr><td/><td/><td>et al., 2018)</td></tr><tr><td/><td colspan="2">Autoencoder and DPGAN (ours)</td></tr><tr><td>Other</td><td>SmallDB (?), PMW</td><td/></tr><tr><td>models</td><td/><td/></tr></table>

Peer review (iclr2020):
This paper proposed a new algorithm for synthetic data generation under differential privacy. The algorithmic architecture combines autoencoder and GAN in a way that it only needs to add the DP-SGD noise to the decoder of the autoencoder and the discriminator of the GAN. This seems to be a good idea to be explored further. The authors claimed that the proposed new evaluation metrics are novel contributions of the paper but there is no discussion on why they are good metrics for evaluating the quality of synthetic datasets nor which metric should be used in what scenarios. It is unclear how the experimental results (Figure 2, 3, 4 and Table 2, 3) are interpreted. The authors mentioned comparison with DP-GAN but it is not marked in the figures the performance of DP-GAN and how its results compared with DP-auto-GAN. Please clearly state what each figure means and why the results are significant. I wonder if it is possible to have a version of GAN that also predicts the labels of data so you can use classification task as evaluation metrics, which might be easier and more interpretable. In Section 3.1, “not adding noise to the decoder” should be encoder. In Section 3.3, “do not add noise to decoder” should be encoder. The presentation of the paper needs to be improved. There are too many typos and grammar mistakes. The labels of figures in the experiment section are too small. The paper does not have a conclusion section.