DP-ImgSyn: Dataset Alignment for Obfuscated, Differentially Private Image Synthesis
Authors: Efstathia Soufleri
Deepak Ravikumar
Kaushik Roy
Venue: Accepted by TMLR ()

Abstract: The availability of abundant data has catalyzed the expansion of deep learning vision algorithms. However, certain vision datasets cannot be publicly released due to privacy reasons. Releasing synthetic images instead of private images is a common approach to overcome this issue. A popular method to generate synthetic images is using Generative Adversarial Networks (GANs) with Differential Privacy (DP) guarantees. However, GAN-generated synthetic images are visually similar to private images. This is a severe limitation, particularly when the private dataset depicts visually sensitive and disturbing content. To address this, we propose a non-generative framework, Differentially Private Image Synthesis (DP-ImgSyn), to generate and release synthetic images for image classification tasks. These synthetic images: (1) have DP guarantees, (2) retain the utility of the private images, i.e., a model trained using synthetic images results in similar accuracy as a model trained on private images, (3) the synthetic images are visually dissimilar to private images. DP-ImgSyn consists of the following steps: First, a teacher model is trained on the private images using a DP training algorithm. Second, public images are used as initialization for the synthetic images which are optimized to align them with the private images. The optimization uses the teacher network's batch normalization layer statistics (mean, standard deviation) to inject information about the private images into the synthetic images. Third, the synthetic images and their soft labels, obtained from the teacher model, are released and can be deployed for neural network training on image classification tasks. Our experiments on various image classification datasets show that when using similar DP training mechanisms, our framework performs better than generative techniques (up to $\approx$ 20% in terms of image classification accuracy).

Published in Transactions on Machine Learning Research (04/2024)
Im-Im: Dataset Alignment for Obfuscated, Differen- tially Private Image Synthesis
Efstathia Soufleri
esoufler@purdue.dl edu
Department of Electrical and Computer Engineering
Purdue Universily
Deepak Ravikumar
drawikum edu
Deportment Electrical and Computer Engineering Purdue University
Kaushik Roy
kaustik-@purdue edu
Department of Electrical and Computer Engineering Purdue University
Reviewed on OpenReview: htips: // openreview. net/ forum? id- Kle. KeJJ99zY
Abstract
The availability of abundant data has catalyzed the expansion of deep learning vision algo- rithms. However, certain vision datasets cannot be publicly released due to privacy reasons. Releasing synthetic images instead of private images a common approach to overcome this issue. popular method to generate synthetic images using Generative Adversar- ial Networks (GANs) with Differential Privacy (DP) guarantees. However, GAN-generated synthetic images visually similar to private images. This is severe limitation, par- licularly when the private dataset depicts visually sensitive and disturbing content. To address this, we propose a non -on-generative framework, Differentially Private Image Synthesis (DP-IngSyn), to generate and release synthetic images for image classification tasks. These synthetic images: (1) have DP guarantees, (2) retain the utility of the private images, i.c., a model trained using synthetic images results similar accuracy as model trained on pri- vate images, (3) the synthetic images are visually dissimilar to private images. DP-ImgSyn consists of the following steps: First, teacher model is trained on the private images using DP training algorithm. Second, public images used as initialization for the synthetic im- ages which are optimized to align them with the private images. The optimization uses the teacher network's 's batch normalization layer statistics (mean, standard deviation) to inject information about the private images into the synthetic images. Third, the synthetic images and their soft labels, obtained from the teacher model, are released and can be deployed for neural network training on image classification tasks. Our experiments on various image classification datasets show that when using similar DP training mechanisms, our framework performs better than generative techniques (up to z= 20% in terms of image classification accuracy)
1 Introduction
Deep Learning has benefited from large statistical data available to the broad community. Large dalasets in the domain of image classification (Deng et al. 2009), object recognition (Lin et al.) 2014), language modeling (Lewis et al. 2004). and recommendation systems [Bennett et al. (007) have helped these domains make signilicant ad vanances However, there are cases in which data cannol be publicly released due to
The Torch implementation can found htttpp://gghhubb.cm///easaaaaaat http://faaaaaaaaoooaoeerr////aao

---

Published in Transactions on Machine Learning Research (04/2024)
Private Images (CelebA Hair)
Visually Dissimilar
DP Model Training
Batch Statistics
Synthetic Images
DP IygSyn
Public Images (65)
Visually
Similar
Figure 1: Visualization of the private images (CelebA-Hair), public images (Places365), and the DP-ImgSyn generated images (synthetic images). Dm-Img synthesized images are visually dissimilar to private im- ages. DP-ImgSyn is initialized with Places365 (public images/dataset) The public images and the batch statistics of a DP model (trained on the private images) are the inputs to DP-ImgSyn. DP-ImgSyn outputs synthetic images. A model trained on the synthetic images achieves x 99.5% the performance of model trained with DP on the CeiebA-Hair dataset (private images). Figure 6 in the Appendix contains more visualizations of other datasets.
privacy restrictions. To address this issue, several techniques have been proposed in literature which can be categorized into: model release methods (Abadi et al., 2016), and data release methods (Papernot et al. 2017 Model release methods train classifier the private dataset (dataset with privacy restrictions) and release the classifier with privacy guarantees. Data release methods release synthetic data with privacy guarantees instead of the private data.
Data release methods offer more ficxibility than model release methods (Tiasteyn & Faltings 2018 When releasing data, downstream users can choose any model architecture (i.e., ResNet, VGG, MobileNet, etc), unlike model release methods where the user is restricted to use the architecture of the released model. Furthermore, users combine data from different sources to build better models. Also, in the presence of newly available data, they can retrain the model new and past data avoiding catastrophic forgetting (Kirkpatrick et al. 2017). While data release methods offer improved flexibility, synthetic images generated using existing data release techniques are visually similar to private images. This poses significant challenges when the image content is visually sensitive and/or disturbing, for example with cortain medical image datasets, military datasets, content moderation image datasets, elc. This leads us to the question: can we achieve the same learning as with private images using visually dissimilar synthetic images? For example, can neural network learn to classify human faces when it is trained with synthetic images depicting places? In our work, we show that such learning is achievable. Formally, study the problem of releasing synthetic data for image classification tasks that satisfy the following properties:
1. The synthetic image must provide (e. Diff-efrenttial Privecy (DP) guarantees.
2. Retain the utility of the private images, i.e., model trained using synthetic images should result in similar classification accuracy the model trained on private images.
3. Be visually dissimilar to the private images. This critical to protect the users from viewing visually disturbing and sensitive content.
The visual dissimilarity between synthetic and private images is important for vision data depicting visually disturbing and sensitive content. This requirement is not satisfied by the existing approaches. In particular,
2

---

Published in Transactions on Machine Learning Research (04/2024)
existing approaches often use Generative Adversarial Networks (GANs Goodfellow et al. 2014] with DP
guarantees (DP-GANs Xie et al. 2018 Xu et al. 2019 Zhang et al. 2018 Xie et al. 2018 Cao et al. 2021 p
to generate synthetic images. While GAN-generated synthetic images have DP guarantees and retain the
utility of the private images, they are visually similar to private images (see Appendix Figure 7 Thus,
they do not satisly the third requirement of visual dissimilarity. In addition to this, training a GAN can
be challenging due to multiple issucs (Arjovsky & Bottouj (017) such as vanishing gradients (Arjovsky &
Bottou, 2017), mode collapse /Srvastavaa et al.] 2017 Brock et al.] 2019 Arrovssy al., 2017 Salimans
et al.
2018 Miyato et al., 2018 training instability (Brock et al. 2019 and convergence failure (Slimann
ct al.
2016
To address these issues, we propose a non-generative (or discriminative) approach for releasing synthetic images. Our proposed framework, Differentially Private Image Syuthesis (DP-IngSyn), trains a teacher model on the private (sensitive) dataset using a DP training algorithm, such as DP-SGD (Abadi et al., 2016). Next, public datasct selected for distillation (Hinton et al., 2015). This dataset used to distill the sensitive dataset. During distillation, we observe that misalignment between the public-privale dalasets can result in significant performance degradation. To address dataset misalignment, we propose alignment technique on the public dataset to improve the performance of our framework. Finally, we obtain soft labels using the teacher model and the aligned public dataset. The soft label is the teacher model output (logits) after applying the softmax function. The aligned public dataset (synthetic images) and the corresponding DP-teacher model generated soft labels, are released for training neural networks on image classification tasks. detailed explanation of the intuition behind our method is provided in Section 2
The synthetic images generated by DP-ImgSyn satisfy (1) privacy: the synthetic images have (c, DP guarantees because the teacher model is trained with DP guarantees, (2) utility: this is experimentally verified in Section 5 A model trained on the synthetic CelebA-Hai dataset (Liu et al. 2015 Long et al. 2021) achieves ze 99.5% the performance of model trained with DP on the CelebA -Hirr dataset, (3) the synthotic images are visually dissimilar to the private images. This is obscrved in Figure D which visualizes the private dalaset (CelebA-Hair), the synthetic images generated by DP-InggSyn (synthetic images), and the public dataset used for initialization (Places365, Zhou et al.) 2017). Further, we experimentally evaluate the synthotie images in terms of classification accuracy on MNIST, FashionMNIST, CelcbA-Hair, CclebA- Gender, CIFAR10, and ImageNette datasets. We observe that DP- Dm-Img reduces the misalignment between public and private distribution. This alignment improves performance up to R 17% on highly misaligned public-private dataset pairs. Moreover, our method achieves significantly better accuracy (up to z 20%) than statc-of-the-art generative methods using a similar DP training algorithm. We also observe similar trends when the teacher and the student models are of different architectures (see section 5.6 Note that the proposed technique is not a new DP mechanism. It new approach for releasing visually dissimilar images, leveraging dataset alignment to obfuscate sensitive data in public datasets.
Overall, our contributions summarized as follows:
We propose DI-Ingyn,, a non-generative image synthesis framework that generates DP- guaranteed dataset for public release. The synthetic images (1) are DPgguaranteed, (2) have similar utility to the private images, and (3) visually dissimilar to private images.
The DP-ImgSyn framework leverages the teacher model's batch normalization layer statistics to address the distribution misalignment between private and public datasets.
We show the effectiveness of DP-ImgSyn in image classification tasks on various vision datasets. Wo also show that ImgSnn performs better than state-of-tho-art generative methods when using a similar DP training algorithm.
2 Im-Sm: Why and How?
Why? We would like to emphasize that DP-IngSye shows that it is possible to learn to classify a dataset that looks very different from the training datasct. Note, that upon synthetic image release, it is up to the user to determine how to use them. Below, we provide examples showing why visual dissimilarity is important in certain classes of applications:
3

---

Published in Transactions on Machine Learning Research (04/2024)
(a) Latent Space
(b) Vanilla KD Student Latent Space
(c) Teacher Latent Space
Sy- Student Latent Space
0 Private Class 1
High Comfdence
Putlic Class Public 2
High Confidence
Private Class
High Confidence
@ DP-imggnn Class I
High Conficience
Privatu
DP-mmg5n Class
@ Public Set
Good SAA Aerpingg Soppan
t Public
Drvision Boundhay Dow Cord))
Daciston Boundary LLow Cord)
Optimize
Gecisbn Brllbeoflllooo Bollodary thew Cont
High Confiience
Conhdace
High Confidenco
High Conhdence
Figure 2: (a) Illustaates cartoon version of the decision boundary of the teacher model and how public images (gray) sample the latent space. (b) Shows the transfer of the decision boundary of the teacher model in (a) to the student model using public images to sample the latent space. The student in (b) learns a decision boundary that is different from the teacher leading to poor knowledge transfer. (c) Shows the effect of
of the public images with Dm-Img With DP-ImgSyn the latent space is sampled more effectively. Finally, (d) illustrates the transfer of the decision boundary of the teacher model to the student model, using synthetic images generated by Dm-ImgSyn to sample the latent space.
Defense Operational Security (OPSEC) Developing systems for the detection of classified military assets (e.g., new stealth aircraft) presents significant OPSEC challenges. Typically, only the team directly involved in the asset's development (e.g., the aircraft engineers) possesses the necessary security clearances. To develop a detection system, traditional data-sharing methods would necessitate granting security clearances to additional engincers or doclassifying sensitive imagery, both of which introduce socurity risks (Dressler) et al. 2015). DP-ImgSyn offers potential mitigation strategy by using synthetic images that are visually obfuscated to protect classified elements while stilll enabling the development. of effective detection models.
Medical datasets are bound by strict privacy regulations (HIPAA (Act 1996. PIPEDA (anley & Cavoukian 2000), and GDPR. (Regulation 2018)). While differential privacy (DP) offers robust guaran- tees, certain datasets may not. be suitable for release due to the sensitivity of the body regions depicted
(Dnng et al. 2006 Abbas et al.] (022). DP-Ing can facilitate the use of such datasets by generating synthetic images that preserve the essential medical information for model training but visually obscuring
identifiable details.
Content Moderation DP-ImgSyn potentially addresses privacy and ethical concerns with content mod- eration by enabling the generation of visually dissimilar synthetic dalasels for model training, safeguarding individuals in the original data and mitigating exposure of human moderators to such content (Spence et al., 2023). Other applications include automated parental control and movic age rating systems (Akyon & Temizel 2022).
How? The proposed DP-ImgSyn method aims Dearn/transfer the decision boundaries between the teacher and the student model. This is done by sampling the latent space of the teacher model using the public images and having the student model match the decision boundary of the teacher using soft labels. This implies that latent space sampling needs to be effective. The challenge is that public images may not effectively sample the latent space. Thus, our proposed optimization perturbs the public images such that the latent space can be effectively sampled. This leads to better knowledge transfer between the teacher and the student model. Figure 2 (a) illustrates cartoon version of the decision boundary of the teacher model and how a public datasct (gray) samples the latent space. Figure 2 (b) shows the transfer of the
decision boundary of the teacher model in 2 (a) to the student model using the public images to sample
the latent space. We see that the student in 2 (b) learns a decision boundary different from the teacher leading to poor knowledge transfer. Figure 2 shows the effect of optimization of the public images with Dynnng With DP-IngSyn the latent space is sampled more effectively. Finally, Figure 2 (d) illustrates the transfer of the decision boundary of the teacher model to the student model, using synthetic images generated by Dm-Ing to sample the latent space, The experimental results in Section are in line with the DP-Ing intuition. For visualizations on deep neural networks refer to Appendix A.3
4

---

Published in Transactions on Machine Learning Research (04/2024)
3 Background and Related Work
This section briefly introduces the dcfinition of Differential Privacy (DP), followed by an overview of ap- proaches for releasing synthelic images using generative techniques. Differential Privacy (DP) was intro- duced by Dwork et al. (2006). A randomized algorithm A is said to be (e,6) differentially private, if for all S _ Range(A) and for all datascts y E Domain(A) such that || - vlli S 1:
Pr[A(x) E S) S e' Pr[A(y) E S] +
In the case of deep learning, A is the training algorithm, and S is the subset of all possible model parameters that can be output from the training process.
In this paper, consider the task of generating visually dissimilar synthetic images with DP guarantees. While, semi-private learning (Bic et al. 2022 Ganesh et al. 2023) might seem related to our work, there are fundamental differences. We briefly discuss these differences here. Semi-private learning leverages public data to improve the privacy bounds when learning from private data. Specifically, the private model's parameters are updated with public data to improve privacy guarantees and privacy-utility trade-off. In our work, public data used for image synthesis and not during the training of the DP neural network. Hence, we do not compare DP-ImgSyn with semi-private learning techniques. Next, we present generative techniques for model and data release.
As far as generative techniques are concerned, they leverage GANs or other generative models by sharing DP- trained models, embeddings, or generating images for releasing datasets while maintaining privacy. Multiple research articles (Xie et al. 2018 Xu et al., 2019 Zhang et al. 2018 have proposed training GANs with DP for image synthesis using DP-SGD (Abadi et al. 2016) under various contexts and domains. The authors of GS-WGAN (Chen al., 2020) adopt Wasserstein GAN (WGAN, Arjovsky et al. 2017) and propose using Wasserstein-1 loss for training. They show that such an approach can distort gradient information more precisely; thus GS-WGANG generate more informative samples. The authors of DataLens (Wang ct al. 2021 leverage GANs to reduce the gradient noise using gradient compression. In a similar direction, to improve information capture from the gradient, the authors of DPGEN (Chen et al. 2022) deploy an enorgy-guided network. They train on synthetic data to indicate the direction of the actual data distribution via the Langevin Markov chain Monte Carlo sampling method. However, since all of these techniques rely on GANs, they are susceptible to training instability of GANs. To address the issues with GAN-based methods, the authors of DP-MERF (Harder et al., 2021) synthesize images by taking advantage of random feature representations of kernel mean embeddings, while the authors of P3GM (Takagi et al. 2021 uSe variant of the private variational autoencoder.
So far, we have discussed generative DP techniques; next, we discuss discriminative DP techniques. Private Aggregation of Teacher Ensembles (PATE, Papernot et al. 2017 2018) divides the training data into disjoint sets and assigns them to multiple classifiers (teachers). The teachers are queried with either public or GAN- generated images obtain the corresponding soft labels (Long et al., [021 Jordon et al. 2019). The student network is trained using public or GAN-generated images and their soft labels using knowledge transfer (Hinton et al. 2015). To maintain the privacy of multiple teacher models, the soft labels of all the teachers are aggregated, and noise is added before they are released for training the student. When PATE employs a GAN for image generation, it encounters the aforementioned challenges related to GANs. When utilizing public images, it struggles to address situations where public and private images are not aligned. Thus, to address the challenges of generative and discriminative data release techniques, we propose a new framework, Dm-Img,,, for synthetic data relcase.
4 Im-Sm: Differentially Private Image Synthesis
This section introduces our approach's specifics, as Figure 3 illustrates. Our method consists of three steps: 1) train a teacher model with a DP training algorithm on the private images, 2) perform optimization on the
Pllaae note, DPGEN Chen
2022 privacy guarantees compromised due conceptual errors reported Dockhorn
et. al. 2022
5

---

Published in Transactions on Machine Learning Research (04/2024)
Private Image
5
Public Image
Synthetic
Image
Conn
Cony
Conv
GN
BN
BN
Backprop.
Backprop.
Fonward
Com
Comn
Conv
Loss
Loss
Soft Labels
Step 1: DP Training
Step 2: Image Synthesis
Step 3: Soft Label Generation
0 - k iter.
Figure 3: Overview of DD-ImggSnn (1) Train teacher model using DP-training scheme. Capture the batch statistics of the model on the private dataset using the proposed DP guaranteed technique. (2) Perform the proposed public dataset alignment to obtain a better-aligned synthetic public dataset. Note that, the public image is optimized (updated), while the parameters of the model stay constant. (3) Generate soft labels. The synthetic images and their soft labels can be publicly released.
public dataset to align it to the private dataset and 3) generate soft labels for the aligned synthetic public dataset. Finally, the synthetic images and their corresponding soft labels are publicly released to train a student network.
4.1 DP Teacher Model Training
The privacy budget of our method is the combined privacy budget for the teacher model training Etroin and the statistics capture process Ebotch-stats- Firstly, our method involves training the teacher model utilizing a DP training algorithm. Note that the teacher model is not trained with standard SGD but uses a DP training a algorithm to ensure privacy. We select Differentially SivvateSttochasicc Gradient Descent (DP-SGD, Abadi al. 2016) as the DP training algorithm. DP-SGD trains neural network with (e, 6) guarantee. Similar standard SGD, the algorithm converges in multiple training steps. At cach training step t, DP- SGD computes the gradient (x) of the loss function with respect to the model parameters for training image x. Then, it clips each gradient vector to have a maximum l2 norm of C. That is, the gradient vector is replaced by g/ (1, IIglla/C). The clipping ensures that if ||g]|2 < C then g is preserved, whereas if 1ll1e2 > C, gets scaled down to be of norm C. Thus, the contribution of each data point to the batch gradient bound by constant C. Noise is added to the gradient (x(x) + +V C²C²T and the descent step 0+1 = A, NeTt is performed, with ne learning rate. After T itcrations, outputs the (Ctrain; 8D-DP teacher model.
Next, the synthesis (a.k.a dataset alignment) requires the teacher model's batch statistics. These are obtained from the batch norm layer of the teacher model. However, batch norm layers cannot be used for DP training, because the batch norm computes the mean over multiple training data points. Thus, per sample gradient cannot be obtained during training. This implies that the gradient norm cannot be bound, so we cannot provide a DP guarantee. To address this issue, we propose the following DP-guarantcod approach batch statistics. First, we use group norm layers instead of batch norm layers. Next, we train the teacher model using the previously described DP-SGD. Once trained, capture the input to all group norm layers,
obtain

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm 1: DP Image Synthesis
Input: DP-trained teacher model M; number of optimization iterations; synthesis learning rate Ysyni batch statistics p. a for the private set; batch of public images xp
Output: &, one batch of aligned synthctic images
& +- 2p
- Target labels for batch 2, uniformly distributed over all the classes
for i i= 1,,2..., k do
4
(()) + M())
M22)
R s- C-Riotaif y, u. o,
// Compute the loss from Equation
VaR Backward pass
Update f +- 2-- Ysn V R
return &
ign- We clip the imput fgn to have maximum 22 norm of C' and add noise, ign ign +N The noisy input ägn used to calculate the batch statistics of the private set. This process has the same DP guarantee the Gaussian Mechanism (Dwork et al.] 2014) since it uses the Gaussian DP mechanism described in Dwork et al. (0014). Note that obtaining the batch statistics this way consumes some of the privacy budget allocated for training. Thus, when we report our results, the privacy budget is for the combined training and statistics capture process: 6 = composition((raus (batck-stats). We use the accountant implementation from Gopi et al. 2021 to calculate the upper bounds on privacy. To ensure correctness, we also verify the upper bounds on privacy using both Opacus (Youselpour al.] 2021) and Gopi al. (2021) 's implemennaatinn.
4.2 DP Image Synthesis
Image synthesis (a.k.a dataset alignment) optimizes the public dataset to align with the private dataset to ensure that the synthesized (or aligned) images will have good distillation performance. To perform image synthesis, we collect the layerwise batch statistics of the private datasct using the DP-guaranteed technique described in Section 4.1 The batch statistics consists of the mean / = ". ",z.] and the variance a = [0i,... [L] from the all the layers of the teacher model M.. Let & denote a batch of synthetic images and xp denote batch of data sampled from a public dataset D,. Algorithum [ summarizes the DP image synthesis process. We initialize ê with zp, this corresponds to Line in Algorithm [ For each data point in ét we assign target label y (Line 2 in Algorithm D.. The target labels are uniformly distributed over all the classes, and the assignment is such that we have the same number of images for each class. This step of generaling the labels is independent of the private dataset to ensure privacy. For the exact implementation refer to Appendix A...3
The next step is to perform & itcrations of optimization corresponding to Lines 3 These itcrations optimize & to align with the private set. Each iteration consists of a forward pass of & through the DP-trained teacher model M (Line I in Algorithm [ The forward pass is used to obtain the layer-wise batch statistics for ât (μ(2), o (.)). The batch statistics and the generated label y are used to calculate the loss described in Equation [Line]] in Algorithm D. The gradient of the loss R with respect to & (V R) is calculated using back-propogation and is used to update the image A (Lines 6 7 in Algorithm 1 At the end of l update
steps, we have synthesized one batch of aligned images Since the teacher model is DP-trained, and the
private dataset batch statistics are obtained with a DP guarantee, image synthesis is also D-gguaranteed.
The loss used to guide the optimization is critical. The total loss Reotal consists of the following terms: feature loss Reeature classification loss Rolassif, total variance loss ReD and l2 norm loss R2, The sum of total variance loss R2, and the l2 norm loss R1, are referred to as prior loss. Next, we define each of these losses. The feature loss Reeetuure computes the distance between the batch statistics of the private dataset and the synthotic set & and is given by the following equation:
Reeature (u, o, ((8),
[|xx(0) - mil1 + [(ar(() - o1)
(1)

---

Published in Transactions on Machine Learning Research (04/2024)
Where è is the synthesized-aligncel image, (+) and (ā) are the batch-wise mean and variance estimates of feature maps corresponding to the hsh layer when & is fed to M, and 14 and dr are the jsh layer batch statistics obtained from the private dataset described in Section 4.1 The classification loss Relassif is the cross-entropy loss between the teacher output and the target label y and defined as:
Relassi (2, v) = (()
(2)
where & is the cross-entropy loss, (+) is the output of the teacher model M when & is fed as input, and is the target label. The total variance loss Riv ensures sharp transitions in the synthetic image and restricts the adjacent pixels to have similar values. It is defined as:
(+) = E((eisH1 - Big)² + (f141) - 21
(3)
The l2 norm loss is employed to encourage the image range to remain within target interval rather than diverging. The la norm loss R1, for the & is defined as:
(+) ||
(4)
The total loss is the sum of the aforementioned losses:
Reotal (ft, U. o) a y feature (2, 0) + a. Redassi (2, + or Reo + 01 Rig (f)
(5)
Each loss term is multiplied by corresponding scaling factor Ory City Olg. The teacher model is not updated during back- propagaopa,,,, and only is optimized. After k iterations, we obtain the synthetic DP images &.
4.3 DP Image Release
After synthesizing the images, we obtain their corresponding soft labels. The synthetic images are fed to the DP-trained teacher model M, and the corresponding soft labels ğ pM(() are recorded. Because the teacher model is trained with DP-SGD, querying the teacher to obtain the soft labels does not impose privacy risk. The synthetic images é, along with their soft labels g, are publicly released. The student model is trained on the synthetic images & and their corresponding soft labels á using KI-divergence:
min Σ (T))
(6)
KL refers the Kullback-Leibler divergence, P5( is the output (soft labels) of the student model when the synthetie image & is given as input. T is scaling temperature value. The synthetic images and their corresponding soft labels are publicly released and can be used to train any neural network. Note that, after synthetic image generation, the public images Up are no longer needed. Thus, the public images will not be needed during the network training, as the network is trained using synthetic images only.
5
Experimental Evaluation
5.1
Experimental Setup
To evaluate our proposal, we use the same vision datasets as previous works; specifically, we use MNIST (Le-
Cun et al. 1998), FashionMNIST Xiao al. 2017)) CIFAR-10 (Krizhevsky et al.) 2009),
ImageNette (ima
2018), CelebA-Hair (Long al. 2021 Liu et al. 2015 CelebA-Gender (Long et al.
2021 Liu et al. 2015).
TinylmageNes (Li et al. 2015) P Places365 (Zouu al. 2017), LSUN (Yu et al. 2015
and Textures CCimpoi
ct al. 2014). We use the following network architectures: ResNet18 (He et al.
2016). ResNet34 (He et al.
2016), VGG11 (Simonyan & Zisserman, 2014), MobileNet V2 (Sandler et al.] 2018), and ShuflleNet V2 (Ma
et al. 2018). All the models were trained till convergence or privacy budget exhaustion. The detailed hyper-
parameter scttings for image synthesis and model training, computational resources, and dataset statistics are reported in the Appendix. We perform the experiments described the following sections to evaluate our proposal thoroughly.
8

---

Published in Transactions on Machine Learning Research (04/2024)
94.0
Accuraccy
140
94.0
Accuracy
1.95
Accuracy (%)
93.5
Total Loss
100
120
Total Loss R. Raai
Accuracy (%)
TV Loss
1.90
TV Loss Rec
1.85
93.0
93.0
1.80
92.5
60
40
1.75
92.0
20
92.0
1.70
91.5
0
91.5
1.65
15
20
40
60
100
15
20
40
60
100
Number of optimization Iterations
Number of optimization iterations
(a) Accuracy and Riotal versus
(b) Accuracy and Riv versus k.
Figure 4: Plots for student model accuracy, total loss Reotal and total variance loss Rev versus the number of optimization iterations k. The left axis is for accuracy, and the right is for loss Riotat and Rio respectively. Results suggest early stopping is necessary optimize accuracy. For this plot, DP-ImgSyn is initialized with the TinylmageNet datasct, and the ResNet34 is the teacher model.
5.2 Number of Optimization Iterations
k
This section studies the effect of the number of optimization iterations on image synthesis and performance.
Experiment. We train a ResNet34 till convergence on the private dataset CIFAR-10. Next, we perform the aligument optimization detailed in Section 4.2 for various numbers of iterations (i.e., in Dm-Img TinylmageNet as the public dataset. The
Algorithm D ranging from 0-100. For the alignment, DP-ImgSyn generated dataset is used to train ResNet18 student model. The student model accuracy and losses versus the number of optimization iterations k are reported. We use privacy budget c < oc to isolate all variables. The results are visualized in Figure 4
Results. Figure 4a illustrates the accuracy of the student model and the total loss Riotal versus the number of optimization itcrations k. This is visualized with two y-axes: the left axis for accuracy and the right axis for the total loss Rtotal- Note that the student model accuracy peaks around the optimization by increasing reduces student model performance. The plot in Figure 4b explains the reason for this behavior. Figure 4b plots the accuracy of the student model and the total variance loss Rev versus the number of optimization iterations k. This is visualized with two y-axes, left for accuracy and the right for total variance loss Rrv- The loss Res expresses the image prior. The image prior ensures no sharp transitions in the synthetic image. It is used in literature (Dosovitskiy & Brox 2016 Mahendran & Vedaldi, 2015 Nguyen al.] 2015 Simonyyn et al., 2014) as a proxy for how natural synthesized images are. This plot suggests that the images become more artificial synthesized as we optimize past certain threshold. It is observed that around 20 iterations, the Rrv loss starts increasing, and the accuracy starts decreasing.
20 iterations. Continuing
Conclusion. Minimizing Riotal does not guarantee optimal image prior (~ R..). However, ReD significantly impacts student model accuracy. Thus, early stopping is necessary to optimize student model performance.
5.3 Privacy and Performance
We evaluate the performance of DP-ImgSyn with iterations and iterations on various vision datasets as public and private datascts.
Experiment. First, select private dataset. We train teacher ResNet18 model on the private dataset using DP-SGD and capture batch statistics described in Section 4.1 This results in ResNet18 teacher model with batch statistics having privacy budget of c. Next, we select public datasct and generate the synthetic datasel described in Sections 4.2 and 4.3 A student ResNer18 model is trained on the hesized dataset with various public datasets as initialization. We report the accuracy results for privacy budgets E {1, 10} on MNIST, FashionMNIST, ColebA-Hair, and CelebA-Gender as private datascts. We select public datasets whose size is the same or exceeds the size of the private dataset. This ensures that the synthesized dataset has the same size the private dataset. When the public dataset. is larger than the

---

Published in Transactions on Machine Learning Research (04/2024)
Table 1: Accuracy for ResNet18 DP-teacher model with e E {1, for MNIST, FashionMNIST,, CelebA- Hair, and CeldbA-Gender.
MNIST
FashionMNIST CelebA-Hair CelebA-Gender
1
86.87%
76.27%
79.95%
91.02%
= 10
96.30%
81.88%
81.74%
92.35%
private set, we randomly sample from the public set to obtain a subset of the same size as the private set. This public subset is used for the synthesis. The number of classes of the public dataset is not required to be the same as the private set, because we use the soft labels of the teacher model. Note that, there is no one-to-one correspondence between public and private set images. Appendix reports an ablation study of the various loss terms.
Results. Table reports the accuracy of the ResNet18 DP-teacher model trained on varions datascts. Table 2 summarizes the accuracy results for MNIST, MNIS,, CelebA-Hair, and CelebA-Gender as private datasets and lmteNet,,,,, Places365, FashionMNIST,, MNIST, and LSUN as public datasets. Table 2 reports the performance of the student model when performing O iterations, reported as DP-ImgSyn(O), The student model is trained on the DP-IngSyn generated images and the test set is the private test set. We report the performance with early stopping at keep iterations as DP-ImgSyn(Kep)) and the optimal iterations to stop as kopt.
Conclusion. From Table 2 observe that, on average, when the private datasets are aligned over various public datasets, DP-ImgSyn(k) performs similar to DP-ImgSyn(O). For some datasets, we observe hopt 0, i.e., the proposed aligument process does not improve performance much. However, when the datasets are misaligned, like in the case of FashionMNIST and MNIST, we see DP-ImgSyu(k) performs significantly better (=17% improvement in student model accuracy, for FashionMNIST private dataset with MNIST public dataset initialization for C (1, 10)). Morcover, in Table 2 we see that for each private dataset, we use multiple public datasets initialization and they result in similar accuracy. Thus, the impact of the domain gap (accuracy) between public and private images is minimal. For more details about the interference of the private and public images refer to Appendix A.4
5.4 Comparison with other Techniques
Experiment. This section compares proposal with previous approaches: DP-GAN (Xie et al. 2018
DP-MERF (Harder et al., 2021), P3GM (Takaii et al.) (021), DataLens (Wang et al. 2021) and G- G-PATE (Jordon et al., 2019). For a fair comparison, we compare with techniques that use similar DP-training
schemes (i.e., variants of DP-SGD). Note, that we exclude comparison with DPGEN (Chen et al. 2022 because privacy guarantees are compromised due to errors as reported in Dockhorn et al. (2022). The results for other techniques are the best accuracy results reported in prior publications. For our results, we report best performance results from Table 2 Specifically, for
1, we use public dataset initialization FashionMNISTT, MNTST, Places, and Places for the private datasets MNIST, FashionMNIST. CelebA-Hair, and CelebA-Gender respectively. For e = 10, we use as public dataset initialization Tinylm- ageNet, TinyImageNet, LSUN, and Places for the private datasets MNIST, MNSHI, CelebA-Hair, and CelebA-Gender, respectively. Regarding Dm-ImgSyn initialization, we could consider the use of GAN- generated images to initialize the DmgSmn. However, doing this would violate the property of visual dissimilarity between the synthetic and the private images, since GAN-generated images are visually similar to the private images. Therefore, in this paper, only consider the use of public images and noise (see Appendix A.2 for more thorough discussion noise) for the DP-ImgSyn initialization.
Results. Table compares the performance of the proposed technique with state-of-the-ar techniques.
Conclusion. We observe that our proposed method significantly outperforms generative techniques that use similar DP training schemes (up to z 20%, for both CelebA-Hair and CelcbA-Gendor for c <= 1). Wo also provide results when using stronger privacy 0.2 in Appendix A.5 for interested readers, where we see similar results.
10

---

DataLens refers to Published in Transactions on Machine Learning Research (04/2024)
Table 2: Comparative Table with e E {1, 10} for MNIST, FashionMNISTT, CelebA-Hair, and CelebA-Gender as private datasets using TinylmageNet, Places365, FashionMINIST, MNIST, and LSUN public datasets. Results are mean + std over three different seeds. The models are trained on the synthetic images generated by DP-ImgSyn (training set) and evaluated the test set of the private dataset (testing set).
Private Dataset
Public Dataset
DP-ImgSyn(0)
JpP-ImSSn((eepp)
Kexp
Kopt
TinyImageNet
85.83 0.13
85.98 + 0.06
10
10
Places365
85.00 + 0.30
86.01 + 0.22
10
10
PashionMNIST
85.56 + 0.32
86.24 +: 0.03
10
10
MNIST
TinyImageNet
92.97 + 0.65
94.03 + 0.64
10
10
10
Places365
92.63 + 0.23
93.74 + 0.18
10
10
PNshionMNIST
93.61 0.37
93.90 + 0.30
10
TinyImageNet
74.99 +: 0.21
74.93 = 0.02
0
6=
Places365
75.08 0.15
74.94 + 0.20
MNIST
51.58 2.28
68.38 + 0.34
10
MNIST
TinyImageNet
79.04 0.04
78.71 = 0.20
10
Places365
78.73 + 0.14
78.80 + 0.05
MNIST
54.78 + 1.15
71.51 + 1.49
10
10
LSUN
79.89 0.08
79.42 + 0.14
Places365
79.91 +:2 0.08
79.50 = 0.15
CelebA-Has Hair
LSUN
81.31 + 0.04
79.28 + 0.20
0
10
Places365
$1.33 + 0.12
78.73 + 0.44
0
LSUN
89.91 + 0.15
89.17 + 0.26
0
Places365
90.06 +: 0.05
89.03 = 0.19
0
CelehA-Gender
LSUU
90.99 + 0.16
89.90 = 0.26
0
10
Places365
91.22 + 0.04
89.27 = 0.99
Table 3: Comparison Table with stato-of-the-art techniques for c E {1, for MNIST, FashionMNIST, CelebA-Hair, and CelebA-Genderr Results for DP-IngSyn are mean over three different seeds. The best- performing framework is highlighted in bold, and the second-best is underlined. DP-GAN refers to [ie al.)
(2018), DP-MERF refers to Harder al. (2021), P3GM refers to Takagi al. 2021
Wang et al. 2021 and G-PATE refers to Long et al. (2021)
Dataset
DP-GAN
DP-MERF
P3GM
DataLens
G-PATE
DP-IngSyu (ours)
40.36%
63.67%
73.69%
71.23%
58.80%
86.24%
MNIST
10
80.11%
67.38%
79.81%
80.88%
80.92%
94.03%
10.53%
58.62%
72.23%
64.78%
58.12%
75.08%
FashionMNIST
60.98%
61.62%
74.80%
70.61%
69.34%
79.04%
34.47%
44.13%
45.32%
60.61%
49.85%
79.91%
CelebA-Hair
10
39.20%
52.25%
44.89%
62.24%
62.17%
81.33%
53.30%
59.36%
56.73%
69.96%
67. .7.0%%
90.06%
CelebA-Gender
10
52.11%
60.82%
58.84%
72.87%
68.97%
91.22%
5.5
Beyond Generative Methods
This section presents results on higher resolution (224 x 224) and more varied datasets with which generative techniques often have difficulty.
11

---

Published in Transactions on Machine Learning Research (04/2024)
Table 4: Comparative Table with с E {0, for MNIST, and CelebA-Gender as private datasets using TinylmageNet, Places365, FashionMNIST,, and LSUN and Places365 as public datasets respectively. Results are mean std over three different seeds. The models are trained on the synthetic images generated by DP-ImgSyn (training set) using ResNet18 as teacher model. The student model architecture is VGG11, MobileNet V2, ShuflleNetV2, and 18 The student models are evaluated on the test set of the private dataset (testing set).
Student Model Architecture
Private Dataset
Public Dataset
VGG11
MobileNet
V2
ResNet118
TinylmageNet
85.56
84.34 0.70
85.15 =
85.98 0.06
Places365
85.65 = 0.48
84.02 0.33
85.28 = 0.22
86.01 + 0.22
FaSHionMNIST
86.37 = 0.42
84.95 0.28
85.91 =
86.24 + 0.03
MNIST
TinylmageNet
94.37 = 92.08 0.62
94.04 = 0.09
94.03 0.64
Places365
93.91 = 0.26
91.75
0.71
93.22 + 0.21
93.74 + 0.18
MNshionMNIST
94.70 = 0.18
92.66 + 0.95
92.91 = 0.91
93.90 + 0.30
LSUN
88.32 = 0.32
88.31
0.30
89.44 =
89.17 0.26
Places365
88.61 = 0.38
88.49 0.14
89.16 = 0.44
89.03 + 0.19
CelehA-Gender
LSUN
89.36 = 0.55
88.85
0.25
89.54 = 0.28
89.90 + 0.26
Places365
89.40 = 0.50
87.85 + 0.26
89.02 = 0.02
89.27 + 0.99
Experiment. We consider ImageNette and CIFAR-10 private datasets and use the proposed DP- ImgSyn technique. For ImageNette, we use a resolution of 224 x 224 with c= 105, initialized with Textures as public dataset; for CIFAR-10, we select = 10, initialized with TinyImageNet as public dataset. Note that generative methods do not report results on these datasets.
Results. ResNet18 student model trained on the DP-ImgSyn generated dataset for ImageNette (224 224) achieved 39.38% accuracy on the test set, while the teacher model achieves 43.26% accuracy. Similarly, for CIFAR-10, ResNet18 student trained DP-Img generated dataset achieved 45.66% accuracy the test set, while the teacher model achieves 42.95% accuracy.
The ImageNette results suggest that ImgSnn is not limited by the image resolution, unlike existing generative techniques. ImageNette and CIFAR-10 results indicate that the DP-ImgSyn technique is better suited to more complex and higher-resolution datasets when compared to generative techniques.
Conclusion.
5.6 Generalization Other Network Architectures
This section presents results with various network architectures for the student network to evaluate whether our released dataset can be used to train any network architecture.
Experiment. We use Dm-IngSyn to generate the syuthetic images using a ResNet18 as the teacher model. The student model architecture is VGG11 (Simonyan & Zisserman, 2014). MobileNetV2 (Sandler et al. 2018), ShuffleNet V2 (Ma et al.] 2018), and ResNet18 (He et al.) 2016). For 1, 10 we use MNIST and CelebA Gender the private dataset initialized with TinylmageNet, Places365, FashionMNIST, and LSUN, Places365 respectively.
Results. Table 4 summarizes the results across the various student architectures. We observe that the accuracy is similar across the architectures (VGG11, MobileNetV2, Shuffleett22), and similar to ResNet18 which has the same architecture as the teacher model.
Conclusion. The outcome of this experiment is that the synthetic images can be used train any student network architecture. Thus, ImgSSn is not restricted by the teacher model architecture.
12

---

Published in Transactions on Machine Learning Research (04/2024)
6 Conclusion
Deep neural networks are statc-of-the-art solutions for various tasks in multiple domains, but they require significant amount of data for training. However, certain data camnol be publicly released due to privacy restrictions. This paper presents a discriminative approach (DP-ImgSyn) for releasing synthetic images that have DP guarantees, maintain the utility of the private images, and are visually dissimilar to the private images. The proposed framework leverages dataset alignment to obfuscate private sensitive images in public images. This higgnment/yythhesis process improves the performance of even highly misaligned public-private dataset pairs. We observe 17% improvement in the performance of highly misaligned datasets. Also, we show that the non-generative DP-ImgSyn approach significantly outperforms (up 20% improvement in classification accuracy) generative techniques using similar DP-training schemes. Further, we present results higher resolution (224 x 224) and more varied datasets with which generative techniques often have difficulty. Our findings suggest discriminative (a.k.a non-generative) approaches might better suit synthetic dataset release. However, further research is needed to identify the limits of discriminative and generative techniques.
7
Acknowledgments
This work was supported in part by, the Center for the Co-Design of Cognitive Systems (COCOSYS). DARPA-sponsored JUMP2.0 center, the Semiconductor Research Corporation (SRC), and the National Science Foundation.
References
Imagemette a subset of 10 easily classified classes from the imagenet dataset. attts://ggtthin ctt.stasaiiooomaaaaaa//aaoaaa imagenettel 2018.
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Iyy Mironov, Kunal Talwar, and Li Zhang, Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on compuler and communications security, pp. 308 318, 2016.
Tariq O Abbas, Mohamed AbdelMoniem, and Muhammad EH Chowdhury. Automated quantification of penile curvature using artificial intelligence. Frontiers in Artificial Intellignnce, 5:954497, 2022.
Accountability Act. Health insurance portability and accountability act of 1996. Public law, 104:191, 1996.
Fatih Cagatay Akyon and Alptekin Temizel. Deep architectures for content moderation and movie content rating. arXiv preprint (rXiv:2212.04533, 2022.
Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations, 2017. URL
id=1i44 [d=Hik__qwxxe
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. International conference on machine learning, pp. 214-223. PMLR, 2017.
James Bennett, Stan Lanning, al. The netflix prize. In Proceedings of KDD cup and workshop, volume 2007, pp. 35. New York, 2007.
Alex Bie, Gautam Kamath, and Vikrant Singhal. Private estimation with public data. Advances in Neural Information Processing Systems, 15:16653-18666, 2022.
Andrew Brock, Jeff Donahuc, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https://openreriee. het/forrumiid--B1zaq000009.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Tianshi Cao, Alex Bie, Arash Vahdat, Sanja. Fidler, and Karsten Kreis. entially private generative models with sinkhorn divergence. Advances in Neural Information Processing Systems, 34:12480 12492, 2021.
Don't t generate me: Training differ-
Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. Gs-wgan: A gradient-sanitized approach for learn- ing differentially private generators. Advances in Neural Information Processing Systems, 33:12673 12684, 2020.
Jia- Wei Chen, Chia-Mu Yu, Ching-Chia Kao, Tzai-Wei Pang, and Chun-Shien Lu. Dpgen: Differentially private generative energy-guided network for natural image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 8387-8396, 2022.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEFE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pottern recognition, pp. 248-255 Icce,
2009.
Jing Deng, MARGARET A HALL-CRAGGS, Pellorin, Alfred D Linney, William R Loes, Charles H Rodeck, and ANDREW TODD-POKROPEK. Real-time three-dimensional ultrasound visualization of erection and artificial coitus. international journal of andrology, 29(2):374-377 379, 2006.
Tim Dockhorn, Tianshi Cao, Arash Vahdat, and Karsten Kreis. Differentially private diffusion models. arXiv preprint 2r1: 09929, 2022.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4829 4837, 2016.
Judson C Dressler, Christopher Bronk, and Daniel S Wallach. Exploiting military opsec through open-source vulnerabilities. In MILCOM 2015-2015 IEEE Militory Communications Conference, pp. 450 458. IEEE, 2015.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265 284. Springer, 2006.
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations of differential privacy. Foundations and Trends0 in Theoretical Computer Science, 9(3 4):211 407, 2014.
L eon Bottou. Online learning and stochastic approximations. On-linelearning in neural networks, 17(9): 142, 1998.
Arun Ganesh, Mahdi Haghifam, Milad Nasr, Sewoong Oh, Thomas Steinke, Om Thakkar, Abhradeep Guha Thakurta, and Lun Wang. Why is public pretraining necessary for private model training? In International Conference on Machine Learning, pp.
PMLR, 2023.
Pouget- Abadic Goodfellow, Xu Mirza, Ozair Warde-Farley, et al. Goodfellow i. Pouget-Abudic J., Mirza M., Xu B., Warde Farley D., Ozair S., Courville A., Bengio Y., Generative adversorial nets, Advances in neural information processing systems, 27, 2014.
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems, 34:11631-11642, 2021.
Richard Haddad, Ali N Akansu, et al. A class of fast gaussian binomial filters for speech and image processing. IEEE Transactions Signal Processing, 39(3):723 1991.
Frederik Harder, Kamil Adamczewski, and Mijung Park. Dp-merf: Differentially private mean emboddings with randomfeatures for practical privacy-preseriin data generation. In International conference on artificial intelligence and statistics, pp. 1819-1827. PMLR, 2021.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXio preprint arXiv:1503.00 02531, 2015.
James Jordon, Jinsung Yoon, and Mihacla Van Der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees. In International conference on learning represenlations, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar preprint arXiv:1412. 6980, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114((3))3521-355) 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278-238 2324, 1998.
David Lewis, Yiming Yang, Tony Russcll-Roso, and Fan Li. Rev1: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361 397, 2004.
Fci-Foi Li, Andrej Karpathy, and Justin Johnson. Tiny imagenet visual recognition challenge. http://
cs231n. ss23in.stanford edu/ tingenneeeeneet ceneeenett-00.... 200 200. 2015.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCO 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Yunhui Long, Boxin Wang, Zhuolin Yang, Bhavya Kailkhura, Aston Zhang, Carl Gunter, and Bo Li. G-pate: scalable disferentially private data generator via private aggregation of teacher discriminators. Advances in Neurol Information Processing Systems, 34:2965 2977, 2021.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116 116,131, 2018.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188-5196, 2015.
J Manley and A Cavoukian. The personal information protection and electronic documents act (pipeda). Priv. gc. ca, 2000.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gen- erative adversarial networks. In International Conference Learning Representations, 2018. URL
https:///ppenrevier.. wwt.://eewe..eeeeeoooooreeeeeeee
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are casily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 427-436, 2015.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Nicolas Papernot, Martin Abadi, Ülfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. In International Conference on Learning Hepresentations, 2017. URL
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with PATE. In International Conference on Learning Representations, 2018.
URL http//ppwrrriee.eeetfoooormm///a
Adam Pasake, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http://papers. http:///pappre.nnriip..cco/ooo/ neurips..c//paper wwwforeaceeeroeeeeeeeeeee ormance -eoo--eearning--iee library.b..rrr.. pdr
9015 aall---yorcc----impp
General Data Protection Regulation. General data protection regulation (gdpr). Intersoft Consulting, Ac- cessed in October, 24(1), 2018.
Tim Salimans, Tan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving GANs using optimal transport. In International Conference on Learning Representations, 2018. URL https:// www.iewrwwee//rrrrr id=rkQkBnJAb
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenccks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 4510 4520, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXii:1409.1110 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR). ICLR, 2014.
Ruth Spence, Antonia Bifulco, Paula Bradbury, Elena Martellozzo, and Jelfrey DeMarco. The psychological impacts of content moderation on content moderators: A qualitative study. Cyberpsychology: Journal of Psychosocial Research on Cyberspace, 17(4), 2023.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Voegan: Reducing mode collapse in gans using implicit variational learning. Advances in neural information processing systems, 30, 2017.
Shun Takagi, Tsubasa Takahashi, Yang Cao, and Masatoshi Yoshikawa. P3gm: Private high-dimensional data release via privacy preserving phased generative model. In 2021 IEEE 37th International Conference on Data Engincering (ICDE), pp. 169-180. IEEE, 2021.
Aleksei Triasteyn and Boi Faltings. Generating artificial data for private deep learning. arXie preprinl arXim:1803.031 08,4448888888,88 2018.
Boxin Wang, Fan Wu, Yunhui Long, Luka Rimanic, Ce Zhang, and Bo Li. Datalens: Scalable privacy preserving training via gradient compression and aggregation. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 1146,2168, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Pashion-mnist novel image dataset for benchmarking machine learning algorithms. Xiv preprint Xrv 1xXi::178.0077 770807777777 2017.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint arXiv: 1rXii:1802.000 802.00639,,, 2018.
Chugui Xu, Ju Ren, Deyu Zhang, Yaoxue Zhang, Zhan Qin, and Kui Ren. Ganobfuscator: Mitigating information leakage under gan via differential privacy. IEEE Transactions on Information Forensics and Security, 14(9))2358-225) -3771 2019.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opa- ous: User-friendly differential privacy library in pytorch. In NewrIPS 2021 Workshop Privacy in Machine Learning, 2021. URL http:////pppeeeeeeeeee//rorrrrre
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of a large-scale image datasct using doep learning with humans in the loop. arXiv preprint arXiv:1506.c 0336:,1506.03355, 2015.
Xinyang Zhang, Shouling Ji, and Ting Wang. Differentially private releasing via deep generative model (technical report). arXiv preprint orXic:1801.01594 2018.
Bolei Zhou, Agala Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
17

---

Published in Transactions on Machine Learning Research (04/2024)
A Appendix
A.1 Training Details
DP-ImgSyn implementation uses the Pytorch (Paszke al. 2019 framework, and the experiments were conducted on NVIDIA GeForce GTX 1080 Ti with 11 GB of memory with the Ubuntu operating system.
Table 5: Hyperparameters used for DP training and batch statistics capture experiments on the vision datasets: MNIST, FashionMNIST, CelebA-Hair, CelebA-Gender, CIFAR-10, and ImageNette. The de- notes the privacy budget, Thr is the number of training epochs, Uer is the batch size used for DP-SGD training, "ir is learning rate used for training, C. denotes the maximum norm limit for the gradient vector (g/maz(1, (19l12//)))) controls the amount of noise added to 9, Mon is the number of epochs used for capturing batch statistics, and Qnn is the batch size used for capturing batch statistics
Dataset
6
A/tr
Qtr
7tr
?Jbn
Pbn
1
4
128
0.01
1.0
0.8
2
64
MNIST
10
14
128
0.01
1.0
0.5
2
64
1
30
50
0.01
1.2
1
2
64
FNSHI
10
20
128
0.01
1.2
0.5
2
64
1
18
128
0.001
1.0
0.8
5
128
CelebA Hair
10
22
128
0.001
1.0
0.45
3
128
1
18
128
0.001
1.0
0.8
4
128
CclebA Gender
10
22
128
0.001
1.0
0.5
4
128
CIFAR-10
10
12
128
0.001
1.0
0.5
5
128
ImageNette
105
57
8
0.001
1.0
0.3
3
A.1.1 DP Teacher Model Hyperparameters
For the DP statistics capture for the teacher model (Section 4.1 from the main paper), we used the hyper- parameters reported in Table 5 The c denotes the privacy budget, ?hr the number of training epochs, Oer is the batch size used for DP-SGD training, "ir is learning rate used for training, C denotes the maxi- mum limit for the gradient vector g (g/mas(1, |lglla/C)), (199///
controls the amount, of noise added to y o o²(21)), Thon is the number of epochs used for capturing batch statisties, and Oon is the batch size used for capturing batch statistics in Table The accuracy results of the corresponding teacher models are reported in Table in the main paper.
(g WW(00,
A.1.2 DP Image Synthesis Hyperparameters
For the DP Image Synthesis (described in Section 4.2 and Algorithm in the main paper), we use the Adam
optimizer (Kingma & Ba 2014) with synthesis learning rate Teyn 0.1, betas 31 = 0.5, B2 = 0.99. For MNIST and FashionMNIST. we use a batch size of 80; for CelebA-Hair and CelcbA-Gender, we use batch size of 60. The main paper in Table 2 reports the number of optimization iterations for each dataset. and privacy budget.
The total loss optimized during image synthesis is the summation of the following losses: Reatture: Retassif: Riv, and R2, (Equation 5 in the main paper). The scaling coefficients corresponding to each of these losses are denoted as a,, ae, atpy and azz respectively. Table 7 reports the values of the scaling coefficients used our simulations. Furthermore, we perform an ablation study on the scaling coefficients that control the total loss on MNIST the private dataset for epsilon 10, and TinylnageNet as the public dataset initialization. Specifically, search the values for each loss scaling factor while keeping the remaining
18

---

Published in Transactions on Machine Learning Research (04/2024)
Table 6: Ablation study on the scaling coefficients af, des Qtos and at2 for MNIST as the private dataset, 10, and TinylmageNet and Gaussian Noise as the public dataset initialization.
Loss
Dataset
Hyperparameter Value and Accuracy
ay 0.1
a, 1.0
a 10.0
R feature
TinylmageNet
93.99 + 0.30
94.56 + 0.15
94.03 = 0.64
Gaussian Noise
89.55 + 0.92
90.30 + 1.43
88.53 + 0.33
ag 0.01
ag 1.0
ac 10.0
Reloasif
TinylmageNet
94.42 0.31
94.03 + 0.64
94.49 0.04
Gaussian Noise
90.99 0.86
88.53 + 0.33
90.81 0.83
Otty 2e-5
Oto 0.01
Citu 10.0
Rie
TinylmageNet
94.03 0.64
94.27 + 0.42
94.32 - 0.12
Gaussian Noise
88.53 0.33
89.63 + 1.90
89.81 - 2.16
012 3e-8
012 = 0.0001
Qy2 = 0.01
Ri2
TinyImageNet
94.03 0.64
94.45 + 0.10
94.53 + 0.16
Gaussian Noise
88.53 + 0.33
90.93 + 1.33
89.87 + 1.01
Table 7: Values of scaling coefficients for the loss terms R features Relassis, Riv: and Rí in Equation in the main paper: af, Oce Otoy and Ozg-
f
aee
ato
0k22
10
2.5e-5
3e-8
scaling factors constant. Table summarizes our results. We observe that method is robust to the scaling factor hyperparameter selections. Then, we repeated the experiment using the same setup (MNIST as the private dataset with epsilon set to 10), but with Gaussian noise as initialization. The findings in Table consistently affirm our carlier observation, highlighting the robustness our method to different scaling lactor hyperparameter selections.
A.1.3 Dp-Img Label Generation Implementation
Here we provide the PyTorrc code that we use for generating the targets for DP-ImgSyn, given the number of classes and batch size. Since the label generation algorithm is independent of the data, there is no privacy leakage.
def generate Laberate_labllss (num classes, batch_ size)ize
torch. arange (num classes)
targets = torch. squeeze repeat (1, int ( (batch size num classes)) return targets
A.1.4 Student Model Training on DP-ImgSyn Synthetic Images Hyperparameters
We use Stochastic Gradient Descent (SGD) optimizer feon Bottou 1998 with a learning rate = 0.1, momentum 0.9, and weight decay 1o-4 for training student model on the synthetic images. We use the multistep learning rate scheduler with 7 0.1 and milestones at 120, 150, and 180 epochs. We train the models for 200 epochs with 256 batch size.
19

---

Published in Transactions on Machine Learning Research (04/2024)
Table 8: Ablation study on the loss terms for MNIST as the private dataset, = 10, and TinylmageNee Places365, FashionMNIST, and Gaussian Noise as initialization.
R Reuture
Redunsss
Ri
R1
TinyImageNet
Places365
FashionMNNIT
Gaussian Noise
X
x
94.30 i: 0.25
93.74 i 0.13
94.02 1 0.09
90.07 1 1.59
/
x
x
94.31 + 0.09
93.73 + 0.24
94.19 + 0.25
89.42 + 2.36
x
94.34 + 0.31
93.74 + 0.17
93.94 + 0.28
89.82 1.00
/
V
V
V
94.03 + 0.64
93.74 = 0.18
93.90 + 0.30
88.53 + 0.33
X
/
/
V
93.03 + 0.63
93.32 + 0.30
91.70 2.24
63.48 + 13.62
The optimization loss function is the KT-divergence between the soft labels of the teacher model and the soft labels of the student model, defined as:
min Σ (TApw((), ZEX
(7)
Where KL refers to the Kullback-Leibler divergence, (M)()) and ps()) are the output distributions (soft labels) of the leacher and the student model respectively when the synthetic image å is given input. The temperature value used in our simulations is T = 100 for MNIST and FashionMNIST and T = 10 for CelebA-Hai and CeldbA-Gender.
A.2 Loss Term Ablation Study
In this section, we evaluate the effect of each loss term the accuracy of the model: Reatuuee: Relassif, Rew and R2,, The loss term ablation study is summarized in Table We present the results for MNIST as private set with epsilon=10 when using TinylmageNet, Places365, and FNSh as public datasets using three different seeds. To get further insight, we also present results when using Gaussian Noise with mean 0 and standard deviation as initialization for the synthetic images. We make the following observations: 1) from the Gaussian Noise results we see that the feature loss that uses the batch normalization statistics significantly affects the accuracy (<= 26% accuracy drop when feature loss was excluded in the synthesis loss). 2) when all the loss terms used this led to lower standard deviation between the seeds and thus more stability between the runs. The significant effect of the feature loss can be attributed to the value of the scaling coefficient associated with it.
A.3 Visualizations of ImgSyn Using Real Example
In this section, we present real example of Figure 2 In Figure we show how ImgSyn can better sample the latent space and improve the decision boundary of the student model. For this experiment, we generated 2D data by sampling from a Uniform distribution in the range [O, 1]. The labels were set to 0 or 1 based on the cubic = -1.42 + 0.9x12 + 0.34. That is the cubic formed the decision boundary, if > 0 label = 1 and vice versa. The cubic was chosen to have a non-linear decision boundary. Since the data is 2D we can visualize the decision boundary in the input space. The top left in Figure 5 shows the training data, the two classes shown in green (class 1) and orange (class 2). The teacher model was trained on this classification problem. Please note DP was used during training since want to demonstrate ImgSyn. The network confidence is visualized confidence that the sample is in class 2. The top right shows the same visualization for the student model when applying vanilla knowledge distillation (KD) between the teacher and the student model. It shows the decision boundary at the end of training and the data (illustrated in red) that was used for performing knowledge distillation. For this example, we used two Gaussian clusters centered at (0.25, 0.25), (0.75, 0.75) with a deviation of 0.25 as the dataset to perform knowledge distillation from the teacher to the student. Vanilla KD is not able to transfer the decision boundary well from the teacher to the student. The bottom
heatmap. Blue is high confidence that the sample is in class and yellow is high
20

---

Published in Transactions on Machine Learning Research (04/2024)
(a) Teacher
(b) Student Model Vanilla KD
1.5
1.5
1.0
1.0
0.5
0.5
0.0
0.0
-0.5
-0.5
-0.5
0.0
0.5
1.0
1.5
-0.5
(c) ImgSyn on Teacher Model
(d) Student Model Trained ImgSyn
1.5
1.5
1.0
1.0
0.5
0.5
0.0
0.0
-0.5
-0.5
-0.5
0.0
0.5
1.0
1.5
-0.5
1.5
Figure 5: Example visualizing deep neural networks decision boundary when using ImgSyn and not using ImgSyn, a data-driven version Figure 2 (a) visualization of the teacher decision boundary, and the training data belonging to two classes. The two classos are shown in green (class 1) and orange (class 2). The teacher network confidence is visualized a heatmap. Blue illustrates high confidence that the sample is in class and yellow is high confidence that the sample is in class 2, (b) visualization of the decision boundary for the student model when applying vanilla KD between the teacher and the student model, and the Gaussian data used for the vanilla KD (visualized in red) this is analogous to public data, (c) visualization of the teacher decision boundary and the ImgSyn generated data (magenta). The ImgSyn data are generated after applying our ImgSyn on the Gaussian data (illustrated in red color in image (b)) to align it with the source distribution, (d) visualization of the decision boundary of the student model trained on ImgSyn data, and the ImgSyn data (magenta) used for training the student model. The decision boundary of the student model trained ImgSyn data has better match to the teacher model than the student trained with Gaussian data.
left shows the data after applying our ImgSyn on the same Gaussian dataset align with the source distribution, and the teacher decision boundary when passing the ImgSyn generated data. We how the points move closer into the range of [0, 1] to match the source distribution and the bottom right show the result of training student model on the ImgSyn data. The decision boundary of the student model trained on ImgSyn data has a better match to the teacher model than the student trained with vanilla KD on Gaussian data.
A.4 Interference of the features between Public and Private Images
In this section, we quantify the interference of the public image fcatures on the student model test accuracy. To evaluate this, we perform the following experiment: we initialize the synthetic images with Gaussian noise and then perform DP-IngSyn, using MNIST as the private dataset and epsilon equal to 10. This will set the lower bound on the interference that the public dataset has on the optimization since the starting images are random. We evaluate the accuracy of the student model trained on the synthetic images initialized with Gaussian noise (Table D). We observe that the accuracy drops about. 3.4% for the Im-Ing with Gaussian
21

---

Published in Transactions on Machine Learning Research (04/2024)
Table 9: The interference of the features of the public images on the private images. The private dataset is MNIST, 10. The results for the datasets marked with are from the main paper (Table 2 and are included in this table as reference for comparison with the Gaussian Noise.
Private Dataset
Initialization
DP-ImgSyn(0)
DP-ImgSyn(k)
Gaussian Noise
30.12 5.08
90.55 + 1.67
TinylmageNet*
92.97 0.65
94.03 + 0.64
MNIST
10
Places365*
92.63 0.23
93.74 = 0.18
PNshionMNIST**
93.61 + 0.37
93.90 + 0.30
Table 10: Comparison Table with state-of-the-art techniques for 0.2 for MNIST and FNSh and TinyImageNet as the public dataset initialization. Results for Dm-Img are mean over three different
seeds. DP-GAN refers to Xie et al. (2018), DP-MERF refers to Harder et al. (2021), P3GM refers to Takagi
et al. 2021b, DataLens refers to Wang et al. (021) and G-PATE refers to Long et al. 2021
DP-GAN
DP-MERF
P3GM
DataLens
G-PATE
DP-ImgSyn (Our)
MNIST 11.04%
62.61%
8.20%
23.44%
22.30%
77.37%
FashionMNIST
10.21%
52.61%
12.80%
22.26%
18.74%
70.63%
noise initialization compared to best-performing public images. Even though there is interference from the public set, this effect is minimal compared to the effect of DP-ImgSyn. Without DP-ImgSyn, the student model trained using Gaussian noise achieves 30.12% accuracy on MNIST while with DP-ImgSyn achieves 90.55% (Table 9
A.5 Comparison With SOTA under Strong Privacy Guarantees
In this section, we present experiments with MNIST and FashionMNIST as private datasets with an epsilon value of 0.2 and TinyImageNet the public dataset initialization. Table 10 summarizes the comparison results. Our method demonstrates better accuracy than the best-performing SOTA (DP-MERF) by 14.76% on MNIST and 18.02% on FashionMNIST.
A.6 Gaussian Noise Initialization with Low-pass Filtering
In this section, we present the results when Gaussian noise is used as initialization with MNIST the private dataset and 10, and then we apply low pass filtering implemented as a Gaussian blur Haddad et al.) [1991h. The results for various kernel sizes are summarized in Table 11 Notably, low pass filtering improves the accuracy of the Gaussian noise initialization by up to 1.07% (results for kernel size 3). However, is still lower (+s 2.4%) than the best-performing public dataset initialization. Therefore, random noise can as initialization when public images are unavailable, yiclding satisfactory performance albeit with some accuracy degradation.
A.7 More Visualizations
f provides illustrations for MNIST, FNshionMNIST, and CelebA Gender as private images, the DP- ImgSyn generated images and the corresponding public images and Gaussian Noise that were used ini- tialization for DP-ImgSyn.
Figure
A.8 Dataset Statistics
Table summarizes the statistics for the vision datasets used for the experimental evaluation in Section in the main paper: MNIST, FashionMNIST, CIFAR-10, Imagenette, CelebA-Hair, CelebA-Gender,, Tiny-
22

---

Published in Transactions on Machine Learning Research (04/2024)
Table 11: Gaussian Noise with low-pass filtering as initialization with MNIST as the private dataset and 10. The low pass filtering implemented a Gaussian blur Haddad al. (1991) for various kernel sizes.
Low-pass Filtering
Without Low Pass Filtering
Kernel Size 3
Kernel Size 7 Kernel Size 11
Kernel Size 15
90.55 + 1.67
91.62+ 1.35
90.82 + 0.96
91.10 + 0.41
91.18 + 1.23
Private Dataset: MNIST
DP ImgSyn
Public Dataset: TinylmageNet
4
4
6
4
5
6
/
O
0
1
7
6
3
0
Private Dataset: FashionMNIST
DP IygSyn
Public Dataset: TinylmageNet
Private Dataset: CelebA Gender
DP ImgSyn
Public Dataset: Places365
Private Dataset: MNIST
DP ImgSyn
Random Noise
3
4
3
3
6
1 7
3
8
6
4
0
Figure 6: We visualize the private, synthetic, and public images (from left right) for: (first row) MNIST private dataset with TinylmageNet as public dataset (on TinylmageNet we apply grayscale image transfor- mation because MNIST images are grayscale), (second row) FashionMNIST as private dataset with TinyIma- geNet as public dataset (on TinylmageNet we apply grayscale image transformation because FNshionMNIST images are grayscale), (third row) CelebA Gender as private dataset with Places365 as public dataset, (fourth row) MNIST as private dataset with Gaussian Noise with zero mean and one standard deviation as initial- ization.
ImageNet, Places365, LSUN, and Textures. We report the train and test size, the resolution of the images, and the number of classes in each datasct.
A.9 Image Quality
In this section, we use Frochet Incoption Distance (FID) Heusel et al. 2017)) to mcasure image quality which is common practice in literature (Wang al. 2021 FID score is calculated based the feature representations extracted from an ImageNet pretrained deep neural network (Inception-và model). However, metric for grayscale images (MNIST, FashionMNISTT) because it
the FID score is not recommended involves a network pretrained on RGB images. This affects the evaluation and the resulting evaluation is not meaningful, as explained in Wang et. al. (001). Thus we report the results only on the CelebA dataset. Moreover, we compare private dataset images, Dm-Inn generated images, and DP-GAN (iie et al. 2018 generated images Figure
Experiment. We evaluate DP-ImgSSn generated images using the FID score and compare with prior works.
23

---

Published in Transactions on Machine Learning Research (04/2024)
Table 12: Dataset statistics, training, and test set sizes for the datasets used.
Dataset
Train Set Size
Test Set Size
Resolution
Number of Classes
MNIST 60,000
10,000
28x28
10
MNSSI
60,000
10,000
28x28
10
CIFAR-10
50,000
10,000
32x32
10
Imagenette
10,000
5,000
221x224
10
CelebA-Hair
162,770
19,962
64x64
3
CelebA-Gender
162,770
19,962
64x64
2
Tiny Net
100,000
10,000
32x32
200
Places365
1,803,460
10,000
32x32
365
LSUN
9,895,373
303,304
64x64
10
Textures
5,640
1,880
224x224
Private Images (CelebA Hair)
DP-ImgSyn (ours) Generated
DP-GAN Generated (CelebA Hair)
Figure 7: Comparing Private Dataset (left), OP-ImgSyn generated images for 10, 8 10 with
Places365 as the public dataset initialization (center) and DP-GAN Xie et al.) 2018 generated images,
100, 5 = 10-5 (right).
Results. Table 13 compares the FID score of the generated images from various prior works when using CclobA the private datasot. When using DP-ImgSyn we initialize with Places365 (reported A) and LSUN (reported as B) as the public dataset.
Conclusion. Our proposal achieves low FID score (lower is better) and outperforms state-of-tho-art works.
Comparison Table with state-of-the-art techniques using FID score (lower better) for CelebA dataset with Places365 (A) and LSUN (B) as public datasets for DP-ImgSyn.
Table 13:
Methods DP-CAN DP-MERF
P3GM
DataLens
G-PATE
DP- Dm-Inn (ours)
10
104
104
10
10
10
FID 4
403.94
327.24
435.60
320.84
305.92
188.62 (A), 194.88 (B)
24


Reviews:
[{"review_id": "4Nh9eFmexJ", "reviewer": "Reviewer_Uuyb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of leveraging and modifying public images to protect both the instance level privacy (e.g., membership privacy) as well as the feature level privacy (handled straightforwardly using different datasets) is interesting. \n2. The proposed approach is shown to outperform the previous baselines when the results are directly taken the from the respective papers. \n\nWeaknesses:\n1. The impact of optimizing the public dataset seems to be not helping much in many cases, which indicates that existing public datasets are sufficient enough to be released as the synthetic dataset. This is rather surprising to me. There are only a few combinations of public and private datasets that lead to significant performance improvement after optimization, which definitely should be explained well in the paper. \n2. The performance of the previous baselines are directly taken from the paper. I generally do not appreciate such a strategy as the experimental conditions and settings can be different and hence, can lead to different conclusions when tested under different environments. This concern is further amplified when the proposed approach simply works by sending the unoptimized public datasets to the teacher model to generate the labeled synthetic datasets. \n3. In terms of the general recommendation, it is unclear to me whether we should simply expect a very good performance, regardless of what public dataset we use. Will the performance drop significantly when using a public dataset that is completely different from the private image that even using the optimizations still cannot mitigate the domain gap?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of how to generate synthetic images of a private dataset, while also ensuring that the generated images do not look similar to the private one. The authors proposed method leverages some public datasets that are different from the private dataset, and the perturbs the inputs to ensure it aligns with the private data statistics in the latent space. This way, the domain mismatch between the private and public datasets can be minimized. Empirical evaluations also show that then proposed approach can generate synthetic images that look significantly different from the private ones, yet training a model on the synthetic dataset can induce satisfactory performance compared to models trained on the private dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Explain why optimizations are not needed in many settings.\n2. Ideally, compare the baselines within the same testing environment. \n3. Provide a clear answer to the weakness 3.", "broader_impact_concerns": "No ethical concerns for this submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZKwdm0sUyk", "reviewer": "Reviewer_56X9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper compared the proposed method with a lot of similar baselines and the proposed method seems to be efficient.\n- The paper performed extensive empirical evaluations over different vision datasets and demonstrated improved performance compared to prior baselines.\n\nWeaknesses:\n- A major weakness of this work is that the current experiments are unclear to me why (private) synthetic data is important and what downstream tasks are beneficial from the proposed method of generating synthetic data. Synthetic images are used to train a student model that has the same architecture with the teacher model and is evaluated on the private dataset $D_{priv}$. But it is assumed that there already exists a DP model that performs well on $D_{priv}$ (Table 1) and that model is used to label the synthetic data. Then what is the point of having synthetic data that could be used to learn the same model but with compromised utility? Maybe we are more interested in the regime of having a different student model but we don't have a DP version for that? Some examples of empirical results that I'm curious to see are having a much smaller student model (some models is Table 5 are around the same size as Resnet18 right?); having much less synthetic data compared to training data to train the student model; etc.\n- From my understandings, Table 2 seems to be a somewhat strong evidence that in most cases, $k=0$ is good enough for the student model, which is basically saying that if we use a DP model to label some unlabled dataset, that dataset is likely to transfer the knowledge well enough to the dataset used to train the DP model? That finding itself is still interesting. However, it does not seem that perturbing the public dataset does much to the final results.\n- I don't understand why we care about the image quality if the purpose is 1. $(\\epsilon,\\delta)$-DP, 2. retain utility of private images, 3. be visually dissimilar to private images. It does not have to be similar to the public images right? Would that limit the power of the propose method as now it seems that we get strong performance in most cases by simply using $k=0$?\n- There are 4 hyperparameters in the proposed objective. How do we tune hyperparemetrs in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work the authors propose a new method for differentially private image synthesis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While the authors might not have time for everything I mentioned above, having the following changes would be great:\n- Can the authors explain and showcase why private synthetic data is important and how we can use them?\n- Can the authors comment on any drawbacks of setting $k=0$?\n- Can the authors provide some intuition why the transfer from FashionMNIST to MNIST is very bad but the reverse is good?\n- I would suggest emphasize Section 5.7 and Table 5.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4DHOQtdlhH", "reviewer": "Reviewer_jHNE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths:__\n\nOverall, the paper is well-structured and easy to follow. The considered setting and the proposed algorithm differ from most existing literature on differentially private image synthesis. The experiments are comprehensive and clearly show the advantages of the proposed algorithm in terms of utility, provided the downstream task is image classification.\n\n__Weaknesses:__\n\nMy biggest concern about the paper is that the motivation for generating visually dissimilar synthetic images is poorly explained. This is the key distinction between the work and the existing literature on GAN-based DP dataset synthesis methods, so I expect a convincing argument on why this is the case. However, I can only find a relevant sentence throughout the paper in the introduction: “This poses significant challenges when the image content is visually sensitive and disturbing, for example with certain medical image datasets, content moderation image datasets (Steiger et al., 2021), etc.” Can you elaborate on the meaning of visually sensitive and disturbing images and how your experiments support this argument? By looking at the cited reference (Steiger et al., 2021), I fail to understand why it is relevant to differentially private image synthesis. If the authors can provide evidence demonstrating that GAN-based image synthesis methods can generate images “with visually sensitive and disturbing content” for differentially private image synthesis tasks, the motivation will be much strengthened.\n\nMoreover, Table 1 and Table 2 show that the DP-trained teacher model consistently achieves a better utility performance than your method. Since it also satisfies the differential privacy guarantee, I am confused about why directly sharing the DP-trained teacher model is not an option. In the introduction, you mentioned that data release methods are better than model release methods since they are more flexible. If I understand correctly, the flexibility here is based on the variety of downstream tasks. However, it is unclear how much improvement in “flexibility” your methods can improve compared with directly sharing a DP-trained teacher model. More discussions and experimental results supporting the flexibility of your method are expected.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of differentially private image synthesis from a different perspective. Instead of considering the typical GAN-based methods to produce visually similar synthetic datasets, the paper aims to generate synthetic images, which are visually dissimilar to the private images, based on dataset alignment and knowledge distillation techniques. Through comprehensive experiments on Image benchmarks, the paper shows the advantages of the proposed image synthesis algorithm in terms of utility and visual dissimilarity, compared with various GAN-based image synthesis methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please clarify my questions regarding the two major concerns of your work listed in weaknesses and provide empirical evidence as suggested. \n\n__Other suggestions regarding the presentation:__\n\n2. The paper will benefit from providing a general theoretical statement on the differential privacy guarantee your method can achieve, especially for readers unfamiliar with the literature on differential privacy. \n\n3. From my perspective, Section 2 does not provide much help in getting across your story, and it is too short to be a single section. Instead of just presenting the technical solution, I suggest providing detailed discussions and evidence in Section 2 on why generating visually dissimilar synthetic images is important.", "broader_impact_concerns": "There are no broader impact concerns about the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Please take the reviewer comments into account to motivate the paper better.
Decision: Accept (poster)