Does Training with Synthetic Data Truly Protect Privacy?
Authors: Yunpeng Zhao
Jie Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: ML privacy
membership inference
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We rigorously evaluate privacy leakage across various methods based on training with synthetic data, and none of these methods achieve a better trade-off than the differential privacy baselines.

Abstract: As synthetic data becomes increasingly popular in machine learning tasks, numerous methods---without formal differential privacy guarantees---use synthetic data for training. These methods often claim, either explicitly or implicitly, to protect the privacy of the original training data.
In this work, we explore four different training paradigms: coreset selection, dataset distillation, data-free knowledge distillation, and synthetic data generated from diffusion models. While all these methods utilize synthetic data for training, they lead to vastly different conclusions regarding privacy preservation. We caution that empirical approaches to preserving data privacy require careful and rigorous evaluation; otherwise, they risk providing a false sense of privacy.

Published as a conference paper at ICLR 2025
DOES TRAINING WITH SYNTHETIC DATA TRULY PRO- TECT PRIVACY?
Yunpeng Zhao National University of Singapore yunpeng. zhaoêu. nus edu
Jie Zhang ETH Zurich jie Shaaggennf.tth..c.
ABSTRACT
As synthetic data becomes increasingly popular in machine learning tasks, numer- methods methods-withhout formal differential privacy guarantees- use synthetic data for training. These methods often claim, either explicitly or implicitly, to protect the privacy of the original training data. In this work, we explore four differ- ent training paradigms: coreset selection, dataset distillation, data-free knowledge distillation, and synthetic data generated from diffusion models. While all these methods utilize synthetic data for training. they lead to vastly different conclu- sions regarding privacy preservation. We caution that empirical approaches preserving data privacy require careful and rigorous evaluation; otherwise, they risk providing a false sense of privacy.
INTRODUCTION
Synthetic data is increasingly utilized for training machine learning (ML) models, especially in sit- uations where real-world data is scarce, sensitive, costly to obtain, or subject to regulations such as GDPR (GDPR.eu). Synthetic data is particularly beneficial in scenarios where data distributions are atypical, such as in federated learning with non-IID data (Zhang et al.) 2023c), long-tailed learn- ing (Shin et al. 2023), and continual learning (Meng al. 2024). It enables the creation of diverse datasets that include edge cases or rare events that may be underrepresented in real-world data. Con- sequently, training models with synthetic data has proven beneficial for enhancing model robustness and adaptability across a wide range of real-world scenarios.
Many empirical methods- training, such as coreset selection (Feldman 2020), dataset distillation (Wang et al. 2018). data-free knowledge distillation (Yin et al. 2020). and synthetic data generated from diffusion models Yuaan
formal differential privacy guarantees--rey on synthetic data for
ct al. 2024). These approaches involve training ML models using proxy date iinteead of the original private training data. This proxy data can be directly sampled from private sources (Guo et al.
2022 Mirzasoleiman et al. 2020) or out-of-disttibution sources (Wang et al. 2023). iteratively
optimized (Zhang et al.] 2003dd Zhao al. (2020), or generated using GANs (Karras et al., 2019) and diffusion models (Rombach et al. 2022). Since the model may never encounter any private training data and the synthetic images are often visually distinct from the original private data, these
methods often claim to preserve privacy while still maintaining satisfactory performance.
In this work, we aim to address the following question: Does training with synthetic data truly protect privacy
To rigorously measure the privacy leakage of empirical methods trained on synthetic data, we use membership inference attacks (Shokri et al., 2017) as privacy auditing tool. We provide a system- these four training paradigms. For each training paradigm, we interact only with the final model trained on synthetic data, and then determine whether a particular data point part of the private training dataset.
atic privacy evaluation
We also provide a fair comparison with theoretical defenses with differential privacy, such as DPSGD (Abadi et al. 2016), and always report the privacy leakage of these methods in the worst
For simplicity, for the rest of the paper, we will always use the term "synthetic data" (also for coreset). Existing works generating synthetic data with differential privacy outside the scope of this work.

---

Published as a conference paper at ICLR 2025
Coreset DD (Real init) DP-SGD
DFKD DD (Noise init) Undefended
Diffusion DD (OOD init)
0.3
100.0%
Training Time (hours)
0.7
30.0%
Privacy Leakage
(TPR@0.11) 1.% FPR)
1.3
10.0%
2.0
3.0%
2.0
1.0%
5.2
0.3%
17.6
0.1%
17.6
269.8
74%
78%
82%
86%
90%
94%
Utility (CIFAR-10 Test Accuracy)
Figure I: rigorous evaluation of privacy leakage in models trained with synthetic data. We com- pare the privacy-utility tradeoff and efficiency of four training paradigms-corees selection, dataset distillation (DD), data-free knowledge distillation (DFKD), and synthetic data generated from dif- fusion models against DPSGD.
case. An ideal defense should strike good balance between privacy and model utility, while also being efficient.
As shown in Figure | we employed the most rigorous evaluation framework described in Aemi (ct al. (2024). training multiple shadow models to accurately evaluate membership inference success, specifically focusing on the true positive rate (TPR) at very low false positive rates (FPR) on the CIFAR-10 dataset. We ensure that each method achieves at least approximately 73% accuracy on the test set, models with low accuracy may lead to misleading conclusions (e.g., overfitting) and are not meaningful.
Similar to the findings in Aerni al. (024). almost all empirical training paradigms do not provide stronger privacy protection than the basic differential privacy baseline-DPSGIT Another interest- ing finding is that for some dataset distillation methods, the success of membership inference attacks is very low, yet the synthetic data visually resembles the private data almost entirely (see Fig- ure 77 We argue that this does not constitute meaningful privacy protection. Based these find- ings, we caution that empirical approaches to preserving data privacy require careful and rigorous evaluation; otherwise, they risk providing misleading conclusions.
RELATED WORK
Membership Inference Attack Membership inference attack (Shokri al. 2017) is canonical approach to estimating privacy leakage, which aims to determine whether given example part of the training set. State-of-the-art attack methods frame membership inference as a hypothesis testing problem and evaluate privacy leakage by reporting the true positive rate very low false positive rates (Carlini et al. 2022a). Recent work (Aerni et al. (004) suggests that many empirical defenses fail evaluate privacy leakage in worst-case scenarios; instead, they often report average- case privacy leakage, which can significantly underestimate actual privacy risks. Since privacy not an average metrie, we adopt this approach provide a rigorous and careful evaluation of privacy.
Training Synthetic Data Many studies have highlighted that training models synthetic data can remarkably effective in scenarios where data collection or distribution is challenging. For instance, in long-tailed learning and federated learning, generative models can be used to augment data, resulting in a more balanced distribution (Shin et al.) 2023 Zhang et al.] (223c). Similarly, in continual learning, synthetic data is adopted to help mitigate the problem of catastrophic forget-
2

---

Published as a conference paper at ICLR 2025
ting (Meng et al., 2024 Zhang et al.
2023b Shin al. 2017). In scenarios where only the target model is available without access to its private training data, generative models can be used to create synthetic dataset train substitute model. which can achieve similar performance to the target model (Zhang et al. [023a] Lopes et al. 2017). Furthermore, Yuan et al. (202)) demonstrated that a model trained solely on syntheric data can perform well the ImageNet test set. (Cazenavette al.) (2022); Guo et al. (2024) showed that even when the original training set is condensed to just 1% of its size as synthetic data, is still possible train a well-performing model on CIFAR 10 test set. These examples illustrate the effectiveness of synthetic data, but in this work, we are more curious about whether these models trained with synthetic data actually protect privacy.
A related study Aerni et al. (2024) rigorously evaluates several empirical membership inference defenses, including techniques such as label smoothing [hen & Pattabiraman (2024), training loss calibration Chen et al. (2022, and self-supervised learning. While our experiments build on their evaluation framework, we extend this work by specifically investigating empirical methods that train ML models using synthetic data.
3 MISLEADING PRIVACY EVALUATIONS ON SYNTHETIC DATA
3.1 EMPIRICAL DEFENSES: TRAINING MODELS WITH SYNTHETIC DATA
In this section, we provide a brief introduction to the four training paradigms for generating synthetic data that we studied. A comparison of these training paradigms is presented in Table and we show our evaluation setup in Figure 2
Table 1: Comparison of methods with respect to privacy considerations for the final model. "Private Data" indicates that the final model is trained directly on some private data, and "partial" means that some method requires it.
Training Paradigm
Private Data
Public Data
Generative Model
Teacher Model
Coreset Selection!
X partial X
X
X
Dataset Distillation
X
X
partial X
DFKD
X
Diffusion
X
Toneva et al. 2019) Guo et al. 2024
Cazenavette al.
2022
Zhao & Bilen 2021
2023
Fang al. 2022
Yuan al.
2024
Coreset Selection Coreset selection (Toneva et al., 2019, Welling 2009) aims to extract a com- pact and informative subset Doore that captures the essential characteristics of the original private training set D The selection algorithm begins by evaluating the significance of each data in- stance in Dusin: scoring them based criteria such as the frequency of forgetting events during training (Toneva et al.) (209) or their distance from the cluster center in feature space (Welling, 2009). After scoring, Doore is formed by selecting the top samples with the highest scores, which can be expressed as:
Deore = {ar Duraio score( (op(Score(DDunn k));
where score(-) is the scoring metric and Top(:, k) selects the top & highest scores. The selected coreset Deore then replaces the original private dataset for model training. It is obvious that the samples in the subset still face the potential risk of privacy leakage.
Cazenavette al., 2022 Guo et al.] (024) aims to learn a small set of informative synthetic images Dayn from large training dataset Prrain- such that a neural network trained on Dayn achieves similar or comparable generalization performance to a network trained on the original dataset. The process begins by initializing images in Dey from real images or random noise. After initialization, the goal of DD is formulated as optimization problem, seeking to minimize the discrepancy between the syn- thetic and original datasets by aligning their effects on the neural network @a parameterized by 8. The optimization objective Las may involve differences between gradients of two sets of network
Dataset Distillation (DD) DD Wang et al. 2018 Zhao & Bilen 2023 2021
3

---

Published as a conference paper at ICLR 2025
Coreset Selection Dataset Distillation Data-free KD Diffusion
MIA
Private model
Private data
Synthetic data
Final model
Most vulnerable
Figure 2: We evaluate the privacy leakage of private training data in the worst-case scenario for each training paradigm, only interacting with the final model trained on synthetic data.
disparities in feature distributions across multiple sampled em- bedding spaces (Zhao & Bilen 2023), or variations in model training trajectories (Cazenavette et al.)
parameters (Zhao Bilen 2021).
2022
Guo et al. 2024):
min Laa(do (Dyy), @o (Drain)):
Dae
After convergence, the synthesized dataset Dsn can be used train models. However, potential privacy risks may arise during the initialization stage if real images from the private dataset Duain used initialize Dyya- A critical concern is whether the subsequent DD optimization process adequately safeguards the privacy of these initial samples.
Data-Free Knowledge Distillation (DFKD) DFKD (Yn et al. 2020 Fang et al. 2022 transfers
knowledge from teacher model freacher student model Jstudent using synthetic data that approxi- mates the private training data. First, fieacher trained on Drrai- Then, a generative model G creates synthetic data {a to match the teacher model's statistical properties, minimizing inversion loss:
min Lgen (G, frencheri Derain).
The student model is trained on this synthetic data and the teacher's predictions, minimizing the distillation loss:
min Ldistill (Ssuudent, {ar ftencher).
Juste
This process iterates until convergence. Privacy risks may arise if the teacher model memorizes specific instances from Durain-
Synthetic Data from Fine-Tuned Diffusion Models This approach (Yuan al. 2024) involves fine-tuning stable diffusion model private dataset to learn its distribution. Leveraging the model's powerful generative capabilities, synthetic data is then generated and used train models. Potential privacy leakage may occur if the fine-tuned diffusion model retains reveals sensitive information from the private dataset, especially if the generated synthetic data closely resembles the original data.
3.2 PRIVACY LEAKAGE ON MOST VULNERABLE SAMPLES
Our motivation is properly evaluate privacy leakage in ML models trained synthetic data. First, we will highlight some misleading evaluations that give false sense of privacy. Then, we will demonstrate that models tend to strongly memorize the most vulnerable samples, which is why training with synthetic data can lead to privacy leakage.
Misleading Privacy Evaluations on Synthetic data Since privacy is not metric that can be averaged (Steinke & Ullman 2020), Aerni et al. (2024) advocates for rigorous evaluation by report- ing privacy leakage on the most vulnerable samples dataset, rather than the average case. To enhance computational efficiency. they suggest using mislabeled data, which approximates the pri- vacy leakage of the most vulnerable samples. However, none of the existing work has systematically evaluated privacy leakage in the worst-case scenario for synthetic data.
Previous work (Dong et al.) 2022) claims that dataset distillation can significantly improve data privacy, but the level of privacy protection is evaluated in correct way (Carlini al. 20226).
4

---

Published as a conference paper at ICLR 2025
Average case
Most vulnerable
Member Non-member
32.52%
Average
Most vulnerable
30%
TPR@0O11 1%% FPR
25%
20%
16.87%
15%
5.06%
5.02%
5%
1.57%
Random Selection Method Method
Figure 3: Failing to report privacy leakage the most vulnerable data provides a false sense of privacy. We investigate three different de- fenses: one based on coreset selection and two based dataset distillation.
Figure 4: ML models tend to strongly mem- orize the most vulnerable data. We demon- strate this by presenting the loss distribution for both members and non-members, compar- ing average-case data with worst-case data.
Some studies (Hao et al. 2021 Dong al., 2022) claim that training synthetic data protects privacy because synthetic data is visually dissimilar to the private data, However, all these methods give false sense of privacy, which further demonstrate Section
We begin with a toy experiment on coreset selection and dataset distillation In Figure we demonstrate that for coreset selection and dataset distillation, failing report privacy leakage on the most vulnerable samples can severely underestimate the true privacy risk. Thus, in this work, we always report privacy leakage in the worst case") to avoid underestimating the privacy risk.
Strong Memorization on Vulnerable Data ML models can memorize sensitive information from their training data, particularly for vulnerable samples such as mislabeled data or outliers (Feldman Zhang, 2020). Figure 4 show that for mislabeled data point, models can strongly mem- orize it, leading to significant difference in the loss distribution for this sample when it is in the training set versus when it is no This discrepancy makes MIA much easier. Therefore, later will show that no matter how carefully synthetic data training is designed, the strong memorization capabilities ML models still make easy to perform MIA on the most vulnerable samples.
4 CASE STUDIES: PRIVACY LEAKAGE IN TRAINING WITH SYNTHETIC DATA
In this section, we conduct a systematic and rigorous evaluation of privacy leakage across four training paradigms based on synthetic data. We start by introducing the experimental setups. Sub- sequently, we will discuss the potential privacy leakage inherent to each method. Finally, will compare all these methods with a differential privacy baseline (i.e., DPSGD) with respect to model utility, privacy preservation, and computational efficiency.
4.1 EVALUATION SETUP
Evaluation Metric Empirical methods designed preserve data privacy require careful scrutiny.
To ensure precise measurement of privacy leakage in models trained with synthetic data, we employ the state-of-the-art membership inference attack LiRA (Carlini et al. 2022ah) and follow the setup from (Aerni et al. 2024). In this setup, we report the privacy leakage by evaluating the true positive rate a low false positive rate on the most vulnerable samples (e.g.. mislabeled data).
Experimental Setup We conduct all experiments on CIFAR-10 Krizhevsky & Hinton 2009). as all training methods are scalable to CIFAR-10 and achieve good test accuracy. We designate
See Appendix C for detailed experimental setup.
'By default, we mislabeled data strong canaries to simulate the most vulnerable samples, but even stronger canaries may exist, offering hetter capture of privacy leakage.
5

---

Published as a conference paper at ICLR 2025
Random Guess Random Selection
Undefended Forgetting
Undefended
Forgelling
Newly Successful
10
Shadow ID
Shadow
Shadow ID
Shadow
2029
2106
2095
1728
1858
10-1
404
TPR
277
152
Shedow ID
Shadow ID
Shedow ID
Shadow ID
10
2460
2272
1844
1722
1810
1907
10
204
320
462
302
10
10
10-2
10-1
10a
FPR
Figure 6: MI success rate for both the full dataset and the coreset. "Newly successful" de- notes samples that typically very safe when trained on the full dataset but experience a sig- nificant increase in privacy leakage once in- cluded in the coreset.
Figure 5: Coreset selection does guarantee privacy protection, both random selection and forgetting result in significant privacy leakage. The TPR at 0. % FPR for forgetting is 72.94% while is 38.70% for random selection.
500 random data points as "audit samples" on which evaluate membership inference, and use mislabeled data as strong canaries to simulate worst case data; the remaining 49,500 samples are always included in every model's training data. For each method, we train 32 shadow models, ensuring that each audit sample is included in the training data of 16 models. For all defenses, we consistently adopt ResNet-18 (He al. 2016) as the network architecture of shadow models. We report the performance of these methods across three dimensions: privacy leakage (TPR@) @0.1 FPR), model utility (test accuracy), and efficiency (training time). A strong defense should achieve balanced tradeoff among privacy, utility, and efficiency. The implementation details of all discussed defenses are provided Appendix
4.2 PRIVACY LEAKAGE IN CORESET SELECTION
We begin by evaluating coresct selection methods to explore whether privacy protection can be achieved by selecting representative subset to replace the entire private dataset for model training. We compare two representative methods: random selection and forgetting (Toneva et al.] 2019), against the undefended baseline. The coreset size consistently set to 20,000. The average test accuracy across 32 shadow models is 94.78% for the undefended baseline, 92.57% for forgetting, and 90.56% for random selection.
In Figure 5) we find that neither random selection nor forgetting is able to protect the membership privacy of auditing samples as the private subset directly used for training.
One interesting finding is that the forgetting defense achieves better utility than random selection by selecting informative samples that are easily forgotten during training. However, its privacy leakage is significantly higher. This is likely due to the forgetting method's d's tendency to select mislabeled samples. While random selection includes roughly 40% of mislabeled data, the forgetting method increases this to 74%, suggesting that these vulnerable data points also contribute improved gen- eralization (Feldman & Zhang 2020), at the cost reduced privacy.
Another interesting finding is that even in average case evaluations, some samples that would typ- ically be very safe (when trained on the entire dataset) experience a significant increase in privacy leakage once selected for the coreset. This suggests that protecting the privacy of non-coreset sam- ples can degrade the privacy of other samples. In worst-case evaluation scenarios, this becomes even more concerning, as sample would be willing to sacrifice its privacy for the benefit of others. We illustrate this in Figure where users whose privacy appears least at risk may be the most likely not request their data be included in coreset.

---

Published as a conference paper at ICLR 2025
Table A rigorous evaluation privacy leakage on four dataset distillation methods. indicates that the model's 's utility is significantly lower and thus less practical to use.
Methods
Initialization
Test Accuracy
TPR@0. @0.1% FPR
Private
84.33%
18.02%
DM
Noise
72.52%
0.24%
OOD
75.00%
0.18%
Private
84.43%
17.74%
DSA
Noise
OOD
Private
81.81%
1.51% %
MTT
Noise
OOD
80.42%
0.18%
Private
84.54%
1.29%
DATM
Noise
OOD
Distribution Matching
Gradient Matching
Trajecctory Matching
4.3 PRIVACY LEAKAGE IN DATASET DISTILLATION
Current work evaluating the privacy leakage of dataset distillation often contains some flaws. These methods face issues in empirical evaluation, such as using improper averaging metrics (e.g., AUC instead of TPR) (Chen et al., 2023). evaluating models with low performance (around 60% test accuracy), applying incorrect theoretical analyses (Dong et al., 2022). Furthermore, none of these methods are evaluated under worst-case scenarios. Such non-rigone non-rigorous evaluations can result
in misleading conclusions (Carlini et al. 2022b).
In this work, we conduct experiments on four representative DD methods to rigorously eval- uate their privacy leakage, including DM (Zhao & Bilen 2023). DSA (Zhao & Bilen 2021), MTT (Cazenavette et al. 2022), and DATM (Guo et al., 2024). We tune the model performance to around 80% test accuracy and evaluate these methods under worst-case scenarios to capture the full extent of potential privacy leakage.
Popular methods in dataset distillation aim match the behavior of models trained on synthetic data with those trained on private data. This matching can be based on distribution, gradients, or training trajectories. Additionally, synthetic data be initialized using real private data, out-of- distribution (OOD) data, or random noise. These factors can lead to significantly different levels of privacy leakage. We present the overall results in Table 2 and provide detailed analysis of how each factor impacts privacy leakage in the following discussion.
The initialization of synthetic data affects privacy As shown in Table 2 initializing synthetic data with private data leads higher privacy leakage in DM compared to initialization with OOD data or random noise. Con- sequently, while initializing with OOD data random noise offers better privacy protection, it comes at the cost of substantial reductions in model performance.
Table 3:
The MI success rate on initial- ization and non-initialization canaries.
Success rate (%)
Method
Init
Non-init
DM
88.89
0.16
To further illustrate why initialization with private data results in greater privacy leakage, we analyze the privacy risks associated with canaries used during the synthetic data initialization process. As shown in Table the mem- bership inference success rate for canaries involved in initialization is considerably higher than for those not used during initialization. For canaries present in the training set but excluded from initial- ization, the success rate is nearly indistinguishable from random guessing. This finding empirically demonstrates that privacy risks are primarily introduced during the initialization phase, as the model tends to strongly memorize these initialization canaries.
DSA
88.70
0.09
MTT
7.02
0.11
DATM
36.05
0.24

---

Published as a conference paper at ICLR 2025
Original label
Mislabel
Others
Random Guess
Undefended
DFKD
10
Private Data
10-
boued
TPR
10-2
Synthetic Data
10-1
10
10
10 2
10
10
FPR
Figure 8: For DFKD, although the student model never trained on private data and is only trained on synthetic data, due knowledge distillation, still leaks a significant amount of privacy.
Figure 7: For DD, the synthetic data leaks the privacy of the private data visually, but this leakage is not captured by MIA.
Visual privacy leakage in trajectory matching with private initialization As demonstrated Table 2 trajectory-baset methods seem provide superior privacy protection compared to DM and DSA, even when private initialization is utilized. This can be attributed to the fact that trajectory- based approaches, such as MTT and DATM, initially train teacher model on the original private data and save the training trajectories. During each optimization epoch, network is selected from the teacher trajectory and trained for several steps using synthetic data, with the objective of match- ing the trained model's parameters the teacher trajectory. Notably, MTT and DATM focus aligning only the early to mid-training trajectories- stages where the model has not yet memorized canaries.
However, a low MI success rate could also provide a false sense of privacy. In Figure 7 we visualize both the synthetic data and the private data, which clearly demonstrates visual privacy leakage that is not captured by MIA. We present the logits output by the model trained on synthetic data, for both the private data (canaries) used for initialization and the synthetic data itself. This indicates that while the model can strongly memorize the synthetic data, it generalizes well on the canaries, which results a low MI success rate. A potential explanation for this phenomenon is that trajectory matching effectively introduces "weird" data augmentation on the private data, which hard to estimate. This aspect not captured during MIA, which typically employs common augmentation techniques like random flip and random crop.
In summary, despite the low MI success rate, MTT entirely compromises the privacy of private data through visual leakage. A potential defense is to MTT initialized with OOD data, which achieves a low MI success rate (close to random guessing) while offering stronger visual privacy protection (see Figure 13). However, this approach results in reduced model performance, with test accuracy dropping to 80.42%.
4.4 PRIVACY LEAKAGE IN DATA-FREE KNOWLEDGE DISTILLATION
Similar to previous work (Aerni et al. 2024), in Figure g, we also find that since the teacher model strongly memorize canaries, during the distillation stage, the student model can also somewhat memorize canaries, leading to significant privacy leakage.
We further visualize the synthetic data and private data in Figure revealing interesting finding: even when a synthetic image appears completely different from the original private image, it can still somehow trigger the teacher model's memorization of the canary, thereby leaking privacy. The SSIM (Wang et al.) 2004) score only 0.1661, yet the logits distribution shows an extremely high similarity, with correlation of nearly 0.999.
8

---

Published as a conference paper at ICLR 2025
Private Synthetic
Randem Guess
Undefended Text-to-image
image-to-image
Logits Sirvia 9991
10"
X
SSIM: OMM:0..68 0.1661
TPR
Student Thar
logits
10-2
logits
10
10-1
10
10
10-1
10a
Original
FPR
Figure 9: For DFKD, synthetic data acti- vate the teacher model's memorization of pri- vate data, even if the synthetic data appears vi- sually distinct, leaking privacy to student.
Figure 10: For synthetic data from fine- tuned diffusion model, image-to-image genera- tion still leaks privacy, while text-to-image re- sults in predictions close to random guessing.
In detail, the generator in DFKD is capable of producing synthetic images that, while visually dis- tinct, bear strong resemblance in logits to the mislabeled data. These synthetic images activate the memories of corresponding canaries encoded in the teacher model's parameters. The distilla- tion process attempts to align the student model's logits distribution with that of the teacher for the same images. Consequently, the student model may learn some private knowledge from the teacher, including some aspects of the teacher's memorization of mislabeled data.
In summary, the visual dissimilarity between synthetic data and private data does not necessarily guarantee privacy protection.
4.5 PRIVACY LEAKAGE IN SYNTHETIC DATA FROM DIFFUSION MODELS
A representative method for this training paradigm is presented by (Yuan al., 2024). which in- volves three stages: (I) first, fine-tuning a stable diffusion model on a private dataset; (2) then using the fine-tuned model for image-to-image generation by passing text prompt and initial image from the private data to condition the generation of new images; (3) finally, replacing all private data with the generated synthetic images (while keeping the original labels unchanged) and training models solely on this synthetic data.
The potential privacy risks in this approach include: (1) the final model could still memorize misla- beled data, as the labels are retained, and (2) the fine-tuned diffusion model, having been trained on private data, may leak sensitive information.
The first privacy risk could be mitigated by using text-to-image generation, which does not require labels and thus less sensitive to mislabeled canaries. In Figure while image-to-image genera- tion still leaks privacy, text-to-image generation results in predictions close to random guessing.
Regarding the second privacy risk, since stable diffusion is pretrained on large public datasets and has strong generative capabilities, it is largely insensitive to mislabeled data and does not exhibit strong memorization. As seen in Figure I0 when using text-to-image generation, the only privacy leakage stems from the fine-tuning process. However, based membership inference success, this fine-tuning does not result in significant privacy leakage.
A more interesting direction for future research would be investigate the privacy-utility tradeoff in diffusion models that are trained from scratch. This approach would enable a fairer comparison with other methods that do not rely on large-scale pretraining datasets.

---

Published as a conference paper at ICLR 2025
Table 4: Fair comparison with DPSGD. We compare the privacy-utllity-effiieenn tradeoff of the heuristic defenses we study to two DPSGD baselines tuned for different test accuracy.
Method
Test Accuracy
@.. FPR
Efficiency
Undefended
94.78%
100.00%
1.0x
Coreset Selection
92.57%
72.94%
2.9x
MTT (OOD)
80.42%
0.18%
39.7x
DFKD
92.09%
43.38%
2.9x
Diffusion
82.64%
12.65%
385.4x
DPSGD (medium)
83.26%
0.40%
0.4x
DPSGD (high)
90.36%
9.31%
1.9x
4.6 FAIR COMPARISON WITH DPSGD
Since differential privacy is the most standard defense, we provide fair comparison of all four training paradigms against DPSGD (Abadi al.) 2016). We tune each method to achieve compa- rable test accuracy and then evaluate both privacy leakage and training efficiency. Specifically, we consider two DPSGD baselines: one with a test accuracy 83.26% and high-utility baseline achieving 90.36% in accuracy.
We present all results in Table 4 where it is clear that DPSGD remains the best defense, achieving superior trade-off between privacy, utility, and efficiency. It is worth noting that methods like DD and fine-tuning diffusion models are significantly more time-consuming, requiring substantially more training hours compared to DPSGD.
5 CONCLUSION
Empirical methods that claim to preserve data privacy, whether implicitly or explicitly, but lack theo- retical guarantees (e.g., differential privacy), require careful scrutiny. In this work, systematically and rigorously evaluate privacy leakage across all four training paradigms that rely synthetic data. To avoid providing a false sense of security, we consistently report privacy leakage in the worst-case scenario and conduct fair comparisons with differential privacy baseline. Our results show that none of these empirical methods achieve better trade-off than DPSGD.
We hope this work can help researchers gain deeper understanding of privacy, and that any new empirical defenses should also undergo rigorous evaluation.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of results, provide detailed description of our attack protocols, as well as the hyperparameters and training details for all defenses Section 4.1 and Appendix A The source code is available at htttttt///// httpps//githhbb wtwggghu...oo/ynngga...ee. wwwp///ww/oooooooooooooo zhao/ syndata privacy
ACKNOWLEDGMENTS
J.Z. is funded by the Swiss National Science Foundation (SNSF) project grant 214838. We sincerely thank Michael Aemi and Bo Zhao for their insightful discussions during the early stages of our experiments on dataset distillation, as well as the anonymous reviewers for their valuable feedback and constructive suggestions.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC
10

---

Published as a conference paper at ICLR 2025
conference on computer and communications security, pp. 308-318, 2016.
Michael Aerni, Jie Zhang, and Florian Tramèr. Evaluations of machine learning privacy defenses are misleading. CCS 2024, 2024.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Mem- bership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914, IEEE, 2022a.
Nicholas Carlini, Vitaly Feldman, and Milad Nasr. No free lunch in" privacy for free: How does dataset condensation help privacy". arXiv preprint arXiv: 14987, 2022b.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
Dingfan Chen, Ning Yu, and Mario Fritz. RelaxLoss: Defending Membership Inference Attacks without Losing Utility. In International Conference on Learning Representations, 2022.
Zitao Chen and Karthik Pattabiraman. Overconfidence is a Dangerous Thing: Mitigating Member- ship Inference Attacks by Enforeing Less Confident Prediction. In NDSS Symposium, 2024.
Zongxiong Chen, Jiahui Geng, Derui Zhu, Herbert Woisetschlaeger, Qing Li, Sonja Schimmler, Ruben Mayer, and Chunming Rong. A comprehensive study on dataset distillation: Performance, privacy, robustness and fairness. arXiv preprint arXiv:2305.03355, 2023.
Justin Cui, Ruochen Wang, Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet-
1k with constant memory. In International Conference Machine Learning, pp. 6565-6590. PMLR, 2023.
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey, Cinic- 10 is not imagenet or c0f arXiv preprint arXiv:/8100 1810.03505. 2018.
Soham De, Leonard Berrada, Jamie Hayes, Samuel Smith, and Borja Balle. Unlock- ing high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13633 2022.
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In Proceedings of the 39th International Conference Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 5378 -396. PMLR, 17-23 Jul 2022.
Matthijs Douze, Giorgos Tolias, Ed Pizzi, Zoë Papakipos, Lowik Chanussot, Filip Radenovic, Tomas Jenicek, Maxim Maximov, Laura Leal-Taixé, Ismail Elezi, et al. The 2021 image similarity dataset and challenge. arXiv preprint arXiv:2106. 09672, 2021.
Gongfan Fang, Kanya Mo, Xinchao Wang. Jie Song, Shitao Bei, Haofei Zhang, and Mingli Song. Up to 100x faster data-free knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6597-6604, 2022.
Dan Feldman. Introduction core-sets: an updated survey. arXiv preprint arXi::2011.09384 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
GDPR.eu. Gdpr info.
inforr--iifeuuu Accessed: 2024-08-16.
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selec- tion in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficult-aligned trajectory matching. In The Twelfth International Conference Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2025
Zhiwei Hao, Yong Luo, Han Hu, Jianping An, and Yonggang Wen. Data-free ensemble knowledge distillation for privacy privacy-conscouus multimedia model compression. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1803-1811, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535, 2017.
Zichong Meng, Jie Zhang, Changdi Yang, Zheng Zhan, Pu Zhao, and Yanzhi WAng. Diffclass: Diffusion- based class incremental learning, ECCV 2024, 2024.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960.
PMLR, 2020.
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. self- supervised descriptor for image copy detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14532-14544 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Tom Sander, Pierre Stock, and Alexandre Sablayrolles. Tan without a burn: Scaling laws of dp-sgd. In International Conference on Machine Learning, pp. 29937-29949, PMLR, 2023.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Joonghyuk Shin, Minguk Kang, and Jaesik Park. Fill-up: Balancing long-tailed data with generative models. arXiv preprint arXiv:2306.07200 2023.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 6048-6058, 2023.
Thomas Steinke and Jonathan Ullman. The pitfalls of average-case differeet differential privacy. Differential- Privacy.org, 07 2020. http::///dEEeeeaaaaaaaaaaaa///gr
dp/
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. URL https:// openreview n (itrrummmd=BBB11 d=BJ]] xm3 xmm0
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 2018
Yuzheng Wang, Zhaoyu Chen, Jie Zhang, Dingkang Yang, Zuhao Ge, Yang Liu, Siao Liu, Yunquan Sun, Wenqiang Zhang, and Lizhe Qi. Sampling to distill: Knowledge transfer from world data. arXiv preprint 230.:2360...61 2023.
12

---

Published as a conference paper at ICLR 2025
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Max Welling, Herding dynamical weights learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1121-1128. ACM, 2009.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3 3-19, 2018.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8715- 8724, 2020.
Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, and Bo Zhao. Real-fake: Effective training data synthesis through distribution matching. In International Conference on Learning Representa- tions (ICLR). International Conference on Learning Representations (ICLR), 2024.
Jie Zhang, Chen Chen, and Lingjuan Lyu. IDEAL: Query-efficient data-free learning from black- box models. In The Eleventh International Conference Learning Representations, 2023a. URL
Jie Zhang, Chen Chen, Weiming Zhuang, and Lingjuan Lyu. Target: Federated class-continual learning via exemplar-free distillation. In Proceedings the IEEF/CVF International Conference on Computer Vision, pp. 4782-4793, 2023b.
Jie Zhang, Xiaohua Qi, and Bo Zhao. Federated generative learning with foundation models. arXiv preprint arXiv:2306. 16064, 2023c.
Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11950- 11959, June 2023d.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 12674-12685, PMLR, 18-24 Jul 2021. httphttsss////roooo
ngs ml pttppp//11//////222/111 httt:/..hhaaammm httm.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6514-6523, January 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. XrXiv preprint 2xXi:20006005229 2020.
13

---

Published as a conference paper at ICLR 2025
A
TRAINING DETAILS
A.1 UNDEFENDED
For the undefended baseline, we employ the same training procedure as described in (Aerni et al. 2024]. Concretely, ResNet-18 models are trained using the SGD optimizer with a momentum of 0.9 and weight decay of 0.0005. We use batch size of 256 and typical data augmentation techniques, including random horizontal flips and random shifts of up to 4 pixels. The models are optimized over 200 epochs with a base learning rate of 0.1. We employ linear warm-up of the learning rate during the first epoch, followed by a decay of the learning rate by a factor of 0.2 at epochs 60, 120, and 160.
A.2 CORESET SELECTION
We adjust the coreset size to balance the siivacy-uillty--effiiee tradeoff, which set to 20,000 to maintain high utility while using as few samples as possible. Reducing the coreset size further results in significant drop in model test accuracy. The training process and other hyper-parameters for coreset selection remains consistent with the original paper (Toneva et al. 2019). The resulting coresets are used to train the corresponding shadow models in the same way as the undefended baseline in Appendix A.1
A.3 DATASET DISTIILATIOON
Given that we consistently use ResNet-18 as the network architecture for our shadow models across various defenses, we initially considered employing it for DD as well However, in the high image- per-class (ipe) setting, which is essential for improved utility, we faced challenges with the DD process. The optimization had difficulties converging, ultimately failing to deliver satisfactory sults. Consequently, we opted for the ConvNet architecture specified in (Zhao & Bilen (023), which includes three repeated convolutional blocks. Each block comprises 228-kernel convolution layer, an instance normalization layer, a ReLU activation function, and an average pooling layer.
Studies in (Guo et al. 2024) have shown that the synthetic dataset generated through DD exhibits performance, especially with a large ipc. This implies that using ConvNet for DD does not result in significant performance discrepancies when training shadow models compared to using ResNet-18. To maximize utility, we set the ipc for all DD methods to 1,000. Upon generating the synthetic data, we follow the method described Appendix A.I to train our shadow models with the ResNet- 18 architecture.
satisfactory cross-architectes
For each DD defense, we explore three different methods for initializing the synthetic dataset: Pri- vate, Noise, and OOD initialization. Private initialization involves randomly sampling images class- by-class from the original private training set, and Noise initialization employs Gaussian noise di- rectly. Private initialization is commonly used in practice, while Noise initialization is less discussed due to its challenging optimization process and suboptimal performance. In this work, we attempt to use OOD data for initialization. For example, we use CINIC-10 (Darlow et al., 2018), an exten- sion of CIFAR- 10 incorporating downsampled ImageNet images, for initialization. For each class, select 1,000 images from CINIC-10 that corresponded to the classes in CIFAR-10 but not included it. Next we will introduce the specific implementation details for each DD defense.
DM & DSA For DM and DSA defenses, is crucial adjust the learning rate and training iterations based on different synthetic dataset initialization methods. DSA, involving a dual-layer optimization process, additionally requires specification of the number of both the outer and inner loops. Table 5 provides the hyperparameter settings for DM and DSA. All other training settings align with those outlined in their original publications (Zhao & Bilen
Table 5: Hyperparameters used for DM and DSA. Private, Noise, and OOD are initialization meth- ods. LR indicates the learning rate. Outer and Inner refer to the number of outer and inner loops.
Methods
LR
Iteration
Outer
Inner
DM (Private)
30,000
DM (Noise)
50
100,000
DM (OOD)
30
100,000
DSA (Private)
0.1
300
100
2023 2021).
14

---

Published as a conference paper at ICLR 2025
Table 6: Hyperparameters used for the synthetic dataset optimization process MTT and DATM.
Synthetic
Expert
Min Sturt
Max Start
Synthelic Batch Size
LR
LR
LR
Starting Step Size
Methods
Iteration
Steps
Epochs
Epoch
Epoch
(Pixels)
(Labels)
(Step Size)
MIT (Private)
1,500
100
2
40
200
100
le-6
0.01
MIT (OOD) 20,000
100
2
40
200
100
le-6
0.01
DATM (Private)
5,000
100
2
40
60
1,000
50
10
1o-6
MTT & DATM For MTT and DATM, training of the teacher trajectory precedes the optimization of synthetic data. We extend the trajectory length for MTT to 100 epochs while keeping other hyperparameters and settings consistent with those described in the original papers (Cazenavette et al. 2022; Guo et al. 2024). Table [ shows the hyperparameters during the synthetic dataset optimization process. All other training parameters remain in alignment with those specified in the respective original methods. ZCA whitening is used in all experiments for MTT and DATM by default. Considering the heavy reliance of MTT and DATM on GPU memory, adopted the TESLA (Cui et al. 2023) implementation of DATM and also re-implemented MTT as TESLA version, following recommendations (Guo 2024
A.4 DATA-FREE KNOWLEDGE DISTILLATION
In line with the protocol described Fang et al. 2022), using only the "BN" loss able to achieve high accuracy. This loss function matches the which is widely used most DFKD works. Given its effectiveness, we use this method exclusively generate synthetic data. This simplifies our approach and allows us generalize our evaluation to other DFKD methods.
statistics from the teacher model,
The training protocol in Appendix A.1 is adopted to train the teacher model. After that, perform the distillation process involving 240 iterations. In each iteration, we generate 256 new images, collect predictions from the teacher model, and store these in memory bank. We then train the student model for 5 epochs using the data in the memory bank.
A.5 DIFFUSION MODEL
Due to the substantial computational costs associated with Real-Fake (Yuan et al. 2024). we have made slight adjustments to our experimental setup. In all experiments involving diffusion, we re- duced the number of sampling steps during the inference stage to 10. Furthermore, the number of shadow models from 32 to 16. These modifications allow us to reduce computa- tional expenses while still obtaining meaningful conclusions. We only generate synthetic dataset equal in size to the original private dataset. The training process of shadow models is the same as in Appendix A
decreased
A.6 DP-SGD
The two heuristic DPSGD baselines without provable privacy guarantees Table 4 are developed
following (Aerni al., 2024). building upon the state-of-the-art DPSGD training techniques (Del
et 2022 Sander et al. (023) . Here, we replace the batch normalization in ResNet-18 with group
normalization (Wu& & He 2018), swap the order of normalization and ReLU. We carefully tune the
hyperparameters according to the scaling law in (Sander et al. (023) to achieve medium utility and
high utility (see Table 7) Finally, the privacy budgets e a0 4.4 x 1 for medium-utill -uility DPSGD.
1.8 10 for high-utility DPSGD and
Table 7: Hyperparameters of heuristic DPSGD baselines with medium utility and high utility.
Method
Noise multiplier
Clipping norm
Batch size
Epochs
LR
Augment
DPSGD (medium)
0.00625
64
8
DPSGD (high)
0.00625
64
16
4
8
15

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTS AND ANALYSIS
B. PRIVACY LEAKAGE IN DD WITH PRIVATE INITIALIZATION
When using Private initialization, a portion of mislabeled canaries inadvertently selected as the starting images for synthesizing categories, e.g., cat mistakenly labeled as dog could be used for initializing dog category images. As discussed in (Guo et al.] 2024). under high ipc settings, the images optimized through DD remain visually similar to the original images. It is important to emphasize that visual similarity should not be deemed the determinant of privacy leakage; however, such similarity does indicate potential privacy risks.
We define True Positive samples at an FPR of 0.1% as those successfully attacked, and divide the training set's canaries into init canaries and non-init canaries based their selection for initializa- tion. T shows the attack success rates for these two categories under different DD defenses. It evident that the success rate on init canaries is significantly higher than on non-init canaries. For canaries in the training set but not used for initialization, the success rate is nearly equivalent random guessing. This empirically supports our assertion that privacy risks are predominantly introduced during the initialization phase rather than the distillation process.
B.2 PRIVACY LEAKAGE DURING THE DISTILLATION PROCESS IN DD
We then perform a case-by-case analysis to explain why the distillation processes of these DD de- fenses do not leak the membership privacy of mislabeled canaries.
DM matches the means of neural network-encoded representations of real and synthetic data class by class. During the optimization of synthetic data, the neural network is randomly sampled from the parameter space and undergoes only forward propagation without updating its parameters, thus avoiding any memorization effects related to mislabeled canaries. Given that mislabeled samples constitute minor fraction of the training data (about 1%), their influence on the average representa- tion of the original real data is negligible. Therefore, the update process of synthetic data is virtually unaffected by the canaries.
However, defenses such as DSA, MTT, and DATM initially require training models on original pri- vate data to obtain teacher gradients or parameters. Subsequently, models are trained from the same starting point using synthetic data, aiming to elicit similar parameter updates from both real and synthetic data. The privacy risk here stems from the potential retention of canary-related memories in the gradient parameter trajectories during the teacher model's training process. The optimiza- tion goal previously mentioned could lead to models trained on the final synthetic dataset inheriting strong memories of the canaries.
Fortunately, we find that model memories of misla- beled canaries do not form rapidly (see Figure IT]. In early training epochs, the model have not formed strong memorization to those mislabeled canaries. This finding may provide a reasonable explanation for why there is no privacy leakage during the syn- thetic data optimization processes in DSA, MTT. and DATM.
0.5
0.4
TPR PM
0.3
0.2
0.1
0.0
DSA randomly samples neural network at each optimization epoch and trains it incrementally with both synthetic and real data, minimizing the differ- ence between their gradients to update the synthetic images. Since the number of training steps is gen- crally small, the teacher gradients do not yet contain effective memories of the canaries.
20
40
60
80
100
Training Epechs
Figure 11: The TPR@0. I&FP of 500 mis- labeled canaries during the training process of teacher trajectories.
Trajectory- bassd methods like MTT and DATM first train a teacher model on original private data and save the training trajectory. In each optimization epoch, network is selected from the teacher trajectory and trained for several steps with synthetic data to match the trained model parameters to the teacher trajectory. However, both MTT and DATM match only the early to mid-training tra-
16

---

Published as a conference paper at ICLR 2025
bird
Top
Top
Top
truck
Tbp
Top 2
Thp 3
CMr
Thp
Thp 2
Tbo 3
deer
Spp
Top
Tpp
track
Top
Top
Top 3
Top
1ep-2
Top 3
door
Top
Top 2
Top 3
dog
Top
Top
Top 3
frog
Top 1
Top
Too 3
cot
Tep
Top
Tpp 5
arplone
Tup
Top
Top 3
Eend
Top Ep
Top 3
Figure 12: Visualization synthetic data generated by DM (OOD) and retrieved private data with SSCD, The synthetic data are with orange border. The synthesized data does not exhibit evident visual similarity to any of the private data.
bird
Top
Top
Tpp 3
truck
Top
Top
The 3
cst
Thp
Thp
Tbo 3
deer
1pp1
Top
1pp
trank
Top
Top 3
Top
deer
Tp
Top 2
Top
dog
Top
Top
Thp 3
frog
Thp
Thp
Tho
Tep
Tep
1pp
arpkkn
Top
Top
Figure 13: Visualization on synthetic data generated by MTT (OOD) and retrieved private data with SSCD. The synthetic data are with orange border. The synthesized data does not exhibit evident visual similarity to any of the private data.
jectories, during which neither the model parameters in the teacher trajectory have formed strong memories of the canaries. Hence, these gradient and matching optimization processes prevent models trained on synthetic data from retaining memories of the canaries, thereby safe- guarding member privacy. This also explains why using noise and OOD images for initialization can achieve decent privacy protection.
B.3 VISUAL DISSIMILARITY BETWEEN PRIVATE DATA AND DATA SYNTHESIZED BY DD WITH OOD INITIALIZATION
Following the previous works (Somepalli et al. 2023 Guo et al. 2024 we employ the Self- Supervised Content Duplication (SSCD) (Pizz et al. 2022) method for content plagiarism detection. We use the ResNet50 model trained on the DISC dataset (Douze et al. 2021). For given query syn- thetic image and all reference images from the original private dataset, we infer through the detection model to obtain 5 312-dimensional feature vector for each image. The similarity between them is calculated by computing the inner product between the query feature and each reference feature. For each synthetic image, we present the corresponding private images with top similarities.
We select some synthetic images from DM and MTT, both with OOD initialization, and attempt to retrieve similar images from the original private dataset. The results are shown in Figure 12 and Figure [3, respectively. obvious that for DD methods with OOD initialization, the synthetic data is visually dissimilar those private data.
17

---

Published as a conference paper at ICLR 2025
B.4 DETAILED ANALYSIS OF THE PRIVACY LEAKAGE IN DFKD
In this section, we investigate how membership information of mislabeled canaries is transferred from the teacher the student model through the use of dissimilar synthetic data. As demonstrated in Figure the teacher model can strongly memorize the incorrect labels associated with mislabeled samples, resulting in logits distribution where the value of wrongly-labeled class significantly ex- ceed that of the correct labels. The generator in DFKD is capable of producing synthetic images that, while visually distinct, bear a strong resemblance in logits the mislabeled data. These synthetic images activate the memories of corresponding canaries encoded in the teacher model's parame- ters. The distillation process attempts to align the student model's s logits distribution with that the teacher for the same images. Consequently, the student model may inherit similar parameters to the teacher, including some aspects of the teacher's memorization mislabeled data.
B.5 DETAILED ANALYSIS OF FIGURE
Figure reveals that, despite having no direct exposure the original mislabeled data, the student model still inherits a portion of the teacher model's memory, exhibiting high logit value for the mislabel. Notably, the student model's memorization of mislabels comparatively weaker than teacher. Therefore, influenced by its generalization capabilities, the student model tends to classify canaries under their correct labels rather than the mislabels.
We assessed the accuracy with which both teacher and student models classify the training set's canaries into their respective incorrect labels. The results were 100% for the teacher and 5.28% for the student, indicating that using accuracy the metric to evaluate whether model remembers mislabels misleading. Even the logits for a mislabeled class are not the highest, their magnitude may still suffice for effective LiRA.
C EVALUATION SETUP OF FIGURE AND FIGURE 4
For Figure 3 the experiments were conducted on CIFAR- 10, where used 500 mislabeled samples to simulate the most vulnerable data. The attack settings follow those of LiRA. For each defense, trained 16 models, ensuring that each sample appeared in the training set of only half of the models. We used leave-one-out cross-validation-ed eech time using one model as the victim and the remaining 15 models the attacker's shadow models. The shadow models were implemented using a ConvNet architecture. We evaluated privacy protection levels for both the average case and the worst case.
Figure 4 follows the same experimental setting as the undefended baseline without any defense, described in Appendix [..]] except that we set the number shadow models to 256 instead of 32. For the "average-case" subfigure, we selected normal sample and plotted its loss distributions when was member and when it was not. For the "most vulnerable" subfigure, we selected a mislabeled canary and did the same.
18


Reviews:
[{"review_id": "l5xrct26Ht", "reviewer": "Reviewer_qB2Z", "summary": "The paper studies the privacy-preserving properties of commonly-used synthetic image generation methods. The measuring of privacy-preservation is carried out using membership inference attacks using the attacks given by [Aerni et al. (2024)](https://arxiv.org/pdf/2404.17399). The paper compares the privacy-preservation of four commonly-used techniques: coreset selection, dataset distillation, data-free knowledge, model distillation, and synthetic data generated from diffusion models. The experiments are carried out on CIFAR-10 and the most vulnerable samples are mislabeled samples.", "questions": "- Do you think similar comparisons could be easily carried out in different image datasets or in other domains (e.g., tabular data) ? \n\n- What is the setting behind Figure 4? There are no details given on that experiment.\n\n- When focussing on practical scenarios: do you think the situation would differ, if instead of using synthetic samples (e.g., mislabeled samples) for the auditing, you would try to find the most vulnerable data samples in the dataset? \n\n- Before the experimental setup is presented on page 6, there are some experimental results presented on page 5 (Figures 3 and 4) for which no sufficient details are given. I cannot be sure for which dataset is the result of Figure 4 is, and what is exactly the setting behind that figure.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is very well written and the experiments are well-explained and seem solid.\n\n- The paper can serve as a good reference for showing the strength of DP-SGD for obtaining good privacy-utility trad-eoff for synthetic image data.", "weaknesses": "- The paper does not truly present many novel ideas, though it is another valuable demonstration about the effectiveness of DP-SGD for obtaining good privacy-utility tradeoff for ML models when the privacy protection is measured via the most vulnerable samples. \n\nRegarding the idea of auditing with worst-case samples, for example: it has been studied extensively and is already considered by Carlini et al, 2019, \"The secret sharer: Evaluating and testing unintended memorization in neural networks\" and Jagielski et al., 2020, \"Auditing differentially private machine learning: How private is private sgd?\" So, although auditing with worst-case samples seems to be a central theme in the paper, there is not much novelty in there (and I also think these references should be included).\n\nAlso, the vulnerability of synthetic data against membership inference attacks has been considered in the literature, and there also seem to be some central references missing: see e.g. the work by Hayes et al., 2019, \"LOGAN: Membership Inference Attacks Against Generative Models.\"\n\n- The experimental comparison is a bit restricted, after all, since only the CIFAR-10 dataset is considered. Perhaps one could consider datasets from other domains as well?\n\nDespite of its strengths, I am leaning towards reject as I think the paper does not provide sufficiently novel results to reach the bar for this venue. Nevertheless, I think that with some rewriting this will be a nice paper and can serve as a reference for this topic (privacy-protection of synthetic image data).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1X2GCjirDE", "reviewer": "Reviewer_cuks", "summary": "This paper measures empirical privacy for several methods of training vision models from synthetic data that claim to preserve privacy to some degree with respect to the training data: CoreSet selection, Dataset Distillation, Date-Free Knowledge Distillation, and synthetic data from Fine-Tuned Diffusion models. Empirical privacy is evaluated using the Likelihood Ratio Attack (LiRA) and the general setup follows [0]. As a private baseline, DP-SGD is used.\n\nFor the evaluation, a ResNet architecture is used with the CIFAR-10 dataset. For each of these methods, the authors measure privacy leakage (true positive rate given a fixed false positive rate), utility (test set accuracy), and efficiency (# of training hours). The main findings are that there's a clear privacy-utility tradeoff among methods once privacy is considered as a worst-case rather than an average case notion and that none of the synthetic data methods outperform DP-SGD with the three metrics jointly considered.", "questions": "1. Why are the gridlines in Figure 1 not uniform? Consider changing this or explaining in figure caption.\n2. As with [0], are the DP-SGD models reported in Table 4 only non-private due to hyperparameter tuning? If so, what privacy parameters were these models trained with?\n\nAdditional stylistic notes:\n- Odd phrasing on Line 83: \"none of these fancy methods with synthetic data\"\n\n- Typo on Line 95: \"it can provide a decent privacy protection\"\n\n- Odd phrasing Line 244: \"but not evaluated in the right way\"\n\nAdditional suggestions:\n- When discussing the intuition for why initialization on public data provides privacy guarantees, consider referencing [1, 2, 3], which illustrate how public data can be used to improve differentially private synthetic data. \n\n[0] Aerni, Michael et al. \"Evaluations of Machine Learning Privacy Defenses are Misleading.\" 2024.\n\n[1] Liu, Terrance, et al. \"Leveraging public data for practical private query release.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Liu, Terrance, et al. \"Iterative methods for private synthetic data: Unifying framework and new methods.\" Advances in Neural Information Processing Systems 34 (2021): 690-702.\n\n[3] Fuentes, Miguel, et al. \"Joint Selection: Adaptively Incorporating Public Information for Private Synthetic Data.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.\n\n[4] Papernot, Nicolas, and Thomas Steinke. \"Hyperparameter tuning with renyi differential privacy.\" ICLR, 2022.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- This paper provides an apples-to-apples empirical privacy comparison for several methods of training vision models that claim to preserve privacy to some degree. In doing so, it improves upon prior empirical privacy analyses, some that were rather flawed.\n- The presentation is clear and the text is well-written. The figures throughout are particularly helpful. See question 1 on this point.", "weaknesses": "- The DP-SGD comparison is potentially misleading. The \"baseline\" method does not satisfy differential privacy. See question 2. The paper would benefit from including an additional baseline that satisfies DP e.g. [4] discusses how to incorporate hyperparameter tuning into the privacy analysis. It may also be interesting to include a fully non-private baseline using a standard training routine i.e. just train ResNet with SGD. \n- The paper would benefit from a discussion of how the formal guarantee of differential privacy differs from heuristic privacy - particularly, with respect to DP applying to adding or removing any possible possible record within the data domain; whereas, the MI attacks in this paper are only considered with respect to the training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TY9OfcesFy", "reviewer": "Reviewer_ybiF", "summary": "This paper investigates whether using synthetic data in machine learning genuinely safeguards privacy, as often claimed. The evaluations is done on 4 different training paradigms-coreset selection, data distillation, data-free knowledge distillation and synthetic data generated from diffusion models. To test the privacy claims of these methods, the study uses membership inference attacks (MIAs), focusing on worst-case scenarios to rigorously assess privacy leakage. The paper also compares these methods to Differential Privacy Stochastic Gradient Descent (DPSGD), a technique known for providing formal privacy guarantees, and finds that DPSGD consistently outperforms synthetic data-based approaches in terms of the privacy-utility-efficiency balance. The findings reveal that none of the synthetic data techniques match DPSGD in safeguarding privacy effectively. Notably, the study also discovers that visual dissimilarity between synthetic and private data does not necessarily imply privacy protection, as even visually distinct synthetic data can leak information when model logits are similar. This highlights a risk that methods relying solely on visual or distributional differences may offer a false sense of privacy.", "questions": "Why do you consider coreset selection as synthetic data?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This broad approach offers a thorough understanding of various methodologies in synthetic data utilization and their impact on privacy.\nThe study juxtaposes synthetic data-based techniques with Differential Privacy-SGD (DPSGD) as a baseline, which helps readers contextualize the efficacy of synthetic data methods in privacy preservation compared to a gold-standard approach like DPSGD.\nThe study identifies instances where synthetic data, despite visual dissimilarity from private data, can still leak privacy information through logit similarities. This nuanced finding enhances the paper's depth by showing that visual similarity alone isn’t sufficient to evaluate privacy.", "weaknesses": "The experiments focus on CIFAR-10 and specific models, such as ResNet-18, which may limit the generalizability of findings. The paper’s findings could vary across more complex datasets or architectures, and broader experiments could better represent the implications for privacy in diverse real-world scenarios​.\n\nTechniques like DPSGD are noted for efficiency, yet they are resource-intensive. The paper briefly mentions but does not deeply engage with the practical constraints of computational cost and scalability, which are critical factors for real-world implementation of privacy-preserving methods​.\n\nWhile the empirical evaluation is thorough, the paper lacks an in-depth theoretical framework to explain why certain synthetic data techniques lead to privacy leakage. A theoretical grounding could bolster the empirical findings and offer predictive insights for synthetic data privacy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q80Zycg8KC", "reviewer": "Reviewer_YLFf", "summary": "This paper provides a systematic analysis for privacy risk of the usage of synthetic data. The investigated methods include core set selection, dataset distillation, data-free distillation and diffusion models. The estimated privacy leakage is based on 500 canaries that are mislabeled and LiRA by shadow models. The results show that the privacy risk still remains and higher than DP-SGD at the same comparison of utility. This work also provides several detail analysis including the logits similarity.", "questions": "1. It is interesting to see the analysis for visual similarity and logit similarity. I did not quite follow the analysis for why the logit similarity of the synthetic data in DFKD leads to the higher MIA metric of private data. Figure 9 shows that the logit of the private data in student model prediction is much different than the remaining three, therefore I wonder if the authors could explain why this logits similarity of the synthetic data could lead to the higher MIA metric.\n\n2. Figure 7 shows that private data that has high confidence in original label and Figure 9 shows the private data that has higher confidence in the mislabel label. Would this show the different canary property for different kind of methods.\n\n3. The dataset distillation method (random/OOD) shows utility drop compared to DP-SGD. Would increasing the number of synthetic data help improve utility?\n\n4. It seems to me the TPR value granularity for 500 canaries is 0.2% (1/500) and I wonder how is the TPR reported in Tables are calculated. Or in other words, the authors may provide a detailed description about how MIA metric is calculated.\n\n5. I wonder if the author could provide an adaptive attack that achieves higher attacker performance based on Figure 7 findings where the LiRA achieves a lower MIA metric. \n\nI am happy to increase my score if I find my concerns addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This work presents a systematic evaluation of privacy metrics TPR-FPR by LiRA for four synthetic data generation methods, and show that privacy leakage of synthetic data is much higher in than the results in previous literature.\n- This work also provide some detailed analysis including considering visually similarity and logits similarity together instead of relying on the single MIA metric.", "weaknesses": "- The analysis could be improved (see questions below).\n- The technical contribution is somewhat limited. For example, the privacy estimation metric is based on previous work LiRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)