# Generating Higher-Fidelity Synthetic Datasets with Privacy Guarantees
## 3 Results
### Learning Performance  
Here, we evaluate the generalisation ability of the student model trained on artificial data. More specifically, we train a student model on generated data and report test classification accuracy on a real held-out set. The goal of this experiment is to show that having a privacy-preserving generative model we can use synthetic samples to fully replace the real data. Not only does it allow us to eliminate manual labelling of real (and potentially sensitive) data, but also expand the set of problems that can be solved by FL (task T5 in [PERSON] et al. [8] classification). For example, some medical data cannot be automatically annotated, and users are not qualified to do that, so high-quality synthetic data would allow the annotation to be performed by doctors without privacy risks for users. We imitate human annotation by training a separate classifier (with the same privacy guarantee as the generative model) and using it to label synthetic images. While this approach is somewhat different from prior work on generating data for training ML models, comparisons in this section are still valid because our annotator maintains the same privacy guarantee. We choose to compare with the method called G-PATE [29], because it is one of the best recent techniques in terms of privacy-utility trade-off. The authors showed that it outperforms another PATE-based approach, PATE-GAN [28], as well as DP-GAN [26], based on DP-SGD. Student model accuracy is shown in Table 1. Apart from G-PATE, we compare our method to a non-private classifier trained directly on the real dataset, and a private classifier, trained on the real dataset with Bayesian DP. In the case of generative models, the same (non-private) classifier is trained on the private synthetic output. All results in the table are obtained with the privacy guarantee of \((1,10^{-5})\)-DP, or \((1,10^{-10})\)-BDP, which is equivalent to \((1,10^{-5})\)-DP for this data with high probability. Although [PERSON] et al. [29] report improved results for \((10,10^{-5})\)-DP, we do not include those in the study for two reasons. First, \(\varepsilon=10\) is too high for providing a meaningful guarantee [10]. Second, their accuracy is still lower than the one our method attains under stricter \(\varepsilon=1\). Generally, we observe that on these datasets switching from real to synthetic data does not significantly deteriorate accuracy of the student model while maintaining strong theoretical privacy guarantees. On MNIST, the drop in performance between a private discriminative and a private generative approach is less than 2%. It is more noticeable on Fashion-MNIST, but is still within about 7% and is still lower than the drop between non-private and private classifiers. In another experiment, Figure 3a,b show a percentage of maximum accuracy (i.e., numbers reported in Table 1) achievable by the model when only a part of the generated data is labelled. Notably, the models trained on synthetic data achieve the same percentage faster than the ones trained on the real data, which is especially evident for the Fashion-MNIST dataset. Moreover, as little as 100 labelled samples is enough to outperform models trained on data generated with comparable DP guarantees. Using non-private synthetic data (not shown in the table) allows us to achieve somewhat better results. On MNIST, the student model reaches 96.09%, while on Fashion-MNIST, 84.86%. It suggests that about half of the accuracy loss comes from the limited capacity of \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Dataset** & **Non-Private** &
\begin{tabular}{c} **Private** \\ **Classifier** \\ \end{tabular} & **G-PATE** & **Our Approach** \\ \hline MNIST & 99.20\% & 95.59\% & 56.31\% & 93.64\% \\ Fashion-MNIST & 91.51\% & 82.20\% & 51.74\% & 76.83\% \\ \hline \hline \end{tabular}
\end{table}