# Generating Higher-Fidelity Synthetic Datasets with Privacy Guarantees
## 2 Materials and Methods
### Related Work  
A rapidly expanding area of privacy-preserving machine learning research has been recently focused on the attacks that compromise privacy of training data, such as model inversion [1] and membership inference [2]. The former is based on observing the output probabilities of the target model for a given class and performing gradient descent on an input reconstruction. The latter assumes an attacker with access to similar data, which is used to train \"shadow\" models, mimicking the target, and the attack model, which predicts if a certain example has already been seen during training based on its output probabilities. Both attacks can be performed in a black-box setting, without access to the model internal parameters. Differential privacy (DP) [6] is widely accepted as the gold standard for preventing such attacks. One of the early takes on the problem is to use disjoint datasets and distributed training with DP. For example, ref. [11] proposes to train a model in a distributed manner by communicating sanitised updates from participants to a central authority. Such a method, however, yields high privacy losses [5; 12]. An alternative technique suggested by [12] also uses disjoint training sets and builds an ensemble of independently trained teacher models to transfer knowledge to a student model by labelling public data. This result has been extended in [13] to achieve state-of-the-art image classification results in a private setting (with single-digit DP bounds). A different approach is taken by [5]. They propose using differentially private stochastic gradient descent (DP-SGD) to train deep learning models in a private manner. This approach achieves high accuracy maintaining relatively low DP bounds and being simpler to implement, but may also require pre-training on public data. Due to the fact that the DP threat model is extremely broad, achieving a reasonable guarantee may be difficult or even impossible. For this reason, a number of alternative definitions has been proposed over recent years, aimed at relaxing the guarantee or providing tighter composition bounds under certain assumptions. Examples are computational DP [14], mutual-information privacy [15; 16], different versions of concentrated DP (CDP [17], zCDP [18], tCDP [19]), and Renyi DP (RDP) [20]. Some other relaxations [21; 22; 23] tip the balance even further in favour of applicability at the cost of weaker guarantees, for example considering the average-case instead of the worst-case [24]. In this work, we rely on another relaxation, called Bayesian differential privacy [10]. This notion utilises the fact that data come from a particular distribution, and not all data samples are equally likely (e.g., unlikely to find a sound record among ECG samples). At the same time, it maintains a similar probabilistic interpretation of its parameters \(\varepsilon\) and \(\delta\). It is worth noting, that unlike some of the relaxations mentioned above, Bayesian DP can provide a tail bound on privacy loss, similarly to the moments accountant (MA) [5], and is not limited to a particular dataset, but rather a particular type of data (e.g., emails, MRI images, etc. ), which is a much more permitting assumption. Up until recently, another aspect of privacy in machine learning has been largely overlooked: the human involvement in the development cycle and manual data processing. These issues can be mitigated, at least partially, by federated learning (FL) [7], which brings a great promise for user privacy. Yet, FL paradigm creates additional problems of its own. [PERSON] et al. [8] provide a good starting point, systematising these problems and proposing a solution by the use of synthetic data. Although privacy-preserving data synthesis using GANs has been introduced in earlier works [24; 25; 26; 27; 28; 29], these papers mainly focused on achieving high utility of synthetic data without addressing a broader scope of privacy leakage via manual data handling.