# Generating Higher-Fidelity Synthetic Datasets with Privacy Guarantees
## 1 Introduction  
With machine learning (ML) becoming ubiquitous in many aspects of our society, questions of its privacy and security take centre stage. A growing field of research in privacy attacks on ML [1; 2; 3; 4] tells us that it is possible to infer information about training data even in a black-box setting, without access to model parameters. A wider population, however, is concerned with privacy practices used in the ML development cycle, such as company employees or contractors manually inspecting and annotating user data ([[https://www.theguardian.com/technology/2020/jan/10/skype-audio-graded-by-workers-in-china-with-no-security-measures](https://www.theguardian.com/technology/2020/jan/10/skype-audio-graded-by-workers-in-china-with-no-security-measures)]([https://www.theguardian.com/technology/2020/jan/10/skype-audio-graded-by-workers-in-china-with-no-security-measures](https://www.theguardian.com/technology/2020/jan/10/skype-audio-graded-by-workers-in-china-with-no-security-measures)), accessed on 29 May 2022, [[https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio](https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio)]([https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio](https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio)), accessed on 29 May 2022). The problem of privacy attacks is often tackled by adding a differentially private mechanism to the model training procedure [5]. Differential privacy (DP) [6] provides a rigorous theoretical guarantee, which states (informally) that the algorithm output would not significantly change when a single user adds or removes their data, except with small (failure) probability. Another approach gaining popularity is _federated learning_ (FL), introduced by [PERSON] et al. [7], where a central entity trains a model by computing updates locally on-device and then securely aggregating these updates on a server. This way user data never leave their devices. In spite of significant progress, neither of these approaches solves the problem of manual data labelling. Moreover, it creates an additional hurdle for developers, as they cannot inspect data, especially in decentralised settings, making it difficult to understand the model behaviour and find bugs in data and implementations. [PERSON] et al. [8] provide a more complete characterisation of these questions. This paper follows [8] in adopting generative adversarial networks (GAN) [9] trained in a privacy-preserving manner for addressing these issues. More specifically, we use the notion of Bayesian differential privacy (BDP) [10], which takes into account the data distribution and provides a more meaningful guarantee for in-distribution samples than classical DP. Intuitively, the traditional DP has uniform failure probability for all data points, including the ones that come from a completely different distribution and even pure noise. This kind of data creates difficulties for machine learning as they will significantly reduce the accuracy of the resulting model, and other measures must be taken before applying a privacy mechanism so that it does not enter the dataset. BDP allows us to focus the privacy guarantee on datasets that do not contain instances that are \"impossible\" or come from a different distribution, and thus provides a more realistic privacy bound. The fact that the privacy guarantee may not hold for such outliers can be seen as an advantage as it would allow us to recognise and filter them to eliminate their influence on the learned model. Since both DP and BDP can use the same obfuscation mechanism, while computing two privacy bounds in parallel, a DP guarantee would still hold for out-of-distribution samples. More details on the overall approach and privacy are provided in Section 2.3.