# Generating Higher-Fidelity Synthetic Datasets with Privacy Guarantees
## 2 Materials and Methods
### Background  
We use \(D,D^{\prime}\) to represent neighbouring (adjacent) datasets. If not specified, it is assumed that these datasets differ in a single example. Individual examples in a dataset are denoted by \(x\) or \(x_{i}\), while the example by which two datasets differ--by \(x^{\prime}\). We assume \(D^{\prime}=D\cup\{x^{\prime}\}\), whenever possible to do so without loss of generality. The private learning outcomes (i.e., noised gradients) are denoted by \(w\). **Definition 1**. : _A randomised function (mechanism) \(\mathcal{A}:\mathcal{D}\to\mathcal{R}\) with domain \(\mathcal{D}\) and range \(\mathcal{R}\) satisfies \((\varepsilon,\delta)\)-differential privacy if for any two adjacent inputs \(D,D^{\prime}\in\mathcal{D}\) and for any set of outcomes \(\mathcal{S}\subset\mathcal{R}\) the following holds:_  
\[\Pr[\mathcal{A}(D)\in\mathcal{S}]\leq e^{\varepsilon}\Pr\bigl{[} \mathcal{A}(D^{\prime})\in\mathcal{S}\bigr{]}+\delta. \tag{1}\]  
**Definition 2**. : _Privacy loss of a randomised mechanism \(\mathcal{A}:\mathcal{D}\to\mathcal{R}\) for inputs \(D,D^{\prime}\in\mathcal{D}\) and outcome \(w\in\mathcal{R}\) takes the following form:_  
\[L(w,D,D^{\prime})=\log\frac{\Pr[\mathcal{A}(D)=w]}{\Pr[\mathcal{A}(D^{\prime} )=w]}. \tag{2}\]  
**Definition 3**. : _The Gaussian noise mechanism achieving \((\varepsilon,\delta)\)-DP, for a function \(f:\mathcal{D}\to\mathbb{R}^{m}\), is defined as_  
\[\mathcal{A}(D)=f(D)+\mathcal{N}(0,I\sigma^{2}), \tag{3}\]_where \(\sigma>C\sqrt{2\log\frac{1.25}{\delta}}/\varepsilon\) and \(C\) is the L2-sensitivity of \(f\)._  
For more details on differential privacy, the Gaussian mechanism, and how to use it in machine learning, we refer the reader to [5; 30]. Let us now present the notion of Bayesian DP, initially introduced in [10]. **Definition 4**. : _A randomised function (algorithm) \(\mathcal{A}:\mathcal{D}\rightarrow\mathcal{R}\) with domain \(\mathcal{D}\) and range \(\mathcal{R}\) satisfies \((\varepsilon_{\mu},\delta_{\mu})\)-Bayesian differential privacy if for any two adjacent datasets \(D,D^{\prime}\in\mathcal{D}\), differing in a single data point \(x^{\prime}\sim\mu(x)\), and for any set of outcomes \(\mathcal{S}\subset\mathcal{R}\) the following holds:_  
\[\Pr[\mathcal{A}(D)\in\mathcal{S}]\leq e^{\varepsilon_{\mu}}\Pr\big{[} \mathcal{A}(D^{\prime})\in\mathcal{S}\big{]}+\delta_{\mu}. \tag{4}\]  
Subscript \(\mu\) highlights the main difference with the classic DP: in the original definition the probability is taken only over the randomness of the outcome (\(w\)), while in BDP over two random variables (\(w\) and \(x^{\prime}\)). Therefore, the privacy parameters \(\varepsilon\) and \(\delta\) depend on the data distribution \(\mu(x)\), although it is not required to be known. This modification changes the meaning of \(\delta_{\mu}\) compared to the \(\delta\) of DP: it now also accounts for the privacy mechanism failures in the tails of data distributions in addition to the tails of the outcome distributions. Lastly, data samples are assumed to be exchangeable [31], a requirement for tighter privacy accounting, which is satisfied in many real-world scenarios. Basic properties of DP are maintained by BDP, making it widely applicable:  
**Proposition 1** (Post-processing). : _Let \(\mathcal{A}:\mathcal{D}\rightarrow\mathcal{R}\) be a \((\varepsilon_{\mu},\delta_{\mu})\)-Bayesian differentially private algorithm. Then for any arbitrary randomised data-independent mapping \(f:\mathcal{R}\rightarrow\mathcal{R}^{\prime}\), \(f(\mathcal{A}(D))\) is \((\varepsilon_{\mu},\delta_{\mu})\)-Bayesian differentially private._  
**Proposition 2** (Basic composition). : _Let \(\mathcal{A}_{i}:\mathcal{D}\rightarrow\mathcal{R}_{i}\), \(\forall i=1\ldots k\), be a sequence of \((\varepsilon_{\mu},\delta_{\mu})\)-Bayesian differentially private algorithms.