# Generating Higher-Fidelity Synthetic Datasets with Privacy Guarantees
## 2 Materials and Methods
### Privacy Analysis  
In order to compute privacy guarantees of the synthetic dataset w.r.t. the real one, we need to bound privacy loss of the generative model. As noted before, we effectively enforce privacy on the critic and then rely on preservation of guarantees under post-processing. This arrangement ensures a simple adoption of privacy accounting for discriminative models. Privacy accounting is performed by using the Bayesian accountant [10]. To benefit from the data distribution information, it needs to sample a number of gradients at each iteration in addition to the one used in the update. These gradients are then used to estimate the upper confidence bound on the privacy cost, generally defined as \(c_{r}(\lambda)=\log\mathbb{E}_{x^{\prime}}\big{[}e^{r\lambda D_{\lambda+1}(p_ {D}\|p_{O^{\prime}})}\big{]}^{\frac{1}{\lambda}}\) for some exponent \(r\). More precisely, for the Gaussian noise mechanism that we use in this paper, let  
\[c_{t}(\lambda)=\max\{c_{t}^{L}(\lambda),c_{t}^{R}(\lambda)\}, \tag{5}\]where  
\[c_{t}^{L}(\lambda) =\frac{1}{T}\log\mathbb{E}_{x^{\prime}}\Bigg{[}\mathbb{E}_{k\sim B (\lambda+1,\phi)}\bigg{[}e^{\frac{\lambda^{2}-k}{2\phi^{2}}\|g_{t}-g_{t}^{2}\|^ {2}}\bigg{]}^{T}\Bigg{]}, \tag{6}\] \[c_{t}^{R}(\lambda) =\frac{1}{T}\log\mathbb{E}_{x^{\prime}}\Bigg{[}\mathbb{E}_{k\sim B (\lambda_{d})}\bigg{[}e^{\frac{\lambda^{2}+k}{2\phi^{2}}\|g_{t}-g_{t}^{2}\|^{ 2}}\bigg{]}^{T}\Bigg{]}. \tag{7}\]  
Here, \(B(\lambda,q)\) is the binomial distribution with \(\lambda\) experiments (a hyper-parameter) and the probability of success \(q\) (equal to the probability of sampling a single data point in a batch), \(g_{t}\) and \(g_{t}^{\prime}\) are two gradient samples differing in one data point, and \(T\) is the total number of learning iterations/rounds. These expressions can be derived from the definition, as shown in [10; 33]. In practice, gradient sampling can be performed in the following way. At each optimisation step, we simply add or remove \(m\) random data points to/from the batch and recompute the corresponding gradient. Let \(\overline{G}_{t}\) denote a sample mean of the expressions inside \(\mathbb{E}_{x^{\prime}}[\cdot]\) in Equations (6) and (7), computed with these gradients, and \(S_{t}\) be a sample standard deviation. We can then estimate the privacy costs (6) and (7) for the given gradient distribution using a Bayesian estimator:  
\[\hat{c}_{t}(\lambda;\gamma,m)=\frac{1}{T}\log\bigg{[}\overline{G}_{t}+\frac{F^ {-1}(1-\gamma,m-1)}{\sqrt{m-1}}S_{t}\bigg{]}, \tag{8}\]  
where \(F^{-1}(1-\gamma,m-1)\) is the inverse of the Student's \(t\)-distribution CDF at \(1-\gamma\) with \(m-1\) degrees of freedom. This estimate could be interpreted as an upper bound on the privacy cost with probability \((1-\gamma)\) under the assumption of an uninformative prior. Again, we refer the reader to [10; 33] or the accountant implementation ([[https://github.com/[PERSON]/bayesian-differential-privacy](https://github.com/[PERSON]/bayesian-differential-privacy)]([https://github.com/[PERSON]/bayesian-differential-privacy](https://github.com/[PERSON]/bayesian-differential-privacy)), accessed on 29 May 2022) for further details. Using these per-iteration privacy costs, we can calculate the total privacy cost via the advanced composition theorem:  
**Theorem 1** (Advanced Composition). : _Let \(\mathcal{A}^{(1:T)}=(\mathcal{A}^{(1)},\ldots,\mathcal{A}^{(T)})\) be a sequence of privacy mechanisms. Then the total privacy cost \(c(\lambda)\) of \(\mathcal{A}^{(1:T)}\) satisfies_  
\[c(\lambda)\leq\sum_{t=1}^{T}c_{t}(\lambda).\]  
The privacy guarantee is calculated from the privacy cost, by fixing either \(\varepsilon_{\mu}\) or \(\delta_{\mu}\):  
\[\log\delta_{\mu}\leq\sum_{t=1}^{T}c_{t}(\lambda)-\lambda\varepsilon_{\mu}. \tag{9}\]  
Note that here we use a more lightweight version of notation for readability. In practice, one needs to keep in mind, for example, possible auxiliary information and maximise privacy cost over it. For more in-depth analysis and related proofs, see [33].