SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widehat {\mathcal {L} _ {\mathrm {M M D} ^ {2}}} (p, q) = \frac {1}{N ^ {2}} \sum_ {i = 1} ^ {N} \sum_ {p = 1} ^ {N} k (\mathbf {x} _ {i}, \mathbf {x} _ {p}) - \frac {2}{N M} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {M} k (\mathbf {x} _ {i}, \mathbf {w} _ {j}) + \frac {1}{M ^ {2}} \sum_ {j = 1} ^ {M} \sum_ {q = 1} ^ {M} k (\mathbf {w} _ {j}, \mathbf {w} _ {q}). \tag {6}
$$
DP-MERF approximates the kernel by:  $k(\mathbf{x},\mathbf{w}) = \widehat{\phi} (\mathbf{x})^{\top}\widehat{\phi} (\mathbf{w})$ , where  $\widehat{\phi} (\mathbf{x})\in \mathbb{R}^{d}$  is the random Fourier feature [25] and  $d$  is the feature dimension. Now the loss becomes:
$$
\widehat {\mathcal {L} _ {\mathrm {M M D} _ {r f} ^ {2}}} (p, q) = \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \widehat {\phi} (\mathbf {x} _ {i}) - \frac {1}{M} \sum_ {j = 1} ^ {M} \widehat {\phi} (\mathbf {w} _ {j}) \right\| _ {2} ^ {2} = \left\| \widehat {\mu_ {p}} - \frac {1}{M} \sum_ {j = 1} ^ {M} \widehat {\phi} (\mathbf {w} _ {j}) \right\| _ {2} ^ {2},
$$
where  $\widehat{\mu_p} = \frac{1}{N}\sum_{i = 1}^N\widehat{\phi} (\mathbf{x}_i)$  is the empirical KME of  $p$ . Calibrated Gaussian noise is added to  $\widehat{\mu_p}$  to obtain  $\widetilde{\mu_p}$  with DP guarantees, which can be viewed as privatized statistics of the whole real dataset. Thereafter, the training objective is to match the KME of generated data with  $\widetilde{\mu_p}$ , without querying the real data any longer. A line of recent follow-up works, e.g., PEARL [19] and DP-HP [31], boil down to improving the finite-dimensional truncation and show some further improvement in utility.
Compared to other DPGM approaches via DP-SGD [1], DP-MERF is appealing in two aspects: (1) training efficiency, since noise is added to the KME of the whole database once and for all, whereas DP-SGD has to clip and perturb the gradient in each training iteration, which leads to significant training time overhead; (2) more scalable to smaller  $\epsilon$ , e.g.,  $\epsilon = 1$  or below.
However, we observe that the generation by DP-MERF (and related followup methods) resembles "mean-like" images (e.g. see Figures 1 to 3), which can be explained by their training objective, because matching the KME of all data is likely to lead to mode collapse. Moreover, truncating the RKHS into a finite-dimensional space makes it easy to add Gaussian noise, but at the cost of potentially losing the fine ability to distinguish the data distribution from generation (any finite-dimensional RKHS is not characteristic). Therefore, we turn to study the possibility of adding noise in the infinite-dimensional RKHS directly.
<sup>7</sup>By Moore-Aronszajn theorem, any symmetric, positive definite kernel  $k$  on  $\mathcal{X}$  defines a unique Hilbert space of functions on  $\mathcal{X}$  where  $k$  is a reproducing kernel. We can find a feature map  $\phi : \mathcal{X} \to \mathcal{H}$ , such that  $k(\mathbf{x},\mathbf{w}) = \langle \phi(\mathbf{x}), \phi(\mathbf{w}) \rangle_{\mathcal{H}}$  and  $\langle \cdot, \cdot \rangle_{\mathcal{H}}$  is the inner product on  $\mathcal{H}$ .
4.3 Methodology
We use Gaussian kernel as our kernel function  $k$ , i.e.,  $k(\mathbf{x}, \mathbf{w}) = \exp \left\{-\frac{\|\mathbf{x} - \mathbf{w}\|_2^2}{2h^2}\right\}$ , since it is widely used in related works [13, 19, 31]. Define
$$
f _ {D} = \frac {1}{N} \sum_ {i = 1} ^ {N} \phi (\mathbf {x} _ {i}) = \frac {1}{N} \sum_ {i = 1} ^ {N} k (\mathbf {x} _ {i}, \cdot),
$$
so we can rewrite Eq. (6) by plugging in  $f_{D}$ :
$$
\widehat {\mathcal {L} _ {\mathrm {M M D} ^ {2}}} = \frac {1}{N} \sum_ {p = 1} ^ {N} f _ {D} (\mathbf {x} _ {p}) - \frac {2}{M} \sum_ {j = 1} ^ {M} f _ {D} (\mathbf {w} _ {j}) + \frac {1}{M ^ {2}} \sum_ {j = 1} ^ {M} \sum_ {q = 1} ^ {M} k (\mathbf {w} _ {j}, \mathbf {w} _ {q}).
$$
Privatizing the terms relating to real data  $\mathbf{x}_i\in D$ $(i = 1,2,\ldots ,N)$  amounts to privatizing the function  $f_{D}\in \mathcal{H}$  by Corollary 2, which leads to our private training objective:
$$
\widetilde {\mathcal {L} _ {\mathrm {M M D} ^ {2}}} = \frac {1}{N} \sum_ {p = 1} ^ {N} \widetilde {f _ {D}} (\mathbf {x} _ {p}) - \frac {2}{M} \sum_ {j = 1} ^ {M} \widetilde {f _ {D}} (\mathbf {w} _ {j}) + \frac {1}{M ^ {2}} \sum_ {j = 1} ^ {M} \sum_ {q = 1} ^ {M} k (\mathbf {w} _ {j}, \mathbf {w} _ {q}), \tag {7}
$$
where  $\mathbf{w} = g_{\theta}(\mathbf{z})$ , i.e.,  $\mathbf{w}$  is generated from standard Gaussian noise  $\mathbf{z}$  by a generative neural network  $g_{\theta}$ . Given the kernel is Gaussian, we can easily bound the sensitivity of  $f_{D}$  in the RKHS norm. Without loss of generality, assume  $D, D'$  only differ in the last element, i.e.,  $\mathbf{x}_N \neq \mathbf{x}_N'$ . Then,
$$
\left(f _ {D} - f _ {D ^ {\prime}}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} \phi (\mathbf {x} _ {i}) - \frac {1}{N} \sum_ {i = 1} ^ {N} \phi \left(\mathbf {x} _ {i} ^ {\prime}\right) = \frac {1}{N} \left(\phi \left(\mathbf {x} _ {N}\right) - \phi \left(\mathbf {x} _ {N} ^ {\prime}\right)\right).
$$
Now we play the kernel trick  $k(\mathbf{x},\mathbf{y}) = \langle \phi (\mathbf{x}),\phi (\mathbf{y})\rangle_{\mathcal{H}}$  again:
$$
\| f _ {D} - f _ {D ^ {\prime}} \| _ {\mathcal {H}} ^ {2} = \frac {1}{N ^ {2}} \big (k (\mathbf {x} _ {N}, \mathbf {x} _ {N}) - 2 k (\mathbf {x} _ {N}, \mathbf {x} _ {N} ^ {\prime}) + k (\mathbf {x} _ {N} ^ {\prime}, \mathbf {x} _ {N} ^ {\prime}) \big) \leq \frac {2}{N ^ {2}} := \Delta^ {2}.
$$
We follow the batch method in Hall et al. [12] to release function  $\widetilde{f_D}$  in practice, as it naturally fits the batch training manner, which amounts to sampling a path from the Gaussian process specified by any finite collection (batch) of points. Assuming the batch size is  $m$ , we concatenate  $(\mathbf{x},\mathbf{w}) = \mathbf{s}$  (of size  $2m$ ,  $\mathbf{x} = \mathbf{s}_{[1:m]},\mathbf{w} = \mathbf{s}_{[m + 1:2m]}$ ), to save an additional privacy cost incurred by an additional sample path in each training iteration. Now we release  $\widetilde{f_D} (\mathbf{x}_i)$  and  $\widetilde{f_D} (\mathbf{w}_j)$  via  $\widetilde{f_D} (\mathbf{s})$  as needed in Eq.(7) by:
$$
\left( \begin{array}{c} \widetilde {f _ {D}} (\mathbf {s} _ {1}) \\ \vdots \\ \widetilde {f _ {D}} (\mathbf {s} _ {2 m}) \end{array} \right) \sim \mathcal {N} \left(\left( \begin{array}{c} f _ {D} (\mathbf {s} _ {1}) \\ \vdots \\ f _ {D} (\mathbf {s} _ {2 m}) \end{array} \right), \sigma \Delta \left( \begin{array}{c c c} k (\mathbf {s} _ {1}, \mathbf {s} _ {1}) & \dots & k (\mathbf {s} _ {1}, \mathbf {s} _ {2 m}) \\ \vdots & \ddots & \vdots \\ k (\mathbf {s} _ {2 m}, \mathbf {s} _ {1}) & \dots & k (\mathbf {s} _ {2 m}, \mathbf {s} _ {2 m}) \end{array} \right)\right). \tag {8}
$$
In summary, each training iteration proceeds with (1) subsample a batch  $S$  from  $D$  to index the function  $f_{S}$ ; (2) perturb  $f_{S}$  by Corollary 2; (3) compute the loss function Eq. (7) by Eq. (8).
Extension to the conditional setting: We follow the approach in DP-MERF to encode labels in the MMD loss. Consider a new kernel  $k^*$  as a product of two existing kernels:  $k^*((\mathbf{x}, \mathbf{y}), (\tilde{\mathbf{x}}, \tilde{\mathbf{y}})) = k_{\mathbf{x}}(\mathbf{x}, \tilde{\mathbf{x}}) k_{\mathbf{y}}(\mathbf{y}, \tilde{\mathbf{y}})$ , where we set  $k_{\mathbf{x}}$  the same as the unconditional setting (i.e., Gaussian kernel) and  $k_{\mathbf{y}}$  to be polynomial kernel of order-1, i.e.,  $k_{\mathbf{y}}(\mathbf{y}, \tilde{\mathbf{y}}) = \mathbf{y}^\top \tilde{\mathbf{y}}$ . Now the function that we want to privately release becomes  $f_D^* = \frac{1}{N} \sum_{i=1}^{N} k^*((\mathbf{x}_i, \mathbf{y}_i), (\cdot, \cdot))$ . The sensitivity of  $f_D^*$  is the same as  $f_D$ . Thus, releasing  $f_D^*$  by the batch method is achieved by replacing  $f_D, k$  with  $f_D^*, k^*$  in Eq. (8).
4.4 Experiments
Since our method is based on making the RKHS features of a (Gaussian) kernel differentially private without truncating the kernel, we term our method DP-kernel. In this section, we will evaluate DP-kernel both qualitatively and quantitatively on three image benchmarks, including both grayscale and colorful images. All implementation details are given in Appendix C.
Datasets: We consider widely used image benchmarks in related works, i.e. MNIST [15], Fashion MNIST [34], and CelebA [20]. For MNIST and Fashion MNIST, we generate images conditioned on 10 respective labels. For CelebA, we condition on gender. Descriptions and pre-processing of the datasets are given in Appendix B.
Evaluation metrics: For quantitative comparison with baselines, we compute the following two metrics via 60k generated images under the same DP guarantees as baselines:
- Generation fidelity. We mainly use Fréchet Inception Distance (FID) [14], as it is widely compared in related works. However, recent studies suggest that FID is not always consistent with human evaluations (e.g. [28]), where the authors found that Inception-V3 (the feature extractor used to compute FID, which was proposed in 2015) is not competent enough, and switching to a more recent and powerful feature extractor (e.g. DINOv2 ViT-L/14) seems to overcome this issue. Therefore, we follow Stein et al. [28] and use their code to compute an additional Fréchet Distance, which is denoted by  $\mathrm{FD}_{\mathrm{DINOv2}}$ , for our presentation only.
- Generation utility. We train a convolutional neural network (CNN) as the classifier on generated images, then test the classifier on real images, where the performance is measured by the classification accuracy. We take 5 runs and report the average and standard deviation.
Baselines: We compare our method with the following related works: (1) kernel-based methods: DP-MERF [13], DP-HP [31], PEARL [19]; (2) others: DP-CGAN [29], GS-WGAN [5], DP-Sinkhorn [4], G-PATE [21], DataLens [33], DPDM [6]. All baselines are developed from v-RDP or v-DP.
Privacy regimes: Note that according to Definition 1, the DP guarantee is weak when  $\epsilon \geq 10$ , because  $\exp(10) \approx 2.2 \times 10^4$ , which is meaningless, because the two probabilities are presumed to be comparable for practical deployment (e.g.  $\epsilon \leq 1$ ). However, a line of recent SoTA DPGMs only generate acceptable images at  $\epsilon = 10$  [4, 5, 29]. Instead, we consider three values of  $\epsilon$ , i.e.,  $10, 1, 0.2$ , indicating three levels of DP guarantees, where we put comparison under  $(10, 10^{-5})$ -DP in Appendix D for completeness.
4.4.1 Conditional vs. parallel
The downstream classification task requires access to labels, so we need to consider conditional generative modeling, as introduced at the end of Section 4.3.
Alternatively, we can partition the dataset by labels, train  $K$  ( $K$  is the number of total classes) unconditional sub-models in parallel with DP guarantees, and release the union in the end, which could retain the same level of DP guarantee as the conditional one by parallel composition theorem for f-RDP (Theorem 4). Specifically, we partition the dataset  $D$  into  $K$  subsets by labels, and set the target  $\epsilon$  (for  $f_{D}$ ) the same for all  $f_{D_K}$ . Thus, releasing  $(f_{D_1},\dots ,f_{D_K})$  also satisfies  $(\alpha ,\epsilon)$ -RDP. We will show both conditional and parallel generation results in the following sections.
4.4.2 Grayscale image sets: MNIST and Fashion MNIST
We first tried our method on MNIST and Fashion MNIST. Qualitatively, while all baselines are able to generate reasonable images under  $(10, 10^{-5})$ -DP guarantee (see Appendix D), our method indicates more visual improvements for smaller  $\epsilon$  with more diversity and less artifacts, as shown in Figures 1 and 2. The quantitative comparison is summarized in Table 1. Although  $\mathrm{DPDM}^8$  is the only related work that is on a par with our method (both variants) when  $\epsilon = 1$ , DP-kernel significantly outperforms other baselines when  $\epsilon = 0.2$ . Specifically, our method improves the SoTA FID from 61.9 to 26.5 on MNIST, and from 78.4 to 46.2 on FMNIST, while SoTA classification accuracy is generally improved for more than  $20\%$  on MNIST and more than  $10\%$  on FMNIST.
The  $\mathrm{FD}_{\mathrm{DINOv2}}$  of our conditional variant on MNIST and FMNIST are 454.8 and 608.9 when  $\epsilon = 1$ , and they are 410.3 and 589.2 when  $\epsilon = 0.2$ .
DPDM requires 8 GPUs to train, whereas our method only requires 1 GPU.
Figure 1: Qualitative comparison under  $(1,10^{-5})$ -DP on MNIST and Fashion MNIST
Figure 2: Qualitative comparison under  $(0.2,10^{-5})$ -DP on MNIST and Fashion MNIST
4.4.3 RGB image sets: CelebA
To test the versatility and scalability of our method, we also evaluate our method on a more complex colorful image dataset, i.e. CelebA. Figure 3 shows that our method generates more diverse face images with identifiable gender attributes compared to baselines, which is quantitatively verified by significant improvement in both FID and Acc in Table 1. Note that DPDM is unconditional on CelebA. Notably, for  $\epsilon = 0.2$ , our method improves the SoTA FID from 264.8 to 85.9, and improves SoTA Acc from 52.7 to 86.5.
The  $\mathrm{FD}_{\mathrm{DINOv2}}$  of our conditional variant on CelebA are 1028.2 when  $\epsilon = 1$  and 998.9 when  $\epsilon = 0.2$ .
Figure 3: The qualitative comparison on CelebA.  $\delta = 10^{-5}$ . DPDM is unconditional.
Table 1: Quantitative comparison on MNIST, Fashion MNIST (FMNIST) and CelebA. Acc denotes classification accuracy, which is shown in%. ↑ and ↓ refer to higher is better or lower is better, respectively. We use boldface for the best performance. Results of DP-CGAN, GS-WGAN, DP-Sinkhorn are cited from Cao et al. [4] and Long et al. [21]. Results of PEARL and DPDM are cited from Dockhorn et al. [6]. Results of G-PATE and DataLens are cited from their papers, respectively.  $(^{*})$ : DPDM did unconditional generation on CelebA.
Method	ε	MNIST	FMNIST	CelebA
FID ↓	Acc ↑	FID ↓	Acc ↑	FID ↓	Acc ↑
DP-MERF	1	118.3	80.5	104.2	73.1	219.4	57.6
GS-WGAN	1	489.8	14.3	587.3	16.6	437.3	62.9
DP-HP	1	-	74.0	-	67.0	-	-
PEARL	1	121.0	78.2	109.0	68.3	-	-
G-PATE	1	153.4	58.8	214.8	58.1	293.2	70.2
DataLens	1	186.1	71.2	195.0	64.8	297.7	70.6
DPDM	1	23.4	93.4	37.8	73.6	*71.0	-
Ours (conditional)	1	29.5	93.4±0.5	49.5	78.8±0.4	81.8	86.2±0.9
Ours (parallel)	1	21.8	95.5±0.6	48.4	80.0±0.5	83.8	87.0±0.8
DP-MERF	0.2	119.3	75.2	151.3	67.4	264.8	52.7
PEARL	0.2	133.0	77.6	160.0	68.0	-	-
G-PATE	0.2	-	22.0	-	18.0	-	-
DataLens	0.2	-	23.4	-	22.3	-	-
DPDM	0.2	61.9	71.9	78.4	57.0	-	-
Ours (conditional)	0.2	26.5	91.3±0.8	46.2	78.4±0.7	85.9	84.3±1.1
Ours (parallel)	0.2	37.3	93.7±0.7	65.4	79.2±0.5	95.8	86.5±1.2
We generalize RDP for vectors to functional mechanisms and develop all building blocks, e.g. (sub-sampled) Gaussian mechanisms, composition theorems, and post-processing theorems, to facilitate its adoption in deep learning frameworks. We show that those main properties or results of v-RDP also hold for f-RDP. Equipped with f-RDP, we propose a novel approach for training a DPGM, by making the loss function in the RKHS private without truncating the RKHS feature map. Experimental results across different datasets and privacy costs indicate that our method (equipped with f-RDP and retaining the full discriminative capability of the kernel) consistently outperforms other kernel-based methods (with v-RDP) as well as non-kernel-based methods by a large margin. We expect our work to bridge the gap between RDP and functional mechanisms and enrich the family of DPGM.
We thank the reviewers and the area chair for thoughtful comments that have improved our final draft. YY gratefully acknowledges NSERC and CIFAR for funding support. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
[1] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. “Deep Learning with Differential Privacy”. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. 2016, pp. 308–318.
[2] E. Bagdasaryan, O. Poursaeed, and V. Shmatikov. "Differential privacy has disparate impact on model accuracy". In: Advances in neural information processing systems. 2019.
[3] M. Balog, I. Tolstikhin, and B. Schölkopf. "Differentially Private Database Release via Kernel Mean Embeddings". In: Proceedings of the 36th International Conference on Machine Learning. 2018.
[4] T. Cao, A. Bie, A. Vahdat, S. Fidler, and K. Kreis. "Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence". In: Proceedings of the 34th Advances in Neural Information Processing Systems. 2021.
[5] D. Chen, T. Orekondy, and M. Fritz. "GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators". In: Proceedings of the 33rd Advances in Neural Information Processing Systems. 2020, pp. 12673-12684.
[6] T. Dockhorn, T. Cao, A. Vahdat, and K. Kreis. "Differentially private diffusion models". arXiv:2210.09929. 2022.
[7] J. Dong, A. Roth, and W. J. Su. "Gaussian differential privacy". arXiv:1905.02383. 2019.
[8] C. Dwork. "Differential privacy". In: Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14, 2006, Proceedings, Part II 33. 2006, pp. 1-12.
[9] C. Dwork and A. Roth. “The algorithmic foundations of differential privacy”. Foundations and Trends in Theoretical Computer Science, vol. 9, no. 3-4 (2014), pp. 211–407.
[10] M. S. Esipova, A. A. Ghomi, Y. Luo, and J. C. Cresswell. "Disparate Impact in Differential Privacy from Gradient Misalignment". In: The Eleventh International Conference on Learning Representations. 2023.
[11] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Scholkopf, and A. Smola. “A kernel two-sample test”. The Journal of Machine Learning Research, vol. 13, no. 1 (2012), pp. 723–773.
[12] R. Hall, A. Rinaldo, and L. Wasserman. "Differential privacy for functions and functional data". The Journal of Machine Learning Research, vol. 14, no. 1 (2013), pp. 703-727.
[13] F. Harder, K. Adamczewski, and M. Park. "DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-preserving Data Generation". In: International Conference on Artificial Intelligence and Statistics. 2021, pp. 1819-1827.
[14] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. "GANs trained by a two time-scale update rule converge to a local Nash equilibrium". In: Proceedings of 30th Advances in Neural Information Processing Systems. Vol. 30. 2017.
[15] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based learning applied to document recognition". Proceedings of the IEEE, vol. 86, no. 11 (1998), pp. 2278-2324.
[34] H. Xiao, K. Rasul, and R. Vollgraf. "Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms". arXiv:1708.07747. 2017.
[35] D. Xu, W. Du, and X. Wu. “Removing disparate impact on model accuracy in differentially private stochastic gradient descent”. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, pp. 1924–1932.
Proof. Without loss of generality, given two neighboring datasets  $D$  and  $D'$ , assume that  $D$  contains one more element than  $D'$ . Executing  $P$  on  $D$  and  $D'$ , we have partitions  $D_1, D_2, \ldots, D_m$  and  $D'_1, D'_2, \ldots, D'_m$ , respectively. There exists  $j$  such that (1)  $D_j$  contains one more element than  $D'_j$ , and (2)  $D_s = D'_s$  for  $s = 1, 2, \ldots, m$  and  $s \neq j$ . Denote  $\mathcal{M}_1(D_1), \ldots, \mathcal{M}_m(D_m)$  by  $\mathcal{M}(D)$ . Using additivity of Rényi divergence in [30] (Thm 28):
$$
\begin{array}{l} \mathbb {D} _ {\alpha} (\mathcal {M} (D) | | \mathcal {M} (D ^ {\prime})) = \sum_ {i = 1} ^ {m} \mathbb {D} _ {\alpha} (\mathcal {M} _ {i} (D _ {i}) | | \mathcal {M} _ {i} (D _ {i} ^ {\prime})) \\ = \sum_ {i = 1, \dots , m, i \neq j} \mathbb {D} _ {\alpha} \left(\mathcal {M} _ {i} \left(D _ {i}\right) | | \mathcal {M} _ {i} \left(D _ {i} ^ {\prime}\right)\right) + \mathbb {D} _ {\alpha} \left(\mathcal {M} _ {j} \left(D _ {j}\right) | | \mathcal {M} _ {j} \left(D _ {j} ^ {\prime}\right)\right) \\ \leq \epsilon_ {j} \leq \max _ {i = 1, 2, \ldots , m} \epsilon_ {i}. \\ \end{array}
$$
Proof. Given any finite subsets  $S$ , we reach the reduction of proof of the post-processing theorem in [23]:
$$
\mathbb {D} _ {\alpha} \left(f _ {D} (S) | | f _ {D ^ {\prime}} (S)\right) \geq \mathbb {D} \left(g \left(f _ {D} (S)\right) | | g \left(f _ {D ^ {\prime}} (S)\right)\right).
$$
Lemma 1. Definition 4 implies:
$$
\mathbb {P} \left(\widetilde {f _ {D}} \in A\right) \leq \left(\exp (\epsilon) \mathbb {P} \left(\widetilde {f _ {D ^ {\prime}}} \in A\right)\right) ^ {\frac {\alpha - 1}{\alpha}}, \quad \forall A \in \mathcal {F} _ {0}, \tag {9}
$$
where we reuse all the notations in Definition 3.
Proof. Proof sketch: we will show that Eq. (9) holds whenever Eq. (4) holds.
By taking logarithm and rearranging from Eq. (9), we have
$$
\frac {1}{\alpha - 1} \log \left(\left(\frac {\mathbb {P} \left(\widetilde {f _ {D}} \in A\right)}{\mathbb {P} \left(\widetilde {f _ {D ^ {\prime}}} \in A\right)}\right) ^ {\alpha} \mathbb {P} \left(\widetilde {f _ {D ^ {\prime}}} \in A\right)\right) \leq \epsilon . \tag {10}
$$
By the definition of cylinder sets,  $\widetilde{f_D} \in A$  implies that  $\widetilde{f_D}(S)$  is in some sets  $B$  for any finite subsets  $S = (\mathbf{x}_1, \ldots, \mathbf{x}_n)$  of  $T$ . Thus,
$$
\mathbb {P} (\widetilde {f _ {D}} \in A) = \mathbb {P} (\widetilde {f _ {D}} (S) \in B) = \int_ {S} p (x) \mathrm {d} \mu (x).
$$
where  $p$  is the density of  $\widetilde{f_D} (S)$ . Now we can translate Eq. (10) into
$$
\frac {1}{\alpha - 1} \log \left(\left(\frac {\int_ {S} p (x) \mathrm {d} \mu (x)}{\int_ {S} q (x) \mathrm {d} \mu (x)}\right) ^ {\alpha} \int_ {S} q (x) \mathrm {d} \mu (x)\right) \leq \epsilon . \tag {11}
$$
Recall that Eq. (4) in Definition 4 can be translated to:
$$
\frac {1}{\alpha - 1} \log \left(\int (\frac {p (x)}{q (x)}) ^ {\alpha} q (x) \mathrm {d} \mu (x)\right) \leq \epsilon .
$$
Note that  $p, q$  are non-negative, so  $\int \left(\frac{p(x)}{q(x)}\right)^{\alpha} q(x) \mathrm{d}\mu(x) \geq \int_{S} \left(\frac{p(x)}{q(x)}\right)^{\alpha} q(x) \mathrm{d}\mu(x)$ . Compared to Eq. (11), it now suffices to show
$$
\int_ {S} \left(\frac {p (x)}{q (x)}\right) ^ {\alpha} q (x) \mathrm {d} \mu (x) \geq \left(\frac {\int_ {S} p (x) \mathrm {d} \mu (x)}{\int_ {S} q (x) \mathrm {d} \mu (x)}\right) ^ {\alpha} \int_ {S} q (x) \mathrm {d} \mu (x).
$$
Define  $p^*(x) = \frac{p(x)\mathbb{1}(x \in S)}{\int_S p(x) \, \mathrm{d}\mu(x)}$ , and  $q^*$  is similarly defined. All we want to show reduces to
$$
\int \left(\frac {p ^ {*} (x)}{q ^ {*} (x)}\right) ^ {\alpha} q ^ {*} (x) \mathrm {d} \mu (x) \geq 1 \iff \mathbb {E} _ {x \sim q ^ {*}} [ \left(\frac {p ^ {*}}{q ^ {*}}\right) ^ {\alpha} ] \geq \left(\mathbb {E} _ {x \sim q ^ {*}} [ \frac {p ^ {*}}{q ^ {*}} ]\right) ^ {\alpha} = 1,
$$
where the final step follows from Jensen's inequality.
Proof. We reuse the notations in Definition 3 in this proof. To show that an  $(\alpha, \epsilon)$ -RDP function satisfies  $(\epsilon', \delta)$ -DP for functions, where  $\epsilon' = \epsilon + \frac{\log 1 / \delta}{\alpha - 1}$ , the objective becomes to show  $\mathbb{P}[\widetilde{f_D} \in A] \leq \exp(\epsilon') \times \mathbb{P}[\widetilde{f_{D'}} \in A] + \delta$ . With the help of Lemma 1, denote  $\mathbb{P}[\widetilde{f_{D'}} \in A]$  by  $Q$  and we perform a case analysis:
- If  $\exp (\epsilon)Q\leq \delta^{\alpha /(\alpha -1)}$  , then
$$
\mathbb {P} (\widetilde {f _ {D}} \in A) \leq \big (\exp (\epsilon) Q \big) ^ {\frac {\alpha - 1}{\alpha}} \leq \delta \leq \exp (\epsilon^ {\prime}) Q + \delta .
$$
- If  $\exp (\epsilon)Q > \delta^{\alpha /(\alpha -1)}$ , then
$$
\begin{array}{l} \mathbb {P} (\widetilde {f _ {D}} \in A) \leq \left(\exp (\epsilon) Q\right) ^ {\frac {\alpha - 1}{\alpha}} \\ = \exp (\epsilon) Q \left(\exp (\epsilon) Q\right) ^ {- \frac {1}{\alpha}} \\ \leq \exp (\epsilon) Q \cdot \delta^ {- \frac {1}{\alpha - 1}} \\ = \exp (\epsilon + \frac {\log 1 / \delta}{\alpha - 1}) Q \\ \leq \exp (\epsilon^ {\prime}) Q + \delta . \\ \end{array}
$$