SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Functional Rényi Differential Privacy for Generative Modeling
Dihong Jiang $^{1,2}$ , Sun Sun $^{1,3}$  and Yaoliang Yu $^{1,2}$
School of Computer Science, University of Waterloo
National Research Council Canada
{dihong.jiang,sun.sun, yaoliang.yu}@uwaterloo.ca
Differential privacy (DP) has emerged as a rigorous notion to quantify data privacy. Subsequently, Rényi differential privacy (RDP) has become an alternative to the ordinary DP notion in both theoretical and empirical studies, because of its convenient compositional rules and flexibility. However, most mechanisms with DP (RDP) guarantees are essentially based on randomizing a fixed, finite-dimensional vector output. In this work, following Hall et al. [12] we further extend RDP to functional outputs, where the output space can be infinite-dimensional, and develop all necessary tools, e.g. (subsampled) Gaussian mechanism, composition, and post-processing rules, to facilitate its practical adoption. As an illustration, we apply functional RDP (f-RDP) to functions in the reproducing kernel Hilbert space (RKHS) to develop a differentially private generative model (DPGM), where training can be interpreted as iteratively releasing loss functions (in an RKHS) with DP guarantees. Empirically, the new training paradigm achieves a significant improvement in privacy-utility trade-off compared to existing alternatives, especially when  $\epsilon = 0.2$ . Our code is available at https://github.com/dihjiang/DP-kernel.
Modern machine learning has achieved impressive success thanks to the availability of big data and computing resources. However, there are increasing privacy concerns when training with personal or sensitive data. Differential privacy [DP, 8] has become the de-facto standard technique for releasing statistics of sensitive databases, which is designed to bound the output change of a randomized mechanism  $\mathcal{M}$  given an incremental input deviation, such that  $\mathcal{M}$  does not depend too much on any individual point in the dataset. Recently, Mironov [23] generalizes DP to Rényi differential privacy (RDP) through  $\alpha$ -Rényi divergence [26], which shares many properties with the ordinary DP, yet with tighter and easier composition analysis, thereby attracts more attention in practice.
The popular mechanisms (e.g. Gaussian or Laplace) towards DP or RDP essentially randomize a finite-dimensional vector output with (Gaussian or Laplace) noises. However, if we want to privately release a function, vector-based DP mechanisms are not readily amenable, because a function over a real-valued domain is characterized by infinitely many points [12], which will lead to  $\infty$  bound on the L2/L1-sensitivity if we follow the same sensitivity analysis in Dwork and Roth [9] for the vector output, and will effectively lead to infinitely large Gaussian/Laplacian noise and is impractical. Moreover, it is unclear how to add noise to an infinite-dimensional output. Examples that require a functional DP mechanism include privately releasing the reward function in reinforcement learning [32], the kernel function in kernel density estimation and kernel support vector machine (SVM) [12], and the kernel function in DPGM in this work.
Hall et al. [12] made the most fundamental contribution to extending DP from vectors to functions. Essentially, the functional Gaussian mechanism is achieved by adding a sample path of Gaussian process to a function, in contrast to adding Gaussian noise to a vector. Evaluating the released DP function at arbitrarily many points (which will form a vector) will retain the same DP guarantee. It is worth mentioning that there are no composition theorems and subsampled Gaussian mechanisms developed for functional DP in Hall et al. [12], thus restricting its use in deep learning.
Due to the theoretical convenience and practical flexibility of RDP, in this work, we aim to extend RDP to functions, with all necessary tools to facilitate its adoption in deep learning. Furthermore, we demonstrate its value via a particular application in DPGM where the loss function is in an RKHS. Our contributions can be summarized as:
- Theoretically, we develop the functional RDP, which is equipped with many useful tools including (subsampled) Gaussian mechanism, composition, and post-processing theorems. We will show that functional RDP will share many important properties and results with the vector-based variant.
- Empirically, with functional RDP, we propose a novel DPGM training paradigm by privatizing the loss function in an RKHS, rather than truncating the RKHS to a finite-dimensional space and injecting Gaussian noise therein as in existing works. Our method is evaluated and compared across a wide variety of image datasets and DP guarantees, where our method consistently outperforms other baselines by a large margin. Notably, our method indicates better scalability at more stringent DP guarantees (e.g.,  $\epsilon = 1$  and 0.2), compared to state-of-the-art (SoTA) baselines.
Differential privacy quantifies and restricts the output change of a randomized mechanism given an incremental change in the input dataset, such that the privacy of an individual point is protected. The output of a randomized mechanism can be either a vector (Sections 2.1 to 2.3) or a function (Section 2.4). In this section, we recall a few important related works in differential privacy.
2.1 Differential privacy for vectors (v-DP)
Definition 1  $((\epsilon, \delta)$ -DP for vectors, [8, 9]). A randomized mechanism  $\mathcal{M}: \mathcal{D} \to \mathcal{R}$  with domain  $\mathcal{D}$  and range  $\mathcal{R}$  satisfies  $(\epsilon, \delta)$ -differential privacy if for any two adjacent inputs  $D, D' \in \mathcal{D}$  and for any (measurable) subset of outputs  $\mathcal{S} \subseteq \mathcal{R}$  it holds that
$$
\Pr [ \mathcal {M} (D) \in \mathcal {S} ] \leq \exp (\epsilon) \cdot \Pr [ \mathcal {M} (D ^ {\prime}) \in \mathcal {S} ] + \delta ,
$$
where adjacent inputs (a.k.a. neighbouring datasets)  $D$ ,  $D'$  only differ in one entry. Particularly, when  $\delta = 0$ , we say that  $\mathcal{M}$  is  $\epsilon$ -DP.
2.2 Rényi differential privacy for vectors (v-RDP)
Mironov [23] first formalizes Rényi differential privacy (RDP) which extends ordinary DP by using  $\alpha$ -Rényi divergence [26]. RDP is shown to provide easier composition properties than the ordinary DP notion, and it can be easily converted to  $(\epsilon, \delta)$ -DP.
Definition 2  $((\alpha, \epsilon)$ -RDP for vectors, [23]). A randomised mechanism  $\mathcal{M}$  is  $(\alpha, \epsilon)$ -RDP if for all adjacent inputs  $D, D'$ , Rényi's  $\alpha$ -divergence (of order  $\alpha > 1$ ) between the distributions of  $\mathcal{M}(D)$  and  $\mathcal{M}(D')$  satisfies:
$$
\mathbb {D} _ {\alpha} (\mathcal {M} (D) \| \mathcal {M} (D ^ {\prime})) := \frac {1}{\alpha - 1} \log \mathbb {E} _ {x \sim q} \left(\frac {p (x)}{q (x)}\right) ^ {\alpha} \leq \epsilon ,
$$
where  $p$  and  $q$  are the density of  $\mathcal{M}(D)$  and  $\mathcal{M}(D')$ , respectively.
Conveniently, RDP is linearly composable:
Theorem 1 (Sequential composition of v-RDP, [23]). Let  $f: \mathcal{D} \to \mathcal{R}_1$  be  $(\alpha, \epsilon_1)$ -RDP,  $g: \mathcal{R}_1 \times \mathcal{D} \to \mathcal{R}_2$  be  $(\alpha, \epsilon_2)$ -RDP, then running  $f, g$  sequentially to obtain  $h: \mathcal{D} \to \mathcal{R}_1 \times \mathcal{R}_2$ ,  $h(D) := (f(D), g(f(D), D))$  satisfies  $(\alpha, \epsilon_1 + \epsilon_2)$ -RDP.
Similar to the parallel composition theorem for ordinary  $\epsilon$ -DP as in McSherry [22], we complement the parallel composition for v-RDP:
Theorem 2 (Parallel composition of v-RDP). If mechanism  $\mathcal{M}_i$  satisfies  $(\alpha, \epsilon_i)$ -RDP for  $i = 1, 2, \ldots, m$ , and let  $D_1, D_2, \ldots, D_m$  be the disjoint partitions by executing a deterministic partitioning function  $P$  on  $D$ . Releasing  $\mathcal{M}_1(D_1), \ldots, \mathcal{M}_m(D_m)$  satisfies  $(\alpha, \max_{i \in \{1, 2, \ldots, m\}} \epsilon_i)$ -RDP.
2.3 Gaussian mechanism for v-DP and v-RDP
Among multiple choices, the Gaussian mechanism is more suitable for the  $(\epsilon, \delta)$ -DP notion (where  $\delta > 0$ ) and provides additional flexibility (e.g., the sum of Gaussians is still a Gaussian). It is achieved by adding calibrated spherical Gaussian noise to a vector output.
Proposition 1 (Gaussian mechanism for v-DP and v-RDP, [9, 23]). Given a  $d$ -dimensional function  $f: \mathcal{D} \to \mathbb{R}^d$ . The Gaussian mechanism is given by:
$$
\mathcal {M} (D) = f (D) + \sigma \Delta_ {2} f \cdot \mathcal {N} (0, \mathbb {I} _ {d}),
$$
where  $\Delta_2f = \max_{D\sim D',D,D'\in \mathcal{D}}\| f(D) - f(D')\| _2$ .  $\mathcal{M}(D)$  is said to be: (1)  $(\epsilon ,\delta)$ -DP if  $\sigma \geq \sqrt{2\ln(1.25 / \delta)} /\epsilon$  for  $\epsilon \in (0,1)$ , or (2)  $(\alpha ,\frac{\alpha}{2\sigma^2})$ -RDP.
2.4 Differential privacy for functions (f-DP)
Prior DP works mainly focused on the setting when the output of a query is a vector. However, if the output is a function, such as the kernel density estimation example in Hall et al. [12] and the DPGM example in our work (Section 4.3), a few challenges render the direct application of previous DP results on a functional output<sup>1</sup> intractable: it is unclear (1) how to analyze the sensitivity and (2) how to add Gaussian noise to an infinite-dimensional output. To the best of our knowledge, Hall et al. [12] are the pioneers in addressing those challenges by starting with defining the  $(\epsilon, \delta)$ -DP for functions:
Definition 3  $((\epsilon, \delta)$ -DP for functions, [12]). Consider a class of functions indexed by database  $D$  over  $T = \mathbb{R}^d$ , i.e.  $\{f_D : D \in \mathcal{D}\} \subseteq \mathbb{R}^T$ . Define cylinder sets  $C_{S,B} = \{f \in \mathbb{R}^T : (f(\mathbf{x}_1), \ldots, f(\mathbf{x}_n)) \in B\}$ , for all finite subsets  $S = (\mathbf{x}_1, \ldots, \mathbf{x}_n)$  of  $T$  and Borel sets  $B \in \mathbb{R}^n$ . Then, define  $\mathcal{C}_s = \{C_{S,B} : B \in \mathcal{B}(\mathbb{R}^n)\}$  and  $\mathcal{F}_0 = \bigcup_{S:|S| < \infty} \mathcal{C}_s$ . We say the mechanism  $\widetilde{f_D}$  satisfies  $(\epsilon, \delta)$ -DP over the field of cylinder sets, if for all  $D, D' \in \mathcal{D}$ :
$$
\Pr \left[ \widetilde {f _ {D}} \in A \right] \leq \exp (\epsilon) \times \Pr \left[ \widetilde {f _ {D ^ {\prime}}} \in A \right] + \delta , \quad \forall A \in \mathcal {F} _ {0}. \tag {1}
$$
Hall et al. [12] point out that whenever Eq. (1) holds, for any finite set of points  $\mathbf{x}_1,\ldots ,\mathbf{x}_n$  in  $T$  chosen a-priori, or after the construction of the function, or even adaptively chosen based on the outputs given, releasing the vector  $[\widetilde{f_D} (\mathbf{x}_1),\dots ,\widetilde{f_D} (\mathbf{x}_n)]$  satisfies  $(\epsilon ,\delta)$ -DP.
The Gaussian mechanism for f-DP  $^4$  is reached by injecting a sample path of a calibrated Gaussian process into the function:
Proposition 2 (Gaussian mechanism for f-DP, [12]). Let  $G$  be a sample path of a Gaussian process having mean zero and covariance function  $k$ . Let  $M$  denote the Gram matrix
$$
M \left(\mathbf {x} _ {1}, \dots , \mathbf {x} _ {n}\right) = \left( \begin{array}{c c c} k \left(\mathbf {x} _ {1}, \mathbf {x} _ {1}\right) & \dots & k \left(\mathbf {x} _ {1}, \mathbf {x} _ {n}\right) \\ \vdots & \ddots & \vdots \\ k \left(\mathbf {x} _ {n}, \mathbf {x} _ {1}\right) & \dots & k \left(\mathbf {x} _ {n}, \mathbf {x} _ {n}\right) \end{array} \right). \tag {2}
$$
Let  $\{f_D:D\in \mathcal{D}\}$  be a family of functions indexed by database  $D$ . Releasing  $\widetilde{f_{D}} = f_{D} + \sqrt{2\ln(1.25 / \delta)}\Delta /\epsilon \cdot G$  satisfies  $(\epsilon ,\delta)$ -DP whenever
$$
\sup  _ {D \sim D ^ {\prime}} \sup  _ {n <   \infty} \sup  _ {(\mathbf {x} _ {1}, \dots , \mathbf {x} _ {n}) \in T ^ {n}} \left\| M ^ {- \frac {1}{2}} (\mathbf {x} _ {1}, \dots , \mathbf {x} _ {n}) \left( \begin{array}{c} f _ {D} (\mathbf {x} _ {1}) - f _ {D ^ {\prime}} (\mathbf {x} _ {1}) \\ \vdots \\ f _ {D} (\mathbf {x} _ {n}) - f _ {D ^ {\prime}} (\mathbf {x} _ {n}) \end{array} \right) \right\| _ {2} \leq \Delta . \tag {3}
$$
It means the output is a function.
${}^{2}$  Intuitively,the cylinder set defined here is a set of functions where any finite evaluation of the function is in a "ball".
<sup>3</sup>A Borel set is formed via countable union, intersection and complement of open sets in a topological space.
4Not to confuse with the function DP in Dong et al. [7], where the type-I and type-II errors are controlled by a pre-specified function.
A Gaussian process is a generalization of the multivariate Gaussian distribution to the infinite-dimensional case. Any arbitrarily collected finite random variables from a Gaussian process follow a multivariate Gaussian distribution.
Particularly, Hall et al. [12] studied how to achieve  $(\epsilon, \delta)$ -DP for functions in an RKHS  $^6\mathcal{H}$ :
Corollary 1 (Corollary 9 in [12]). For  $\{f_D : D \in \mathcal{D}\} \subseteq \mathcal{H}$ , releasing  $\widetilde{f_D} = f_D + \sqrt{2\ln(1.25 / \delta)}\Delta /\epsilon \cdot G$  is  $(\epsilon ,\delta)$ - $DP$  (with respect to the cylinder  $\sigma$ -field) whenever  $\sup_{D,D'}\| f_D - f_{D'}\|_{\mathcal{H}} \leq \Delta$  and when  $G$  is a sample path of a Gaussian process with mean zero and covariance function  $k$  that is given by the reproducing kernel of  $\mathcal{H}$ .
3 Rényi differential privacy for functions (f-RDP)
In this section, we aim to extend the definition of RDP to functional outputs, along with its associated calculus rules to facilitate practical adoption. In particular, we will show that the main results for v-RDP all extend to f-RDP.
3.1 Definition
Consider a class of functions over  $T = \mathbb{R}^d$ , i.e.  $\{f_D : D \in \mathcal{D}\} \subseteq \mathbb{R}^T$ . We can define a functional generalization of the  $\alpha$ -Rényi divergence:
Definition 4  $((\alpha, \epsilon)$ -RDP for functions). Denote the evaluation of function  $\widetilde{f_D}$  on any finite subsets  $S = (\mathbf{x}_1, \ldots, \mathbf{x}_n)$  of  $T$  by  $\{\widetilde{f_D}(\mathbf{x}_1), \ldots, \widetilde{f_D}(\mathbf{x}_n)\} := \widetilde{f_D}(S)$ . We say  $\widetilde{f_D}$  is  $(\alpha, \epsilon)$ -RDP, if for all adjacent inputs  $D, D' \in \mathcal{D}$ , Rényi's  $\alpha$ -divergence (of order  $\alpha > 1$ ) between the distributions of  $\widetilde{f_D}(S)$  and  $\widetilde{f_{D'}}(S)$  satisfies:
$$
\mathbb {D} _ {\alpha} \left(\widetilde {f _ {D}} (S) \| \widetilde {f _ {D ^ {\prime}}} (S)\right) := \frac {1}{\alpha - 1} \log \mathbb {E} _ {x \sim q} \left(\frac {p (x)}{q (x)}\right) ^ {\alpha} \leq \epsilon , \tag {4}
$$
where  $p, q$  are the density of  $\widetilde{f_D}(S)$  and  $\widetilde{f_{D'}}(S)$ , respectively.
Remark 1. Definition 4 essentially claims that the distribution of any finite number of evaluations of function  $\widetilde{f_D}$  and  $\widetilde{f_{D'}}$  satisfies Definition 2 ( $v$ -RDP).
We have a weaker definition of f-RDP (see Lemma 1 for the weaker definition) also based on cylinder sets as in Definition 3. We also refer interested readers to Appendix F for an alternative unified definition of RDP for both vectors and functions.
3.2 Post-processing theorem
As any data-independent post-processing preserves the DP guarantee for v-RDP (Definition 2), it also preserves DP guarantee for f-RDP with Remark 1. Specifically,
Theorem 3 (Post-processing theorem of f-RDP). If a function  $f_{D}$  is  $(\alpha, \epsilon)$ -RDP, so is  $g \circ f_{D}$ , where  $g$  is a post-processing mechanism that only depends on a finite number of outputs of  $f_{D}$ .
3.3 Conversion to  $(\epsilon, \delta)$ -DP
The following conversion is a direct extension of its vector version in Mironov [23]:
Proposition 3 (f-RDP conversion to f-DP). A function  $\widetilde{f_D}$  that is  $(\alpha, \epsilon)$ -RDP is  $(\epsilon + \frac{\log 1 / \delta}{\alpha - 1}, \delta)$ -DP.
3.4 Composition theorems
Similar to Theorem 2, we first derive the parallel composition theorem of RDP for functions:
Theorem 4 (Parallel composition of f-RDP). Given a deterministic partitioning function  $P$ , let  $D_{1}, D_{2}, \ldots, D_{m}$  be the disjoint partitions by executing  $P$  on  $D$ . If function  $f_{D_i}$  satisfies  $(\alpha, \epsilon_i)$ -RDP for  $i = 1, 2, \ldots, m$ , releasing  $(f_{D_1}, \ldots, f_{D_m}) := f_D$  satisfies  $(\alpha, \max_{i \in \{1, 2, \ldots, m\}} \epsilon_i)$ -RDP.
Now we move on to the sequential composition theorem of RDP for functions (extension of Theorem 1 to functional mechanisms), which is important and required for composing the total privacy cost when we sample different sample paths from the Gaussian process over training iterations.
<sup>6</sup>A Hilbert space of functions with continuous pointwise evaluation  $f \mapsto f(x)$ . We give more explanations in Section 4.1.
First, we note that Theorem 1 implicitly defines  $g$  as a functional mechanism, i.e.  $g: \mathcal{D} \to \mathcal{R}_2^{\mathcal{R}_1}$ . It means for any  $r_1 \in \mathcal{R}_1$ , we have  $g_{r_1} := g(r_1, \cdot): \mathcal{D} \to \mathcal{R}_2$  is  $(\alpha, \epsilon_2)$ -RDP.
Theorem 5 (Sequential composition of f-RDP). Let  $\{f_D : D \in \mathcal{D}\}$  and  $\{g_D : D \in \mathcal{D}\}$  be two families of functions indexed by dataset  $D$ , where  $f_D \in \mathcal{R}_1^T$  is  $(\alpha, \epsilon_1)$ -RDP and  $g_D : \mathcal{R}_1^T \to \mathcal{R}_2^S$  is  $(\alpha, \epsilon_2)$ -RDP. Releasing the sequentially composed functional mechanism  $h_D = (f_D, g_D \circ f_D) \in \mathcal{R}_1^T \times \mathcal{R}_2^S = (\mathcal{R}_1 \times \mathcal{R}_2)^{T \times S}$  satisfies  $(\alpha, \epsilon_1 + \epsilon_2)$ -RDP.
3.5 Gaussian mechanism
We also need to develop a Gaussian mechanism to retain f-RDP. To make it more convenient, we first rewrite Proposition 1 in a similar form to Proposition 3 in Hall et al. [12] as:
Definition 5 (Gaussian mechanism for v-RDP, with non-isotropic Gaussian). Let  $M \in \mathbb{R}^{d \times d}$  be a positive definite symmetric matrix, the family of vectors  $\{\mathbf{v}_D : D \in \mathcal{D}\} \in \mathbb{R}^d$  satisfies  $\sup_{D \sim D'} \| M^{-\frac{1}{2}}(\mathbf{v}_D - \mathbf{v}_{D'}) \|_2 \leq \Delta$  for all adjacent datasets  $D, D' \in \mathcal{D}$ . The Gaussian mechanism is given by:
$$
\widetilde {\mathbf {v} _ {D}} = \mathbf {v} _ {D} + \sigma \Delta \cdot \mathcal {N} (0, M). \tag {5}
$$
$\widetilde{\mathbf{v}_D}$  in Eq. (5) satisfies  $(\alpha, \frac{\alpha}{2\sigma^2})$ -RDP. Now we define the Gaussian mechanism for f-RDP:
Proposition 4 (Gaussian mechanism for f-RDP). Let  $G$  be a sample path of a Gaussian process having mean zero and covariance function  $k$ . Let  $M$  denote the Gram matrix (as defined in Eq. (2)). Let  $\{f_D : D \in \mathcal{D}\}$  be a family of functions indexed by database  $D$ . Releasing  $\widetilde{f_D} = f_D + \sigma \Delta \cdot G$  satisfies  $(\alpha, \frac{\alpha}{2\sigma^2})$ -RDP whenever Eq. (3) holds.
Particularly, when the function is in an RKHS  $\mathcal{H}$ , we have the following corollary:
Corollary 2. For  $\{f_D:D\in \mathcal{D}\} \subseteq \mathcal{H}$ , releasing  $\widetilde{f_{D}} = f_{D} + \sigma \Delta \cdot G$  is  $(\alpha ,\frac{\alpha}{2\sigma^2})$ -RDP (with respect to the cylinder  $\sigma$ -field) whenever  $\sup_{D,D'}\| f_D - f_{D'}\|_{\mathcal{H}}\leq \Delta$  and when  $G$  is a sample path of a Gaussian process with mean zero and covariance function  $k$  given by the reproducing kernel of  $\mathcal{H}$ .
3.6 Subsampled Gaussian mechanism (SGM)
Subsampling is a crucial component in existing DP deep learning algorithms (e.g., DP-SGD of [1]), which incurs a small privacy cost for every sampled batch by querying  $D$ , thereby requiring to compose the total privacy loss (measured by  $\epsilon$ ) over training iterations. The same thing could happen for functional mechanisms when we subsample a batch  $S$  from the whole dataset  $D$  to index the function  $f_{S}$ .
The current dominant (python-based) DP implementation packages, e.g., Tensorflow privacy and Opacus, apply v-RDP and compute the total  $\epsilon$  in three main steps: (1) compute the v-RDP guarantee for an SGM, based on a numerical procedure in Mironov et al. [24]; (2) sequentially compose v-RDP over training iterations; (3) convert v-RDP to v-DP. In previous sections we have already shown that f-RDP shares the same results with v-RDP on steps (2) and (3), now we will show that an SGM for f-RDP can be reduced to v-RDP in Mironov et al. [24], thus, the numerical procedure in Mironov et al. [24] (and thereby those DP packages) is amenable to f-RDP in terms of the  $\epsilon$  accumulation.
We apply the same subsampling strategy as in Mironov et al. [24]. Let  $S$  be a subsampled set from  $D$ , where each element of  $S$  is independently drawn from  $D$  with probability  $q$ . SGM for f-RDP is given by  $\widetilde{f_S} = f_S + \sigma \Delta \cdot G$ , where  $\Delta$  and  $G$  are defined in Proposition 4. The reduction is immediate: for any finite set of points  $V = \{\mathbf{x}_1, \dots, \mathbf{x}_d\}$  (where  $d < \infty$ ),  $f_S(V)$  will form a  $d$ -dimensional vector. Let  $g_S(V) = M^{-1/2} f_S(V)$ , we have  $\widetilde{g_S}(V) = g_S(V) + \sigma \Delta \cdot \mathcal{N}(0, \mathbb{I}_d)$ , where  $g_S(V)$  is a  $d$ -dimensional vector and  $\widetilde{g_S}(V)$  is the same SGM as in Mironov et al. [24]. Therefore, f-RDP shares the same guarantee of an SGM with v-RDP. Another intuition is that Mironov et al. [24] reduce computing the  $\alpha$ -Renyi divergence of  $d$ -dimensional Gaussians to 1-dimensional Gaussians, which guides all of their subsequent derivations. When  $d \to \infty$ , we can reach the same reduction from infinite-dimensional Gaussian (Gaussian process) to 1-dimensional Gaussian.
4 An application in DPGM
To demonstrate the practical value of f-RDP, here we consider a particular example of training a differentially private generative model (DPGM) through Maximum Mean Discrepancy (MMD).
Assuming a feature map  $\phi : \mathcal{X} \to \mathcal{H}$ , where  $\mathcal{H}$  is an RKHS of some kernel, MMD [11] is a non-parametric distance measure that compares two distributions  $p, q$  by:
$$
\mathcal {L} _ {\mathrm {M M D} ^ {2}} (p, q) = \| \mathbb {E} _ {\mathbf {x} \sim p} [ \phi (\mathbf {x}) ] - \mathbb {E} _ {\mathbf {w} \sim q} [ \phi (\mathbf {w}) ] \| _ {\mathcal {H}} ^ {2},
$$
where  $\mathbb{E}_{\mathbf{x}\sim p}[\phi (\mathbf{x})]\in \mathcal{H}$  is also known as the (kernel) mean embedding (KME) of  $p$
Given a kernel  $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ , such that  $^7 k(\mathbf{x},\mathbf{w}) = \langle \phi (\mathbf{x}),\phi (\mathbf{w})\rangle_{\mathcal{H}}$ , we can play the kernel trick to compute the (squared) MMD in an alternative way:
$$
\mathcal {L} _ {\mathrm {M M D} ^ {2}} (p, q) = \mathbb {E} _ {\mathbf {x}, \mathbf {x} ^ {\prime} \sim p} k (\mathbf {x}, \mathbf {x} ^ {\prime}) - 2 \mathbb {E} _ {\mathbf {x} \sim p, \mathbf {w} \sim q} k (\mathbf {x}, \mathbf {w}) + \mathbb {E} _ {\mathbf {w}, \mathbf {w} ^ {\prime} \sim q} k (\mathbf {w}, \mathbf {w} ^ {\prime}),
$$
which also implicitly lifts the KME into an infinite-dimensional space.
Sriperumbudur et al. [27] suggest that if  $k$  is a characteristic kernel (e.g., Gaussian kernel), then  $\mathrm{MMD} = 0$  iff  $p = q$ , which makes MMD a practical tool in many applications, such as two-sample test [11] and generative modeling [e.g., 16, 17].
4.2 Related works
Balog et al. [3] first proposed a DP database release mechanism via KME, by truncating the infinite-dimensional RKHS to a finite-dimensional feature space through random Fourier features and adding Gaussian noise to the mean of truncated feature embeddings of all data. This idea is further extended to generative modeling by DP-MERF [13]. Specifically, given the samples drawn from two distributions:  $D = \{\mathbf{x}_i\}_{i=1}^N \sim p(\text{true})$  and  $W = \{\mathbf{w}_j\}_{j=1}^M \sim q(\text{generated})$ , empirical MMD with a kernel function  $k$  can be estimated by: