SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Antti Koskela
antti.h.koskela@nokia-bell-labs.com
Tejas Kulkarni
tejas.kulkarni@nokia-bell-labs.com
Tuning the hyperparameters of differentially private (DP) machine learning (ML) algorithms often requires use of sensitive data and this may leak private information via hyperparameter values. Recently, Papernot and Steinke (2022) proposed a certain class of DP hyperparameter tuning algorithms, where the number of random search samples is randomized. Commonly, these algorithms still considerably increase the DP privacy parameter  $\varepsilon$  over non-tuned DP ML model training and can be computationally heavy as evaluating each hyperparameter candidate requires a new training run. We focus on lowering both the DP bounds and the compute cost of these methods by using only a random subset of the sensitive data for the hyperparameter tuning and by extrapolating the optimal values to a larger dataset. We provide a Rényi differential privacy analysis for the proposed method and experimentally show that it consistently leads to better privacy-utility trade-off than the baseline method by Papernot and Steinke.
Our aim is two-fold: to decrease the computational cost as well as the privacy cost of hyperparameter tuning of DP ML models. The reasons for this are clear. As the dataset sizes grow and models get more complex, blackbox optimization of hyperparameters becomes more expensive since evaluation of a single set of hyperparameters often requires retraining a new model. On the other hand, tuning the hyperparameters often depends on the use of sensitive data, so it requires privacy protection as well, as illustrated by the example by Papernot and Steinke (2022). Intuitively, the leakage from hyperparameters is much smaller than from the model parameters, however, providing tuning algorithms with low additional DP cost has turned out challenging. Current best algorithms (Papernot and Steinke, 2022) still come with a considerable DP cost overhead.
Although our methods and results are applicable to general DP mechanisms, we focus in particular on tuning of the DP stochastic gradient descent (DP-SGD) (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) which has become the most widely used method to train ML models with DP guarantees. Compared to plain SGD, DP brings additional hyperparameters to tune: the noise level  $\sigma$  and the clipping constant  $C$ . Additionally, also the subsampling ratio  $\gamma$  affects the DP guarantees, as well as length of the training. Tuning all the hyperparameters of DP-SGD commonly requires use of sensitive data.
We use the results by Papernot and Steinke (2022) as building blocks of our methods. Their work was based on the analysis of Liu and Talwar (2019) who provided the first results for DP black-box optimization of hyperparameters, where, if the base training algorithm is  $(\varepsilon, 0)$ -DP, then the tuned model is approximately  $(3\varepsilon, 0)$ -DP. Papernot and Steinke (2022) provided a Rényi differential privacy (RDP) analysis for a class of black-box tuning algorithms, where the number of runs in the hyperparameter tuning is randomized. As the privacy bounds are in terms of RDP and assume only RDP bounds about the candidate model training algorithms, they are particularly suitable to tuning
DP-SGD. However, still, running these algorithms increases the  $\varepsilon$ -values two or three-fold or more, and they can be computationally heavy as evaluating each candidate model requires training a new model. Our novelty is to consider using only a random subset of the sensitive data for the tuning part and use the output hyperparameter values (and potentially the model) for training subsequent models. Using a random subset for the privacy and computation costly part automatically leads to both lower DP privacy leakage as well as computational cost. We also consider ways to appropriately extrapolate the optimal value from the small subset of data to a larger dataset.
The RDP bounds for the DP tuning methods by Papernot and Steinke (2022) assume that the RDP-values of the candidate model training algorithms are fixed. We also consider ways to use these bounds for tuning hyperparameters that affect the RDP-values of the base algorithm, being the noise level  $\sigma$ , the subsampling ratio  $\gamma$  and the length of training in case of DP-SGD.
1.1 Related Work on Hyperparameter Tuning
Chaudhuri and Vinterbo (2013) were the first ones to focus on DP bounds for hyperparameter tuning. An improvement was made by Liu and Talwar (2019) who considered black-box tuning of  $(\varepsilon, \delta)$ -DP mechanisms. Mohapatra et al. (2022) showed that for reasonable numbers of adaptively chosen private candidates a naive RDP accounting (i.e., RDP parameters grow linearly w.r.t. the number of model evaluations) often leads to lower DP bounds than the methods by Liu and Talwar (2019). Papernot and Steinke (2022) gave RDP bounds for black-box tuning algorithms that grow only logarithmically w.r.t. the number of model evaluations. In a non-DP setting, hyperparameter tuning with random subsamples has been considered for SVMs (Horváth et al., 2017) and for large datasets in healthcare (Waring et al., 2020). Small random subsets of data have been used in Bayesian optimization of hyperparameters (Swersky et al., 2013; Klein et al., 2017). Recent works (Killamsetty et al., 2021, 2022) consider using subsets of data for hyperparameter tuning of deep learning models.
- We propose a subsampling strategy to lower the privacy cost and computational cost of DP hyperparameter tuning. We provide a tailored RDP analysis for the proposed strategy. Our analysis is in terms of RDP and we use existing results for tuning Papernot and Steinke (2022) and DP-SGD (Zhu and Wang, 2019) as building blocks.
- We propose algorithms to tune hyperparameters that affect the RDP guarantees of the base model training algorithms. We provide a rigorous RDP analysis for these algorithms.
- We carry out experiments on several standard datasets, where we are able to improve upon the baseline tuning method by a clear margin. While our experiments focus mainly on training of deep learning models with DP-SGD and DP-Adam, our framework is currently applicable to any computation that involves selecting the best among several alternatives (consider e.g., DP model selection, Thakurta and Smith, 2013).
2 Background: DP, DP-SGD and DP Hyperparameter Tuning
We first give the basic definitions. An input dataset containing  $n$  data points is denoted as  $X = \{x_{1},\ldots ,x_{n}\}$ . Denote the set of all possible datasets by  $\mathcal{X}$ . We say  $X$  and  $Y$  are neighbors if we get one by adding or removing one data element to or from the other (denoted  $X\sim Y$ ). Consider a randomized mechanism  $\mathcal{M}:\mathcal{X}\to \mathcal{O}$ , where  $\mathcal{O}$  denotes the output space. The  $(\varepsilon, \delta)$ -definition of DP can be given as follows (Dwork, 2006).
Definition 1. Let  $\varepsilon >0$  and  $\delta \in [0,1]$ . We say that a mechanism  $\mathcal{M}$  is  $(\varepsilon ,\delta)$ -DP, if for all neighboring datasets  $X$  and  $Y$  and for every measurable set  $E\subset \mathcal{O}$  we have:
$$
\Pr (\mathcal {M} (X) \in E) \leq \mathrm {e} ^ {\varepsilon} \Pr (\mathcal {M} (Y) \in E) + \delta .
$$
We will also use the Rényi differential privacy (RDP) (Mironov, 2017) which is defined as follows. Rényi divergence of order  $\alpha > 1$  between two distributions  $P$  and  $Q$  is defined as
$$
D _ {\alpha} (P | | Q) = \frac {1}{\alpha - 1} \log \int \left(\frac {P (t)}{Q (t)}\right) ^ {\alpha} Q (t) d t. \tag {2.1}
$$
Definition 2. We say that a mechanism  $\mathcal{M}$  is  $(\alpha, \varepsilon)$ -RDP, if for all neighboring datasets  $X$  and  $Y$ , the output distributions of  $\mathcal{M}(X)$  and  $\mathcal{M}(Y)$  have Rényi divergence of order  $\alpha$  at most  $\varepsilon$ , i.e.,
$$
\max _ {X \sim Y} D _ {\alpha} \big (\mathcal {M} (X) | | \mathcal {M} (Y) \big) \leq \varepsilon .
$$
We can convert from Rényi DP to approximate DP using, for example, the following formula:
Lemma 3 (Canonne et al. 2020). Suppose the mechanism  $\mathcal{M}$  is  $(\alpha, \varepsilon')$ -RDP. Then  $\mathcal{M}$  is also  $(\varepsilon, \delta(\varepsilon))$ -DP for arbitrary  $\varepsilon \geq 0$  with
$$
\delta (\varepsilon) = \frac {\exp \left((\alpha - 1) \left(\varepsilon^ {\prime} - \varepsilon\right)\right)}{\alpha} \left(1 - \frac {1}{\alpha}\right) ^ {\alpha - 1}. \tag {2.2}
$$
As is common in practice, we carry out the RDP accounting such that we do bookkeeping of total  $\varepsilon (\alpha)$ -values for a list of RDP-orders (e.g. integer  $\alpha$ 's) and in the end convert to  $(\varepsilon ,\delta)$ -guarantees by minimizing over the values given by Equation(2.2) w.r.t.  $\alpha$ . RDP accounting for compositions of DP mechanisms is carried using standard RDP composition results (Mironov, 2017).
DP-SGD differs from SGD such that sample-wise gradients of a random mini-batch are clipped to have an  $L_{2}$ -norm at most  $C$  and normally distributed noise with variance  $\sigma^2$  is added to the sum of the gradients of the mini-batch (Abadi et al., 2016). One iteration is given by
$$
\theta_ {j + 1} = \theta_ {j} - \eta_ {j} \left(\frac {1}{| B |} \sum_ {x \in B _ {j}} \operatorname {c l i p} \left(\nabla f (x, \theta_ {j}), C\right) + Z _ {j}\right), \tag {2.3}
$$
where the noise  $Z_{j} \sim \mathcal{N}(0, \frac{C^{2}\sigma^{2}}{|B|^{2}} I_{d})$ ,  $f$  denotes the loss function,  $\theta$  the model parameters,  $\eta_{j}$  the learning rate hyperparameter at iteration  $j$  and  $|B|$  is the expected batch size (if we carry out Poisson subsampling of mini-batches,  $|B_{j}|$  varies). There are several results that enable the RDP analysis of DP-SGD iterations (Abadi et al., 2016; Balle et al., 2018; Zhu and Wang, 2019). The following result by Zhu and Wang (2019) is directly applicable to analyzing DP-SGD, however, we also use it for analyzing a variant of our hyperparameter tuning method.
Theorem 4 (Zhu and Wang 2019). Suppose  $\mathcal{M}$  is a  $(\alpha, \varepsilon(\alpha))$ -RDP mechanism, w.r.t. to the add/remove neighbourhood relation. Consider the subsampled mechanism ( $\mathcal{M} \circ \mathrm{subsample}_{\mathrm{Poisson}(\gamma)}(X)$ ), where  $\mathrm{subsample}_{\mathrm{Poisson}(\gamma)}$  denotes Poisson subsampling with sampling ratio  $\gamma$ . If  $\mathcal{M}$  is  $(\alpha, \varepsilon(\alpha))$ -RDP then  $\mathcal{M} \circ \mathrm{subsample}_{\mathrm{Poisson}(\gamma)}$  is  $(\alpha, \varepsilon'(\alpha))$ -RDP ( $\alpha \geq 2$  is an integer), where
$$
\begin{array}{l} \varepsilon^ {\prime} (\alpha) = \frac {1}{\alpha - 1} \log \left((1 - \gamma) ^ {\alpha - 1} (\alpha \gamma - \gamma + 1) + \binom {\alpha} {2} \gamma^ {2} (1 - \gamma) ^ {\alpha - 2} \exp (\varepsilon (2)) \right. \\ + 3 \sum_ {j = 3} ^ {\alpha} \binom {\alpha} {j} \gamma^ {j} (1 - \gamma) ^ {\alpha - j} \exp ((j - 1) \varepsilon (j)) \Bigg). \\ \end{array}
$$
We remark that the recent works (Koskela et al., 2020; Gopi et al., 2021; Zhu et al., 2022) give methods to carry out  $(\varepsilon, \delta)$ -analysis of DP-SGD tightly. As the state-of-the-art bounds for hyperparameter tuning methods are RDP bounds (Papernot and Steinke, 2022), for simplicity, we will also analyze DP-SGD using RDP.
Intuitively, the leakage from hyperparameters is much smaller than from the model parameters, however, considering it in the final accounting is needed to ensure rigorous DP guarantees. Currently the most practical  $(\varepsilon, \delta)$ -guarantees for DP hyperparameter tuning algorithms are those of (Papernot and Steinke, 2022). In the results of Papernot and Steinke (2022) it is important that the number of runs  $K$  with the hyperparameter tuning is randomized. They analyze various distributions for drawing  $K$ , however, we focus on using the Poisson distribution as it is the most concentrated around the mean among all the alternatives. The corresponding hyperparameter tuning algorithm and its privacy guarantees are given by Thm.5.
First recall:  $K$  is distributed according to a Poisson distribution with mean  $\mu > 0$ , if for all non-negative integer values  $k$ :  $\mathbb{P}(K = k) = \mathrm{e}^{-\mu} \cdot \frac{\mu^k}{k!}$ .
Theorem 5 (Papernot and Steinke 2022). Let  $Q: \mathcal{X}^N \to \mathcal{Y}$  be a randomized algorithm satisfying  $(\alpha, \varepsilon(\alpha))$ -RDP and  $(\widehat{\varepsilon}, \widehat{\delta})$ -DP for some  $\alpha \in (1, \infty)$  and  $\varepsilon, \widehat{\varepsilon}, \widehat{\delta} \geq 0$ . Assume  $\mathcal{Y}$  is totally ordered. Let the Poisson distribution parameter  $\mu > 0$ . Define the hyperparameter tuning algorithm  $A: \mathcal{X}^N \to \mathcal{Y}$  as follows. Draw  $K$  from a Poisson distribution with mean  $\mu$ . Run  $Q(X)$  for  $K$  times. Then  $A(X)$  returns the best value of those  $K$  runs (both the hyperparameters and the model parameters). If  $K = 0$ ,  $A(X)$  returns some arbitrary output. If  $\mathrm{e}^{\widehat{\varepsilon}} \leq 1 + \frac{1}{\alpha - 1}$ , then  $A$  satisfies  $(\alpha, \varepsilon'(\alpha))$ -RDP, where  $\varepsilon'(\alpha) = \varepsilon(\alpha) + \mu \cdot \widehat{\delta} + \frac{\log \mu}{\alpha - 1}$ .
3 DP Hyperparameter Tuning with a Random Subset
We next consider our main tool: we carry out the private hyperparameter tuning on a random subset, and if needed, extrapolate the found hyperparameter values to larger datasets that we use for training subsequent models. In our approach the subset of data used for tuning is generally smaller than the data used for training the final model and thus we extrapolate the hyperparameter values.
3.1 Our Method: Small Random Subset for Tuning
Our method works as below:
1. Use Poisson subsampling to draw  $X_{1} \subset X$ : draw a random subset  $X_{1}$  such that each  $x \in X$  is included in  $X_{1}$  with probability  $q$ .
2. Compute  $(\theta_{1},t_{1}) = \mathcal{M}_{\mathrm{tune}}(X_{1})$  , where  $\mathcal{M}_{\mathrm{tune}}$  is a hyperparameter tuning algorithm (e.g., the method by Papernot and Steinke, 2022) that outputs the vector of optimal hyperparameters  $t_1$  and the corresponding model parameters  $\theta_{1}$
3. If needed, extrapolate the hyperparameters  $t_1$  to the dataset  $X \setminus X_1$ :  $t_1 \to t_2$ .
4. Compute  $\theta_{2} = \mathcal{M}_{\mathrm{base}}(\theta_{1},t_{2},X\backslash X_{1})$  , where  $\mathcal{M}_{\mathrm{base}}$  is the base mechanism (e.g., DP-SGD).
Denote the whole mechanism by  $\mathcal{M}$ . Then, we may write
$$
\mathcal {M} (X) = \left(\mathcal {M} _ {\text {t u n e}} \left(X _ {1}\right), \mathcal {M} _ {\text {b a s e}} \left(\mathcal {M} _ {\text {t u n e}} \left(X _ {1}\right), X \setminus X _ {1}\right)\right), \tag {3.1}
$$
where  $X_{1}\sim \mathrm{subsample}_{\mathrm{Poisson}(q)}(X)$ . Additionally, we consider a variation of our method in which we use the full dataset  $X$  instead of  $X\setminus X_{1}$  from step 3 onwards, i.e., the mechanism
$$
\mathcal {M} (X) = \left(\mathcal {M} _ {\text {t u n e}} \left(X _ {1}\right), \mathcal {M} _ {\text {b a s e}} \left(\mathcal {M} _ {\text {t u n e}} \left(X _ {1}\right), X\right)\right), \tag {3.2}
$$
where  $X_{1}\sim \mathrm{subsample}_{\mathrm{Poisson}(q)}(X)$ . We call these methods variant 1 and variant 2, respectively. The RDP bounds for the variant 2 can be obtained with a standard subsampling and composition result (e.g., Thm 4). We provide a tailored privacy analysis of the variant 1 in Section 3.3.
3.2 Extrapolating the DP-SGD Hyperparameters
We use simple heuristics to transfer the optimal hyperparameter values found for the small subset of data to a larger dataset. The clipping constant  $C$ , the noise level  $\sigma$ , the subsampling ratio  $\gamma$  and the total number of iterations  $T$  are kept constant in this transfer. As a consequence the  $(\varepsilon, \delta)$ -privacy guarantees are also the same for the models trained with the smaller and the larger dataset. For scaling the learning rate, we use the heuristics used by van der Veen et al. (2018): we scale the learning rate  $\eta$  with the dataset size. I.e., if we carry out the hyperparameter tuning using a subset of size  $m$  and find an optimal value  $\eta^*$ , we multiply  $\eta^*$  by  $n / m$  when transferring to the dataset of size  $n$ .
This can be also heuristically motivated as follows. Consider  $T$  iterations of the DP-SGD (2.3). With the above rules, the distribution of the noise that gets injected into the model trained with dataset of size  $n$  is
$$
\sum_ {j = 1} ^ {T} Z _ {j} \sim \mathcal {N} \left(0, \frac {T \cdot \left(\frac {n}{m} \eta^ {*}\right) ^ {2} \sigma^ {2} C ^ {2}}{(\gamma \cdot n) ^ {2}} I _ {d}\right) \sim \mathcal {N} \left(0, \frac {T \cdot \eta^ {* 2} \sigma^ {2} C ^ {2}}{(\gamma \cdot m) ^ {2}} I _ {d}\right)
$$
which is exactly the distribution of the noise that was added to the model trained with the subsample of size  $m$ . This principle of keeping the noise constant when scaling the hyperparameters was also used by Sander et al. (2022).
We arrive at our scaling rule also by taking a variational Bayesian view of DP-SGD. Mandt et al. (2017) model the stochasticity of the SGD mini-batch gradients in a region approximated by a constant quadratic convex loss by invoking the central limit theorem, and arrive at a continuous-time multivariate Ornstein-Uhlenbeck (OU) process for which the discrete approximation is given by
$$
\Delta \theta = - \eta g (\theta) + \frac {\eta}{\sqrt {| B |}} L \Delta W, \quad \Delta W \sim \mathcal {N} (0, I _ {d}), \tag {3.3}
$$
where  $|B|$  denotes the batch size of the SGD approximation,  $g(\theta)$  the full gradient and  $LL^T$  the covariance matrix of the SGD noise. By minimizing the Kullback-Leibler divergence between the stationary distribution of this OU-process and the Gaussian posterior distribution  $f(\theta) \propto \exp(-n \cdot \mathcal{L}(\theta))$ , where  $\mathcal{L}(\theta)$  denotes the quadratic loss function and  $n$  is the size of the dataset, they arrive at the expression
$$
\eta^ {*} = 2 \frac {| B |}{n} \frac {d}{\operatorname {T r} (L L ^ {T})}
$$
for the optimal learning rate value (see Thm. 1, Mandt et al., 2017). We consider the case where the additive DP noise dominates the SGD noise, and instead of the update (3.3) consider the update
$$
\Delta \theta = - \eta g (\theta) + \frac {\eta \cdot \sigma \cdot C}{| B |} \Delta W, \quad \Delta W \sim \mathcal {N} (0, I _ {d}) \tag {3.4}
$$
which equals the DP-SGD update (2.3) with the mini-batch gradient replaced by the full gradient. Essentially the difference between (3.3) and (3.4) is  $\sqrt{|B|}$  replaced by  $|B|$ , and by the reasoning used in (Thm. 1, Mandt et al., 2017), we see that the learning rate value that minimizes the KL divergence between the approximate posterior and the Gaussian posterior  $f(\theta)$  is then given by
$$
\eta^ {*} = 2 \frac {\left| B \right| ^ {2}}{n} \frac {d}{\operatorname {T r} \left(\sigma^ {2} C ^ {2} I\right)} = 2 \frac {\left| B \right| ^ {2}}{n \cdot \sigma^ {2} C ^ {2}} = 2 \frac {\gamma^ {2} n}{\sigma^ {2} C ^ {2}}. \tag {3.5}
$$
The scaling rule (3.5) also indicates that the optimal value of the learning rate should be scaled linearly with the size of the dataset in case  $\gamma$ ,  $\sigma$  and  $C$  are kept constant.
Training of certain models benefits from use of adaptive optimizers such as Adam (Kingma and Ba, 2014) or RMSProp, e.g., due to sparse gradients. Then the above extrapolation rules for DP-SGD are not necessarily meaningful anymore. In our experiments, when training a neural network classifier using Adam with DP-SGD gradients, we found that keeping the value of the learning rate fixed in the transfer to the larger dataset lead to better results than increasing it as in case of DP-SGD. We mention that there are principled ways of extrapolating the hyperparameters in non-DP setting such as those of Klein et al. (2017).
3.3 Privacy Analysis
The RDP analysis of the variant 2 given in Equation (3.2) is straightforward. Since the tuning set  $X_{1}$  is sampled with Poisson subsampling with subsampling ratio  $q$ , we may write the mechanism as an adaptive composition
$$
\mathcal {M} (X) = \left(\widetilde {\mathcal {M}} _ {\mathrm {t u n e}} (X), \mathcal {M} _ {\mathrm {b a s e}} \big (\widetilde {\mathcal {M}} _ {\mathrm {t u n e}} (X), X\right),
$$
where  $\widetilde{\mathcal{M}}_{\mathrm{tune}}(X) = (\mathcal{M}_{\mathrm{tune}}\circ \mathrm{subsample}_{\mathrm{Poisson}(q)})(X)$ . Using the RDP values given by Thm. 5 for  $\mathcal{M}_{\mathrm{tune}}$  and the subsampling amplification result of Thm. 4, we obtain RDP bounds for  $\widetilde{\mathcal{M}}_{\mathrm{tune}}(X)$ . Using RDP bounds for  $\mathcal{M}_{\mathrm{base}}$  (e.g., DP-SGD) and RDP composition results, we further get RDP bounds for the mechanism  $\mathcal{M}$  given in (3.2).
Tailored RDP-Analysis. When we use the variant (3.1), i.e., we only use the rest of the data  $X \backslash X_1$  for  $\mathcal{M}_{\mathrm{base}}$ , we can get even tighter RDP bounds. The following theorem gives tailored RDP bounds for the mechanism (3.1). Similarly to the analysis by Zhu and Wang (2019) for the Poisson subsampled Gaussian mechanism, we obtain RDP bounds using the RDP bounds of the non-subsampled mechanisms and by using binomial expansions (the proof is given in Appendix C).
Theorem 6. Let  $\mathcal{M}$  be the mechanism (3.1), such that the subset  $X_{1}$  is Poisson sampled with subsampling ratio  $q$ ,  $0 \leq q \leq 1$  and let  $\alpha > 1$ . Denote by  $\varepsilon_{\mathrm{tune}}(\alpha)$  and  $\varepsilon_{\mathrm{base}}(\alpha)$  the RDP-values of mechanisms  $\mathcal{M}_{\mathrm{tune}}$  and  $\mathcal{M}_{\mathrm{base}}$ , respectively. Then,  $\mathcal{M}$  is  $(\alpha, \varepsilon(\alpha))$ -RDP for
$$
\varepsilon (\alpha) = \max  \left\{\varepsilon_ {1} (\alpha), \varepsilon_ {2} (\alpha) \right\},
$$
$$
\begin{array}{l} \varepsilon_ {1} (\alpha) = \frac {1}{\alpha - 1} \log \left(q ^ {\alpha} \cdot \exp \big ((\alpha - 1) \varepsilon_ {\text {t u n e}} (\alpha) \big) + (1 - q) ^ {\alpha} \cdot \exp \big ((\alpha - 1) \varepsilon_ {\text {b a s e}} (\alpha) \big) \right. \\ + \sum_ {j = 1} ^ {\alpha - 1} \binom {\alpha} {j} \cdot q ^ {\alpha - j} \cdot (1 - q) ^ {j} \cdot \exp \left((\alpha - j - 1) \varepsilon_ {\text {t u n e}} (\alpha - j)\right) \exp \left((j - 1) \varepsilon_ {\text {b a s e}} (j)\right) \tag {3.6} \\ \end{array}
$$
$$
\begin{array}{l} \varepsilon_ {2} (\alpha) = \frac {1}{\alpha - 1} \log \left((1 - q) ^ {\alpha - 1} \cdot \exp \big ((\alpha - 1) \varepsilon_ {\mathrm {b a s e}} (\alpha) \big) \right. \\ + \sum_ {j = 1} ^ {\alpha - 1} \binom {\alpha - 1} {j} \cdot q ^ {j} \cdot (1 - q) ^ {\alpha - 1 - j} \cdot \exp \left(j \cdot \varepsilon_ {\text {t u n e}} (j + 1)\right) \cdot \exp \left((\alpha - j - 1) \varepsilon_ {\text {b a s e}} (\alpha - j)\right). \tag {3.7} \\ \end{array}
$$
Remark 7. The RDP bound given by Thm. 6 is optimal in a sense that it approaches  $\varepsilon_{\mathrm{tune}}(\alpha)$  and  $\varepsilon_{\mathrm{base}}(\alpha)$  as  $q \to 1$  and  $q \to 0$ , respectively.
Remark 8. We can initialize the subsequent model training  $\mathcal{M}_{\mathrm{base}}$  using the model  $\theta_{1}$ . This adaptivity is included in all the RDP analyses of both mechanisms (3.1) and (3.2).
Notice that in the bounds (3.6) and (3.7) the RDP parameter of the tuning algorithm,  $\varepsilon_{\mathrm{tune}}(\alpha)$ , is weighted with the parameter  $q$  and the RDP parameter of the base algorithm,  $\varepsilon_{\mathrm{base}}(\alpha)$ , is weighted with  $1 - q$ . This lowers the overall privacy cost in case the tuning set is chosen small enough.