SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: Tuning only the learning rate with DP-Adam. Test accuracies are averaged across 10 independent runs and the error bars denote the standard error of the mean. No learning rate scaling was applied. The number in the legends in the first column refer to the scaled mean training timings for the baseline method with respect to the fastest of variant 1 and 2. The second column plots final  $\varepsilon$  vs. mean  $\sigma$ . Our methods inject significantly smaller noise compared to the baseline for all  $\varepsilon$  regimes. We also observe that due to tight analysis in Thm 6,  $\sigma$  for variant 1 is consistently lower than for variant 2. As a result, we see slightly higher accuracy for variant 1 in many cases. The third column plots final  $\varepsilon$  vs. mean optimal  $\eta$ . Note that due to randomness in the candidate selection process, optimal  $\eta$ 's for all three methods need not be the same. For perspective, we also add curves showing the privacy cost of training a single model with optimal hyperparameters obtained from the baseline.
(c) trained with DP-Adam
(d) trained with DP-Adam
Figure 7: Tuning all hyperparameters  $(\eta, \epsilon)$  (epochs, and  $\gamma$ ) with DP-SGD. Test accuracies are averaged across 10 independent runs and the error bars denote the standard error of the mean. The numbers in the legends in the first column refer to the scaled mean training timings for the baseline method with respect to the fastest of variant 1 and 2. The second column plots final  $\varepsilon$  vs. mean  $\sigma$ . Our methods inject significantly smaller noise compared to the baseline for all  $\varepsilon$  regimes even at high values of  $\mu$ . Since there are several hyperparameters at play together, it is difficult to attribute slightly higher accuracy for variant 2 compared to 1 to any one factor. However, we suspect slight inferiority of our bound for higher values of  $\mu$  (Figure 1, right plot) and higher training dataset for the final model in variant 2 could be the main reasons. The third (and fourth) column plots final  $\varepsilon$  vs. mean optimal  $\eta$  (and  $\gamma$ ). Noise level  $\sigma$  is a proxy for number of epochs, since we obtain a  $\sigma$  for each combination of  $\gamma$  and number of epochs. Therefore, we omit plots showing final  $\epsilon$  vs. epochs plot for readability. Note that due to randomness in the candidate selection process, optimal  $\eta$ 's and  $\gamma$ 's for all three methods need not be the same. For perspective, we also add curves showing the privacy cost of training a single model with optimal hyperparameters obtained from the baseline.
Figure 8: Tuning all hyperparameters  $(\eta, \mathrm{epochs},$  and  $\gamma)$  with DP-Adam. Test accuracies are averaged across 10 independent runs and the error bars denote the standard errors of the means. The number in the legends in the first column refer to the scaled mean training timings for the baseline method with respect to the fastest of variant 1 and 2. The second column plots final  $\varepsilon$  vs. mean  $\sigma$ . Our methods inject significantly smaller noise compared to the baseline for all  $\varepsilon$  regimes even at high values of  $\mu$ . Since there are several hyperparameters at play together, it is difficult to attribute slightly higher accuracy for variant 2 compared to 1 to any one factor. However, we suspect slight inferiority of our bound for higher values of  $\mu$  (Figure 1, right plot) and higher training dataset for the final model in variant 2 could be the main reasons. The third (and fourth) column plots final  $\varepsilon$  vs. mean optimal  $\eta$  (and  $\gamma$ ). Noise level  $\sigma$  is a proxy for number of epochs, since we obtain a  $\sigma$  for each combination of  $\gamma$  and number of epochs. Therefore, we omit plots showing final  $\epsilon$  vs. epochs plot for readability. Note that due to randomness in the candidate selection process, optimal  $\eta$ 's and  $\gamma$ 's for all three methods need not be the same.
(b) trained with DP-Adam
(c) trained with DP-Adam
B Hyperparameter Tables
Table 1: Tuning  $\eta$ : rest of the hyperparameters are fixed to these values. The hyperparameter grids for the learning rates are the same as in the second experiment and are given in Table 2.
MNIST	FashionMNIST	CIFAR-10	IMDB
γ = B/N	0.0213	0.0213	0.0256	0.0256
epochs	40	40	40	110
Table 2: Tuning  $\sigma ,\eta$  and  $T$  : datasets and the corresponding hyperparameter grids.
train/test set	parameters	C	B	learning rate (η)	epochs
MNIST	60k/10k	~26k	1	{128, 256}	{10- i}i∈{4,3.5,3,2.5,2,1.5,1,0.5,0}	{10,20,30,40}
FashionMNIST	60k/10k	~109k	3	{128, 256}	{10- i}i∈{3,2.5,2,1.5,1,0.5,0,-0.5,-1,-1.5}	{10,20,30,40}
CIFAR-10	50k/10k	0.65k	3	{64, 128}	{10- i}i∈{3,2.5,2,1.5,1,0.5,0,-0.5,-1,-1.5}	{20,30,40}
IMDB	25k/25k	~464k	1	{64, 128}	{10- i}i∈{4,3.5,3,2.5,2,1.5,1,0.5,0}	{70,90,110}
C Proof of Thm 6
Proof. For the proof, let us denote for short  $\mathcal{M}_{\mathrm{tune}}=: \mathcal{M}_1$  and  $\mathcal{M}_{\mathrm{base}}=: \mathcal{M}_2$ . Let  $X \in \mathcal{X}^n$  and  $x' \in \mathcal{X}$ . We first consider bounding the Rényi divergence  $D_{\alpha}\big(\mathcal{M}(X \cup \{x'\})||\mathcal{M}(X)\big)$ . Looking at our approach which uses Poisson subsampling with subsampling ratio  $q$  to obtain the dataset  $X_1$ , and conditioning the output on the randomness in choosing  $X_1$ , we can write the mechanism as a mixture over all possible choices of  $X_1$  as
$$
\mathcal {M} (X) = \sum_ {X _ {1}} p _ {X _ {1}} \cdot \left(\mathcal {M} _ {1} \left(X _ {1}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1}\right), X \backslash X _ {1}\right)\right), \tag {C.1}
$$
where  $p_{X_1}$  is the probability of sampling  $X_1$ . Since each data element is in  $X_1$  with probability  $q$ , we can furthermore write  $\mathcal{M}(X \cup \{x'\})$  as a mixture
$$
\begin{array}{l} \mathcal {M} \left(X \cup \left\{x ^ {\prime} \right\}\right) = \sum_ {X _ {1}} p _ {X _ {1}} \cdot \left(q \cdot \left(\mathcal {M} _ {1} \left(X _ {1} \cup \left\{x ^ {\prime} \right\}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1} \cup \left\{x ^ {\prime} \right\}\right), X \backslash X _ {1}\right)\right) \right. \tag {C.2} \\ \left. + (1 - q) \cdot \left(\mathcal {M} _ {1} \left(X _ {1}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1}\right), X \backslash X _ {1} \cup \left\{x ^ {\prime} \right\}\right)\right)\right). \\ \end{array}
$$
From the quasi-convexity of the Rényi divergence (Van Erven and Harremos, 2014) and the expressions (C.1) and (C.2), it follows that
$$
\begin{array}{l} D _ {\alpha} \left(\mathcal {M} \left(X \cup \{x ^ {\prime} \}\right) | | \mathcal {M} (X)\right) \leq \sup  _ {X _ {1}} D _ {\alpha} \left(q \cdot \left(\mathcal {M} _ {1} \left(X _ {1} \cup \{x ^ {\prime} \}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1} \cup \{x ^ {\prime} \}\right), X \backslash X _ {1}\right)\right) \right. \\ + (1 - q) \cdot \left(\mathcal {M} _ {1} \left(X _ {1}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1}\right), X \backslash X _ {1} \cup \{x ^ {\prime} \}\right)\right) \| \left(\mathcal {M} _ {1} \left(X _ {1}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1}\right), X \backslash X _ {1}\right)\right)\left. \right). \\ \end{array}
$$
Our aim is to express the right-hand side of the inequality (C.3) in terms of RDP parameters of  $\mathcal{M}_1$  and  $\mathcal{M}_2$ . To this end, take an arbitrary  $X_1 \subset X$ , and denote by
-  $\widetilde{P}(t)$  the density function of  $\mathcal{M}_1(X_1 \cup \{x'\})$ ,
-  $P(t)$  the density function of  $\mathcal{M}_1(X_1)$ ,
-  $\widetilde{Q}(t,s)$  the density function of  $\mathcal{M}_2(t,X\backslash X_1\cup \{x'\})$  for auxiliary variable  $t$  (the output of  $\mathcal{M}_1$ ),
-  $Q(t,s)$  the density function of  $\mathcal{M}_2(t,X\backslash X_1)$  for auxiliary variable  $t$ .
Then, we see that
$$
\mathbb {P} \big (\left(\mathcal {M} _ {1} (X _ {1}), \mathcal {M} _ {2} (\mathcal {M} _ {1} (X _ {1}), X \backslash X _ {1})\right) = (t, s) \big) = P (t) \cdot Q (t, s)
$$
and similarly that
$$
\begin{array}{l} \mathbb {P} \left(q \cdot \left(\mathcal {M} _ {1} \left(X _ {1} \cup \{x ^ {\prime} \}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1} \cup \{x ^ {\prime} \}\right), X \backslash X _ {1}\right)\right) \right. \\ + (1 - q) \cdot \left(\mathcal {M} _ {1} \left(X _ {1}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1}\right), X \backslash X _ {1} \cup \left\{x ^ {\prime} \right\}\right)\right) = (t, s)\left. \right) \\ = q \cdot \mathbb {P} \left(\left(\mathcal {M} _ {1} \left(X _ {1} \cup \{x ^ {\prime} \}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1} \cup \{x ^ {\prime} \}\right), X \backslash X _ {1}\right)\right) = (t, s)\right) \\ + (1 - q) \cdot \mathbb {P} \left(\left(\mathcal {M} _ {1} \left(X _ {1}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1}\right), X \backslash X _ {1} \cup \left\{x ^ {\prime} \right\}\right)\right) = (t, s)\right) \\ = q \cdot \widetilde {P} (t) \cdot Q (t, s) + (1 - q) \cdot P (t) \cdot \widetilde {Q} (t, s). \\ \end{array}
$$
By the definition of the Rényi divergence, we have that
$$
\begin{array}{l} \exp \left((\alpha - 1) D _ {\alpha} \left(q \cdot \left(\mathcal {M} _ {1} \left(X _ {1} \cup \{x ^ {\prime} \}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1} \cup \{x ^ {\prime} \}\right), X \backslash X _ {1}\right)\right) \right. \right. \\ \left. + (1 - q) \cdot \left(\mathcal {M} _ {1} \left(X _ {1}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1}\right), X \backslash X _ {1} \cup \{x ^ {\prime} \}\right)\right) | | \left(\mathcal {M} _ {1} \left(X _ {1}\right), \mathcal {M} _ {2} \left(\mathcal {M} _ {1} \left(X _ {1}\right), X \backslash X _ {1}\right)\right)\right)\left. \right) \\ = \iint \left(\frac {q \cdot \widetilde {P} (t) \cdot Q (t , s) + (1 - q) \cdot P (t) \cdot \widetilde {Q} (t , s)}{P (t) \cdot Q (t , s)}\right) ^ {\alpha} \cdot P (t) \cdot Q (t, s) d t d s. \tag {C.4} \\ \end{array}
$$
which can be expanded as
$$
\begin{array}{l} \iint \left(\frac {q \cdot \widetilde {P} (t) \cdot Q (t , s) + (1 - q) \cdot P (t) \cdot \widetilde {Q}}{P (t) \cdot Q (t , s)}\right) ^ {\alpha} P (t) \cdot Q (t, s)   \mathrm {d} t   \mathrm {d} s \\ = \iint \left(q \cdot \frac {\widetilde {P} (t)}{P (t)} + (1 - q) \cdot \frac {\widetilde {Q} (t , s)}{Q (t , s)}\right) ^ {\alpha} P (t) \cdot Q (t, s) d t d s \\ = \int \int q ^ {\alpha} \left(\frac {\widetilde {P} (t)}{P (t)}\right) ^ {\alpha} P (t) \cdot Q (t, s) d t d s \\ + \iint (1 - q) ^ {\alpha} \left(\frac {\widetilde {Q} (t , s)}{Q (t , s)}\right) ^ {\alpha} P (t) \cdot Q (t, s) d t d s \tag {C.5} \\ + \int \int \alpha \cdot q ^ {\alpha - 1} \cdot (1 - q) \cdot \left(\frac {\widetilde {P} (t)}{P (t)}\right) ^ {\alpha - 1} P (t) \cdot \widetilde {Q} (t, s) d t d s \\ + \int \int \alpha \cdot q \cdot (1 - q) ^ {\alpha - 1} \cdot \left(\frac {\widetilde {Q} (t , s)}{Q (t , s)}\right) ^ {\alpha - 1} Q (t, s) \cdot \widetilde {P} (t) d t d s \\ + \iint_ {j = 2} ^ {\alpha - 2} \binom {\alpha} {j} \cdot q ^ {\alpha - j} \cdot (1 - q) ^ {j} \cdot \left[ \left(\frac {\widetilde {P} (t)}{P (t)}\right) ^ {\alpha - j} P (t) \right] \left[ \left(\frac {\widetilde {Q} (t , s)}{Q (t , s)}\right) ^ {j} Q (t, s) \right]   \mathrm {d} t   \mathrm {d} s. \\ \end{array}
$$
We next bound five integrals on the right hand side of Equation (C.5). For the first two integrals, we use the RDP-bounds for  $\mathcal{M}_1$  and  $\mathcal{M}_2$  to obtain
$$
\iint \left(\frac {\widetilde {P} (t)}{P (t)}\right) ^ {\alpha} P (t) Q (t, s) d t d s = \int \left(\frac {\widetilde {P} (t)}{P (t)}\right) ^ {\alpha} P (t) d t \leq \exp \big ((\alpha - 1) \varepsilon_ {1} (\alpha) \big). \tag {C.6}
$$
$$
\iint \left(\frac {\widetilde {Q} (t , s)}{Q (t , s)}\right) ^ {\alpha} Q (t, s) P (t) \mathrm {d} s \mathrm {d} t \leq \int \exp \left((\alpha - 1) \varepsilon_ {2} (\alpha)\right) P (t) \mathrm {d} t = \exp \left((\alpha - 1) \varepsilon_ {2} (\alpha)\right), \tag {C.7}
$$
where  $\varepsilon_{1}$  and  $\varepsilon_{2}$  give the RDP-parameters of order  $\alpha$  for  $\mathcal{M}_1$  and  $\mathcal{M}_2$ , respectively. The third and fourth integral can be bounded analogously. In the second inequality we have also used the fact
that the RDP-parameters of  $\mathcal{M}_2$  are independent of the auxiliary variable  $t$ . Similarly, for the third integral, we have
$$
\begin{array}{l} \iint \left[ \left(\frac {\widetilde {P} (t)}{P (t)}\right) ^ {\alpha - j} P (t) \right] \left[ \left(\frac {\widetilde {Q} (t , s)}{Q (t , s)}\right) ^ {j} Q (t, s) \right] \mathrm {d} s \mathrm {d} t \\ \leq \int \left[ \left(\frac {\widetilde {P} (t)}{P (t)}\right) ^ {\alpha - j} P (t) \right] \exp \left((j - 1) \varepsilon_ {2} (j)\right) d t \tag {C.8} \\ \leq \exp \left((\alpha - j - 1) \varepsilon_ {1} (\alpha - j)\right) \cdot \exp \left((j - 1) \varepsilon_ {2} (j)\right). \\ \end{array}
$$
Substituting (C.6), (C.7) (and similar expressions for the third and fourth integral) and (C.8) to Equation (C.5), we get a bound for the right-hand side of Equation (C.4). Since  $X_{1} \subset X$  was arbitrary, we arrive at the claim via the inequality (C.3).
Next, we consider bounding  $D_{\alpha}\big(\mathcal{M}(X) || \mathcal{M}(Y)\big)$ . The proof goes similarly as the one for  $D_{\alpha}\big(\mathcal{M}(Y) || \mathcal{M}(X)\big)$ . Denote
$$
\varepsilon_ {2} (\alpha) = D _ {\alpha} \big (\mathcal {M} (X) | | \mathcal {M} (X \cup \{x ^ {\prime} \}) \big).
$$
With the notation of proof of Thm. 6, we see that, instead of the right-hand side of (C.4), we need to bound
$$
\begin{array}{l} \exp \left((\alpha - 1) \varepsilon_ {2} (\alpha)\right) \\ = \int \int \left(\frac {P (t) \cdot Q (t , s)}{q \cdot \widetilde {P} (t) \cdot Q (t , s) + (1 - q) \cdot P (t) \cdot \widetilde {Q} (t , s)}\right) ^ {\alpha} (q \cdot \widetilde {P} (t) \cdot Q (t, s) + (1 - q) \cdot P (t) \cdot \widetilde {Q} (t, s))   \mathrm {d} t   \mathrm {d} s. \\ \end{array}
$$
In order to use here the series approach, we need to use Lemma 10:
$$
\begin{array}{l} \left(\frac {P \cdot Q}{q \cdot \widetilde {P} \cdot Q + (1 - q) \cdot P \cdot \widetilde {Q}}\right) ^ {\alpha} (q \cdot \widetilde {P} \cdot Q + (1 - q) \cdot P \cdot \widetilde {Q}) \\ = \left(\frac {P \cdot Q}{q \cdot \widetilde {P} \cdot Q + (1 - q) \cdot P \cdot \widetilde {Q}}\right) ^ {\alpha - 1} \cdot P \cdot Q \\ = \left(q \cdot \frac {\widetilde {P}}{P} + (1 - q) \cdot \frac {\widetilde {Q}}{Q}\right) ^ {1 - \alpha} \cdot P \cdot Q \\ = \left(q \cdot \frac {\widetilde {P}}{P} \frac {Q}{\widetilde {Q}} + (1 - q)\right) ^ {1 - \alpha} \cdot \left(\frac {\widetilde {Q}}{Q}\right) ^ {1 - \alpha} \cdot P \cdot Q \tag {C.9} \\ = \left(q \cdot \frac {\widetilde {P}}{P} \frac {Q}{\widetilde {Q}} + (1 - q)\right) ^ {1 - \alpha} \cdot \left(\frac {Q}{\widetilde {Q}}\right) ^ {\alpha - 1} \cdot P \cdot Q \\ \leq \left(q \cdot \frac {P}{\widetilde {P}} \frac {\widetilde {Q}}{Q} + (1 - q)\right) ^ {\alpha - 1} \cdot \left(\frac {Q}{\widetilde {Q}}\right) ^ {\alpha - 1} \cdot P \cdot Q \\ = \left(q \cdot \frac {P}{\widetilde {P}} \frac {\widetilde {Q}}{Q} + (1 - q)\right) ^ {\alpha - 1} \cdot \left(\frac {Q}{\widetilde {Q}}\right) ^ {\alpha} \cdot P \cdot \widetilde {Q}, \\ \end{array}
$$
where in the inequality we have used Lemma 10. Now we can expand  $\left(q\cdot \frac{P}{P}\frac{\widetilde{Q}}{Q} +1 - q\right)^{\alpha -1}$
$$
\begin{array}{l} \left(1 - q + q \cdot \frac {P}{\widetilde {P}} \frac {\widetilde {Q}}{Q}\right) ^ {\alpha - 1} \cdot \left(\frac {Q}{\widetilde {Q}}\right) ^ {\alpha} \cdot P \cdot \widetilde {Q} \\ = \left(\sum_ {j = 0} ^ {\alpha - 1} \binom {\alpha - 1} {j} q ^ {j} \cdot (1 - q) ^ {\alpha - 1 - j} \cdot \left(\frac {P}{\widetilde {P}}\right) ^ {j} \left(\frac {\widetilde {Q}}{Q}\right) ^ {j}\right) \cdot \left(\frac {Q}{\widetilde {Q}}\right) ^ {\alpha} \cdot P \cdot \widetilde {Q} \\ = \left(\sum_ {j = 0} ^ {\alpha - 1} \binom {\alpha - 1} {j} q ^ {j} \cdot (1 - q) ^ {\alpha - 1 - j} \cdot \left(\frac {P}{\widetilde {P}}\right) ^ {j} \left(\frac {Q}{\widetilde {Q}}\right) ^ {\alpha - j}\right) \cdot P \cdot \widetilde {Q} \\ = \sum_ {j = 0} ^ {\alpha - 1} \binom {\alpha - 1} {j} q ^ {j} \cdot (1 - q) ^ {\alpha - 1 - j} \cdot \left(\frac {P}{\widetilde {P}}\right) ^ {j + 1} \widetilde {P} \cdot \left(\frac {Q}{\widetilde {Q}}\right) ^ {\alpha - j} \widetilde {Q}. \\ \end{array}
$$
Then, we use the known  $\varepsilon_1(\alpha)$  and  $\varepsilon_{2}(\alpha)$ -values as in the inequalities (C.8) to arrive at the claim.
C.1 Auxiliary Lemma
We need the following inequality for the proof of Thm 6.
Lemma 10 (Lemma 35, Steinke 2022). For all  $p \in [0,1]$  and  $x \in (0,\infty)$ ,
$$
\frac {1}{1 - p + \frac {p}{x}} \leq 1 - p + p \cdot x.
$$
D Additional Details to Section 4
D.1 Random Search
Here, we assume we are given some distributions of the hyperparameter candidates and the algorithm  $Q$  draws hyperparameters using them. In order to adjust the noise level for each candidate, we take a  $\alpha$ -line as an RDP upper bound. More specifically, we require that the candidate models are  $(\alpha, c \cdot \alpha)$ -RDP for some  $c > 0$  and for all  $\alpha \in \Lambda$ . Then the noise scale  $\sigma_{\gamma, T}$  for each draw of  $(\gamma, T)$  is the minimum scale for which the  $(\alpha, c \cdot \alpha)$ -RDP bound holds, i.e.,
$$
\sigma_ {\gamma , T} = \min  \left\{\sigma \in \mathbb {R} ^ {+}: T \cdot \varepsilon_ {\gamma , \sigma} (\alpha) \leq c \cdot \alpha \text {f o r a l l} \alpha \in \Lambda \right\}.
$$
Similarly, we can find the maximum  $T$  based on  $\sigma$  and  $\gamma$  such that the mechanism is  $(\alpha, c \cdot \alpha)$ -RDP for all  $\alpha \in \Lambda$ . Again, by Lemma 9 below, the candidate picking algorithm  $Q$  is then  $c \cdot \alpha$ -RDP and we may use Thm. 5 to obtain RDP bounds for the tuning algorithm.
D.1.1 Adjusting the Parameters  $T$  and  $\sigma$  for DP-SGD
We next discuss the reasons for the success of strategies described in Section 4. It is often a good approximation to say that the RDP-guarantees of the Poisson subsampled Gaussian mechanism are lines as functions of the RDP order  $\alpha$ , i.e., that the guarantees are those a Gaussian mechanism with some sensitivity and noise level values. For example, (Thm. 11, Mironov et al., 2019) show that the Poisson subsampled Gaussian mechanism is  $(\alpha, 2\gamma^2\alpha/\sigma^2)$ -RDP when  $\alpha$  is sufficiently small. Also, (Thm. 38, Steinke, 2022) show that if the underlying mechanism is  $\rho$ -zCDP, then the Poisson subsampled version with subsampling ratio  $\gamma$  is  $(\alpha, 10\gamma^2\rho\alpha)$ -RDP when  $\alpha$  is sufficiently small. Notice that the Gaussian mechanism with  $L_2$ -sensitivity  $\Delta$  and noise level  $\sigma$  is  $(\Delta^2/2\sigma^2)$ -zCDP (Bun and Steinke, 2016).
We numerically observe, that the larger the noise level  $\sigma$  and the smaller the subsampling ratio  $\gamma$ , the better the line approximation of the RDP-guarantees (see Figure 9).
In case the privacy guarantees (either  $(\varepsilon, \delta)$ -DP or RDP) are approximately those of a Gaussian mechanisms with some sensitivity and noise level values, both of the approaches for tuning the hyperparameters  $\gamma$ ,  $\sigma$  and  $T$  described in Section 4 would lead to very little slack. This is because for the Gaussian mechanism, both the RDP guarantees (Mironov, 2017) and  $(\varepsilon, \delta)$ -DP guarantees (Dong et al., 2022) depend monotonously on the scaled parameter
$$
\widetilde {\sigma} = \frac {\sigma}{\Delta \cdot \sqrt {T}}.
$$
This means that if we adjust the training length  $T$  based on values of  $\sigma$  by having some target  $(\delta, \varepsilon)$ -bound for the candidate model with grid search of Section 4.1, the resulting RDP upper bounds of different candidates will not be far from each other (and similarly for adjusting  $\sigma$  based on value of  $T$ ). Similarly, for random search of Section D.1, when adjusting  $T$  based on values of  $\sigma$ , the RDP guarantees of all the candidate models would be close to the upper bound  $(c \cdot \alpha, c > 0)$ , i.e., they would not be far from each other.
Figure 9: DP-SGD RDP curves for different values of noise level  $\sigma$  and number of compositions  $T$ . Left:  $\gamma = 1/100$ , right:  $\gamma = 1/50$  and the corresponding lines with the smallest slope that give upper bounds for the RDP orders up to  $\alpha = 24$ .
D.2 Proof of Lemma 9
Lemma 11. Denote by  $\beta$  the random variable of which outcomes are the hyperparameter candidates (drawing either randomly from a grid or from given distributions). Consider an algorithm  $Q$ , that first randomly picks hyperparameters  $t \sim \beta$ , then runs a randomised mechanism  $\mathcal{M}(t, X)$ . Suppose  $\mathcal{M}(t, X)$  is  $(\alpha, \varepsilon(\alpha))$ -RDP for all  $t$ . Then,  $Q$  is  $(\alpha, \varepsilon(\alpha))$ -RDP.
Proof. Suppose the hyperparameters  $t$  are outcomes of a random variable  $\beta$ . Let  $X$  and  $Y$  be neighbouring datasets. Then, if  $p(t,s)$  and  $q(t,s)$  (as functions of  $s$ ) give the density functions of  $\mathcal{M}(t,X)$  and  $\mathcal{M}(t,Y)$ , respectively, we have that
$$
Q (X) \sim \mathbb {E} _ {t \sim \beta} p (t, s) \quad \text {a n d} \quad Q (Y) \sim \mathbb {E} _ {t \sim \beta} q (t, s).
$$
Since for any distributions  $p$  and  $q$ , and for any  $\alpha > 1$ ,  $\exp \left((\alpha - 1)D_{\alpha}(p||q)\right) = \int \left(\frac{p(t)}{q(t)}\right)^{\alpha} q(t) \, \mathrm{d}t$  gives an  $f$ -divergence (for  $f(x) = x^{\alpha}$ ), it is also jointly convex w.r.t.  $p$  and  $q$  (Liese and Vajda, 2006). Using Jensen's inequality, we have
$$
\begin{array}{l} \exp \left((\alpha - 1) D _ {\alpha} \big (Q (X) | | Q (Y) \big)\right) = \int \left(\frac {\mathbb {E} _ {t \sim \beta} p (t , s)}{\mathbb {E} _ {t \sim \beta} q (t , s)}\right) ^ {\alpha} \cdot \mathbb {E} _ {t \sim \beta} q (t, s) d s \\ \leq \mathbb {E} _ {t \sim \beta} \int \left(\frac {p (t , s)}{q (t , s)}\right) ^ {\alpha} \cdot q (t, s) d s \\ \leq \mathbb {E} _ {t \sim \beta} \exp \left(\left(\alpha - 1\right) D _ {\alpha} \left(\mathcal {M} (t, X) | | \mathcal {M} (t, Y)\right)\right) \\ = \exp \left((\alpha - 1) \varepsilon (\alpha)\right) \\ \end{array}
$$
E  $f$ -Divergence of Parallel Compositions
We first formulate the parallel composition result for general  $f$ -divergences (Lemma 12). We then obtain the RDP bound for parallel compositions as a corollary (Cor. 13).
Our Lemma 12 below can be seen as an  $f$ -divergence version of the  $(\varepsilon, 0)$ -DP result given in (Thm. 4 McSherry, 2009). Corollary 2 by (Smith et al., 2022) gives the corresponding result in terms of  $\mu$ -Gaussian differential privacy (GDP), and it is a special case of our Lemma 12 as  $\mu$ -GDP equals the  $(\varepsilon, \delta)$ -DP (i.e., the hockey-stick divergence) of the Gaussian mechanism with a certain noise scale (Cor. 1, Dong et al., 2022).
We define  $f$ -divergence for distributions on  $\mathbb{R}^d$  as follows. Consider two probability densities  $P$  and  $Q$  defined on  $\mathbb{R}^d$ , such that if  $Q(x) = 0$  then also  $P(x) = 0$ , and a convex function  $f:[0,\infty)\to \mathbb{R}$ . Then, an  $f$ -divergence (Liese and Vajda, 2006) is defined as
$$
D _ {f} (P | | Q) = \int f \left(\frac {P (t)}{Q (t)}\right) Q (t) d t.
$$
In case the data is divided into disjoint shards and separate mechanisms are applied to each shard, the  $f$ -divergence upper bound for two neighbouring datasets can be obtained from the individual  $f$ -divergence upper bounds:
Lemma 12. Suppose a dataset  $X \in \mathcal{X}^N$  is divided into  $k$  disjoint shards  $X_{i}$ ,  $i \in [k]$ , and mechanisms  $\mathcal{M}_i$ ,  $i \in [k]$ , are applied to the shards, respectively. Consider the adaptive composition