SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1 illustrates how the  $(\varepsilon, \delta)$ -bounds of the two variants (3.1) and (3.2) behave as functions of the sampling parameter  $q$  used for sampling the tuning set  $X_{1}$ , when the base mechanism DP-SGD is run for 50 epochs with the subsampling ratio  $\gamma = 0.01$  and noise level  $\sigma = 2.0$ . The bounds for the variant 1 given in Equation (3.1) are computed using the RDP results of Thm. 6 and the bounds for the variant 2 are computed using the subsampling amplification result of Thm. 4. The RDP bounds are converted to  $(\varepsilon, \delta)$ -bounds using the conversion rule of Lemma 3 with  $\delta = 10^{-5}$ . The fact that the bounds for the variants 1 and 2 cross when  $\mu = 45$  at small values of  $q$  suggests that the bounds of Thm. 6 could still be tightened.
Figure 1: Comparison of  $(\varepsilon, \delta)$ -bounds for the variant 1 given in Equation (3.1) and the variant 2 given in Equation (3.2) as a function of the subsampling ratio  $q$  used for sampling the tuning set  $X_{1}$ . Also shown is the  $(\varepsilon, \delta)$ -bound for the baseline algorithm described in Thm. 5. Here  $\mu$  refers to the expected number of model evaluations in the tuning algorithm.
3.4 Computational Savings
Our scaling approach for DP-SGD described in Section 3.2 implies that the DP-SGD subsampling ratio  $\gamma$ , the noise level  $\sigma$  and the number of iterations  $T$  are the same when evaluating the private candidate models using the tuning set  $X_{1}$  and when evaluating the final model using the larger dataset.
Thus, if we run the base algorithm for E epochs, we easily see that the expected number of required gradient evaluations for the variant 1 given in (3.1) is given by  $\mathrm{E} \cdot (\mu \cdot q \cdot n + (1 - q) \cdot n)$  and for the variant 2 given in (3.2) it is given by  $\mathrm{E} \cdot (\mu \cdot q \cdot n + n)$ , whereas the baseline requires in expectation  $\mu \cdot n \cdot \mathrm{E}$  evaluations in case it is carrying out tuning with the same hyperparameter candidates as our method. Since the number of iterations  $T$  is kept fixed, there are some constant overheads in the compute cost such as those coming from the model updates. Therefore the actual speed ups are slightly smaller.
For example, in our experiments with  $\mu = 15$  and  $q = 0.1$ , the baseline requires  $\frac{\mu}{\mu \cdot q + 1} \approx 6$  times more gradient evaluations than our method and when  $\mu = 45$  and  $q = 0.1$  the baseline requires  $\approx 8$  times more gradient evaluations. The actual speed ups are shown in the figures of Section 5.
4 Dealing with DP-SGD Hyperparameters that Affect the DP Guarantees
Thm. 5 gives RDP-parameters of order  $\alpha$  for the tuning algorithm, assuming the underlying candidate picking algorithm is  $(\alpha, \varepsilon(\alpha))$ -RDP. In case of DP-SGD, if we are tuning the learning rate  $\eta$  or clipping constant  $C$ , and fix rest of the hyperparameters, these  $(\alpha, \varepsilon(\alpha))$ -RDP bounds are fixed for all the hyperparameter candidates. However, if we are tuning hyperparameters that affect the DP guarantees, i.e., the subsampling ratio  $\gamma$ , the noise level  $\sigma$  or the length of the training  $T$ , it is less straightforward to determine suitable uniform  $\varepsilon(\alpha)$ -upper bounds. As is common practice, we consider a grid  $\Lambda$  of  $\alpha$ -orders for RDP bookkeeping (e.g. integer values of  $\alpha$ 's).
4.1 Grid Search with Randomization
To deal with this problem, we first set an approximative DP target value  $(\varepsilon, \delta)$  that we use to adjust some of the hyperparameters. For example, if we are tuning the subsampling ratio  $\gamma$  and training length  $T$ , we can, for each choice of  $(\gamma, T)$ , adjust the noise scale  $\sigma$  so that the resulting training iteration is at most  $(\varepsilon, \delta)$ -DP. Vice versa, we may tune  $\gamma$  and  $\sigma$ , and take maximal value of  $T$  such that the resulting training iteration is at most  $(\varepsilon, \delta)$ -DP.
More specifically, we first fix  $\varepsilon, \delta > 0$  which represent the target approximative DP bound for each candidate model. Denote by  $\varepsilon(T, \delta, \gamma, \sigma)$  the  $\varepsilon$ -value of the subsampled Gaussian mechanism with parameter values  $\gamma, \sigma$  and  $T$  and for fixed  $\delta$ . To each pair of  $(\gamma, T)$ , we attach a noise scale  $\sigma_{\gamma, T}$  such that it is the smallest number with which the resulting composition is  $(\varepsilon, \delta)$ -DP:
$$
\sigma_ {\gamma , T} = \min  \{\sigma \in \mathbb {R} ^ {+}: \varepsilon (T, \delta , \gamma , \sigma) \leq \varepsilon \}.
$$
As the RDP values increase monotonously w.r.t. the number of compositions, it is straightforward to find  $\sigma_{\gamma ,T}$ , e.g., using the bisection method. Alternatively, we could fix a few values of  $\sigma$ , and to each combination of  $(\gamma ,\sigma)$ , attach the largest  $T$  (denoted  $T_{\gamma ,\sigma}$ ) such that the target  $(\varepsilon ,\delta)$ -guarantee holds.
We consider a finite grid  $\Gamma$  of possible hyperparameter values  $t$  (e.g.,  $t = (\gamma, \sigma, T)$ , where  $T$  is adjusted to  $\gamma$  and  $\sigma$ ). Then, for all  $t \in \Gamma$ , we compute the corresponding RDP value  $\varepsilon_t(\alpha)$  for each  $\alpha \in \Lambda$ . Finally, for each  $\alpha \in \Lambda$ , we set
$$
\varepsilon (\alpha) = \max_{t\in \Gamma}\varepsilon_{t}(\alpha).
$$
Then, since for each random draw of  $t$ , the DP-SGD trained candidate model is  $\varepsilon(\alpha)$ -RDP, by Lemma 9 given below, the candidate picking algorithm  $Q$  is also  $\varepsilon(\alpha)$ -RDP. This approach is used in the experiments of Figure 4, where we jointly tune  $T$ ,  $\gamma$  and  $\eta$ .
4.2 RDP Analysis
For completeness, in Appendix D we prove the following result which gives RDP bounds for the case we randomly draw hyperparemeters that affect the privacy guarantees of the candidate models.
Lemma 9. Denote by  $\beta$  the random variable of which outcomes are the hyperparameter candidates (drawing either randomly from a grid or from given distributions). Consider an algorithm  $Q$ , that first randomly picks hyperparameters  $t \sim \beta$ , then runs a randomized mechanism  $\mathcal{M}(t, X)$ . Suppose  $\mathcal{M}(t, X)$  is  $(\alpha, \varepsilon(\alpha))$ -RDP for all  $t$ . Then,  $Q$  is  $(\alpha, \varepsilon(\alpha))$ -RDP.
We perform our evaluations on standard benchmark datasets for classification: CIFAR-10 (Krizhevsky and Hinton, 2009), MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017) and IMDB (Maas et al., 2011). Full details of the experiments are given in Appendix A. When reporting the results, we set  $\delta = 10^{-5}$  in all experiments.
Learning rate tuning. Figures 2 and 3 summarize the results for learning rate tuning using our methods and the baseline method by Papernot and Steinke (2022). The learning rate grid size is either 9 or 10 and the grid values are specified in Table 2 (Appendix). We fix the subsampling ratio  $\gamma$  and the number of epochs to the values given in Table 1 (Appendix) for all models. The batch sizes in the submodels are defined by scaling  $\gamma$  on the corresponding dataset sizes. We use  $q = 0.1$  which means that, for example, if the Poisson subsampling of DP-SGD gives in expectation a batch size of 128 in the tuning phase of our methods, the expected batch sizes for the final models of variant 1 and 2 are 1152 and 1280, respectively. We use  $\mu = 15$  (the expected number of runs for the tuning algorithm).
(a) trained with DP-SGD
(b) trained with DP-SGD
(c) trained with DP-SGD
(d) trained with DP-SGD
Figure 2: Tuning learning rate with DP-SGD. Test accuracies are averaged across 10 independent runs and the error bars denote the standard error of the mean. The numbers in the legends refer to the mean training timings of the baseline scaled with respect to minimum of variant 1 and 2. For example, for CIFAR-10, the average training time for the baseline method is 6.06 times bigger than for the fastest of our methods. For perspective, we also add curves showing the privacy cost of training a single model with optimal hyperparameters obtained from the baseline.
Tuning all hyperparameters. Next, we jointly optimize the number of epochs, the DP-SGD subsampling ratio  $\gamma$ , and the learning rate  $\eta$  using the hyperparameter candidates given in Table 2 (Appendix). The remaining setup is the same as in the previous experiment. Figure 4 shows the same
(a) trained with DP-Adam
(b) trained with DP-Adam
(c) trained with DP-Adam
(d) trained with DP-Adam
Figure 3: Tuning learning rate with DP-Adam. Test accuracies are averaged across 10 independent runs and the error bars denote the standard error of the mean. The numbers in the legends refer to the mean training timings of the baseline scaled with respect to minimum of variant 1 and 2. For example, for FashionMNIST, the average training time for the baseline method is 9.03 times bigger than the fastest of our methods. For perspective, we also add curves showing the privacy cost of training a single model with optimal hyperparameters obtained from the baseline. Figure 6 (Appendix) shows a more detailed version of this plot.
quantities as Figures 2 and 3, however, higher values of  $\mu$  are used to accommodate to increased hyperparameter spaces.
Takeaways. Overall, for both DP-SGD and DP-Adam, we observe that both variants of our method provide better privacy-utility trade-off and have a lower computational cost than the baseline method. Additionally, we also note the benefits of tailored analysis (Thm 6) in Figure 2 in terms of slightly higher accuracy for variant 1 compared to variant 2 for DP-SGD. In Figure 4 where we are also tuning the batch size and epochs, the noise levels are higher compared to Figure 2. One reason for slightly better privacy-utility trade-off for variant 2 with DP-SGD is possibly the fact that our tailored bound is less tight for higher values of  $\mu$  and small values of  $q$  (see Figure 1).
We have considered a simple strategy for lowering the privacy cost and computational cost of DP hyperparameter tuning: we carry out tuning using a random subset of data and extrapolate the optimal values to the larger training set used to train the final model. We have also provided methods to tune the hyperparameters that affect the DP guarantees of the model training themselves, those
(d) trained with DP-SGD
Figure 4: Tuning of subsampling ratio, training epochs, and learning rate with DP-SGD. Test accuracies are averaged across 10 independent runs and the error bars denote the standard error of the mean. The numbers in the legends refer to the mean timings of the baseline method scaled with respect to the minimum of variant 1 and 2. For perspective, we also add curves showing the privacy cost of training a single model with optimal hyperparameters obtained from the baseline. Figure 7 (Appendix) shows a more detailed version of this plot.
being the noise level and subsampling ratio in case of DP-SGD. Our experiments show a clear improvement over the baseline method by Papernot and Steinke (2022) when tuning DP-SGD for neural networks and using simple but well-justified heuristics for the extrapolation. One obvious limitation of our method is that it is limited to DP-SGD although we show also positive results for the Adam optimizer combined with DP-SGD gradients. Finding out whether effective scaling rules could be derived for DP-SGD with momentum, DP-FTRL (Kairouz et al., 2021) or more refined adaptive DP optimizers such as  $\mathrm{DP^{2}}$ -RMSprop (Li et al., 2023), is left for future work. An interesting avenue of future work is also to find more black-box type of extrapolation methods, something that has been considered in the non-DP case (see, e.g. Klein et al., 2017). Another interesting question is how to carry out more accurate privacy analysis using the so-called privacy loss distributions (PLDs) and numerical accounting. The main reason for using RDP instead of PLD accounting in this work was that the tightest privacy bounds for DP hyperparameter tuning are given in terms of RDP. We have used Poisson subsampling to obtain the tuning set as it is relatively easy to analyze, however, other sampling methods could be analyzed as well.
7 Broader Impact Statement
Our present work makes DP machine learning more appealing and potentially more widespread in practice. This can improve privacy protection in general but can also carry negative side effects. The proposed method can make DP learning more appealing by improving DP hyperparameter tuning leading to higher utility machine learning models at equivalent provable total privacy cost. Our method has lower compute cost than some alternatives, potentially leading to considerable resource savings. While DP gives strong privacy guarantees, it may have negative impacts as well, as some DP learning methods have for example been shown to have a relatively lower utility for minorities.
We would like to thank our colleague Laith Zumot for discussions about hyperparameter tuning methods at the early stages of the project.
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. (2016). Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308-318.
Balle, B., Barthe, G., and Gaboardi, M. (2018). Privacy amplification by subsampling: Tight analyses via couplings and divergences. In Advances in Neural Information Processing Systems, volume 31.
Bassily, R., Smith, A., and Thakurta, A. (2014). Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464-473. IEEE.
Bun, M. and Steinke, T. (2016). Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography Conference, pages 635-658. Springer.
Canonne, C., Kamath, G., and Steinke, T. (2020). The discrete Gaussian for differential privacy. In Advances in Neural Information Processing Systems, volume 33.
Chaudhuri, K. and Vinterbo, S. A. (2013). A stability-based validation procedure for differentially private machine learning. Advances in Neural Information Processing Systems, 26.
Dong, J., Roth, A., Su, W. J., et al. (2022). Gaussian differential privacy. Journal of the Royal Statistical Society Series B, 84(1):3-37.
Dwork, C. (2006). Differential privacy. In Proc. 33rd Int. Colloq. on Automata, Languages and Prog. (ICALP 2006), Part II, pages 1-12.
Gopi, S., Lee, Y. T., and Wutschitz, L. (2021). Numerical composition of differential privacy. Advances in Neural Information Processing Systems, 34:11631-11642.
Horváth, T., Mantovani, R. G., and de Carvalho, A. C. (2017). Effects of random sampling on svm hyper-parameter tuning. In International Conference on Intelligent Systems Design and Applications, pages 268-278. Springer.
Kairouz, P., McMahan, B., Song, S., Thakkar, O., Thakurta, A., and Xu, Z. (2021). Practical and private (deep) learning without sampling or shuffling. In International Conference on Machine Learning, pages 5213-5225. PMLR.
Killamsetty, K., Abhishek, G. S., Lnu, A., Ramakrishnan, G., Evfimievski, A., Popa, L., and Iyer, R. (2022). Automata: Gradient based data subset selection for compute-efficient hyper-parameter tuning. Advances in Neural Information Processing Systems, 35:28721-28733.
Killamsetty, K., Durga, S., Ramakrishnan, G., De, A., and Iyer, R. (2021). Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 5464-5474. PMLR.
Song, S., Chaudhuri, K., and Sarwate, A. D. (2013). Stochastic gradient descent with differentially private updates. In 2013 IEEE global conference on signal and information processing, pages 245-248. IEEE.
Steinke, T. (2022). Composition of differential privacy & privacy amplification by subsampling. arXiv preprint arXiv:2210.00597.
Swersky, K., Snoek, J., and Adams, R. P. (2013). Multi-task Bayesian optimization. Advances in neural information processing systems, 26.
Thakurta, A. G. and Smith, A. (2013). Differentially private feature selection via stability arguments, and the robustness of the lasso. In Conference on Learning Theory, pages 819-850. PMLR.
van der Veen, K. L., Seggers, R., Bloem, P., and Patrini, G. (2018). Three tools for practical differential privacy. NeurIPS 2018 Privacy Preserving Machine Learning workshop, arXiv:1812.02890.
Van Erven, T. and Harremos, P. (2014). Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3797-3820.
Waring, J., Lindvall, C., and Umeton, R. (2020). Automated machine learning: Review of the state-of-the-art and opportunities for healthcare. Artificial intelligence in medicine, 104:101822.
Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Technical report.
Yousefpour, A., Shilov, I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj, A., Zhao, J., et al. (2021). Opacus: User-friendly differential privacy library in PyTorch. In NeurIPS 2021 Workshop Privacy in Machine Learning, arXiv:2109.12298.
Zhu, Y., Dong, J., and Wang, Y.-X. (2022). Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics, pages 4782-4817. PMLR.
Zhu, Y. and Wang, Y.-X. (2019). Poisson subsampled Rényi differential privacy. In International Conference on Machine Learning, pages 7634-7642.
A Full Description of Experiments
Quality Metric and Evaluation. In all of our experiments, we have a partitioning of the available data into train and test sets and we choose the best model based on the test accuracy. The quality score applied on the test set is a relatively low sensitivity function, and therefore, even for a private test set, parallel composition (for an RDP bound of parallel compositions, we refer to Appendix E) can accommodate DP evaluation of a quality metric in the training budget itself. However, we assume that only the training dataset is private and the test data is public for simplicity. This assumption of test dataset being public and the approach of making only two (train and test) partitions of available data instead of three (train, validation, and test) has been considered in many prior works (Mohapatra et al., 2022; Papernot and Steinke, 2022) to study the proposed method in isolation.
Our utility plots (Figures 2 to Figure 8) show the test accuracy of the final model against the final approximate DP  $\varepsilon$  which includes the privacy cost of the tuning process and of the final model for all methods. We fix  $\delta = 10^{-5}$  always when reporting the  $\varepsilon$ -values. We fix  $q = 0.1$  for all of our methods in all experiments. We mention that in several cases smaller value of  $q$  would have lead to better privacy-accuracy trade-off of the final model, however, we use the same value  $q = 0.1$  in all experiments for consistency.
Methods. We consider the both variants of our proposed approach in our experiments. The RDP parameters for the variant 1 are obtained by using Thm. 6 (new RDP result) and for variant 2 by combining Thm. 4 (subsampling amplification) with Thm. 5 (RDP cost of the hyperparameter tuning). We scale the hyperparameters in our methods for the training data of the final model as discussed in Sec. 3.2. The method by Papernot and Steinke (2022) described in Thm. 5 is the baseline.
Datasets and Models. We carry out our experiments on the following standard benchmark datasets for classification: CIFAR-10 (Krizhevsky and Hinton, 2009), MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017) and IMDB (Maas et al., 2011). For MNIST and IMDB, we use the convolutional neural networks from the examples provided in the Opacus library Yousefpour et al. (2021). For FashionMNIST, we consider a simple feedforward 3-layer network with hidden layers of width 120. For CIFAR-10, we use a Resnet20 pre-trained on CIFAR-100 (Krizhevsky and Hinton, 2009) dataset so that only the last fully connected layer is trained. We minimize the cross-entropy loss in all models. We optimize with DP-SGD and DP-Adam for all datasets. As suggested by Papernot et al. (2020), we replace all ReLU activations with tempered sigmoid functions in CIFAR-10, MNIST, and FashionMNIST networks, which limits the magnitudes of non-private gradients and improves model accuracies.
Hyperparameters. For these datasets, in one of the experiments we tune only the learning rate  $(\eta)$  and in the other one  $\eta$ ,  $\gamma$  (subsampling ratio), and the number of epochs, while fixing the clipping constant  $C$ . For DP-Adam, we do not scale the learning rate. The number of trainable parameters and the hyperparameter grids are provided in Table 2 (Appendix B). The numbers of epochs are chosen to suit our computational constraints. Following the procedure described in Section 4.1, we always adjust  $\sigma$  such that we compute the smallest  $\sigma$  that satisfies a target final  $(\varepsilon, \delta)$  bound for each  $(\gamma, \mathrm{epoch})$  pair. Furthermore following the RDP accounting procedure described in Section 4.1 we obtain uniform RDP guarantees for the candidate models and furthermore RDP guarantees for our methods.
Implementation. For the implementation of DP-SGD and DP-Adam, we use the Opacus library (Yousefpour et al., 2021). For scalability, we explore the hyperparameter spaces with Ray Tune (Liaw et al., 2018) on a dedicated multi-GPU cluster. We use two sets of gpus and one set is much weaker than the other. All three methods have independent randomness (e.g.  $K$  hyperparameter candidates sampled, DP noise). Ray tune maintains a job queue and the number of models trained parallelly depends on the cpu count. Each model is trained on 0.5 cpu, and each cpu has enough memory to accommodate 2 models. Models finish their entire training on the same cpu that was allocated to them at the start of their training.
Measuring training time. We want to reduce the dependence of our measurements on the available hardware resources (e.g. number of gpus). For each method, we store the per epoch time in seconds for each model being trained in parallel, and finally sum the timings for all epochs of all  $K$  models (as if they were trained serially). The clock for each model starts only when it is under execution. Among variant 1 and 2, the final training time depends mainly on the value of  $K$  sampled, though
variant 1 trains the final model with slightly smaller data. The baseline method takes much more time to run compared to our methods on weaker GPUs, which explains the differences in speed gains.
(d) trained with DP-SGD
Figure 5: Tuning only the learning rate with DP-SGD. Test accuracies are averaged across 10 independent runs and the error bars denote the standard error of the mean. The number in the legends in the first column refers to the scaled mean training timings for the baseline method with respect to the fastest of variant 1 and 2. The second column plots final  $\varepsilon$  vs. mean  $\sigma$ . Our methods inject significantly smaller noise compared to the baseline for all  $\varepsilon$  regimes. We also observe that due to tight analysis in Thm 6,  $\sigma$  for variant 1 is consistently lower than for variant 2. As a result, we see slightly higher accuracy for variant 1 in many cases. The third column plots final  $\varepsilon$  vs. mean optimal  $\eta$ . Note that due to randomness in the candidate selection process, optimal  $\eta$ 's for all three methods need not be the same. For perspective, we also add curves showing the privacy cost of training a single model with optimal hyperparameters obtained from the baseline.