SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compare  $\delta_{f\text{-DP}}$  obtained in Corollary 4.3 with the corresponding  $\delta$  derived from [23] using a fixed value of  $\epsilon_0 = 4.444$  and  $n = 10000$ . Notably,  $\delta_{f\text{-DP}}$  is significantly smaller than  $\delta$ .
Table 2: Comparisons with numerical results in [23]
δ	5 × 10-5	3 × 10-6	10-7	4 × 10-9	9 × 10-11
εf-DP (ours)	0.4	0.5	0.6	0.7	0.8
Numerical ε upper bound in [23]	1.014	1.085	ε0	ε0	ε0
Numerical ε lower bound	0.369	0.470	0.575	0.664	0.758
We compare  $\epsilon_{f\text{-DP}}$  obtained from Corollary 4.3 with the corresponding numerical upper bound  $\epsilon$  derived from [23] using a fixed value of  $\epsilon_0 = 4.444$  and  $n = 10000$ . For  $\delta < 10^{-7}$ , the bound in [23] fails as the assumption  $\epsilon_0 \leq \log \left(\frac{n}{8 \log(2 / \delta)} - 1\right)$  is violated while our theory removes this assumption and holds for all  $\epsilon_0$ . Moreover, we compare our theoretical upper bound with the empirical lower bound obtained by binary search in [23] which shows that our bound is near-optimal.
In summary, our non-asymptotic privacy bound for shuffled mechanisms outperforms Theorem 3.2 in [23]. This improvement is a result of our Proposition 4.1, which optimally refines Lemma A.4 in [23]. Besides Proposition 4.1, the remainder of our proof of Theorem 4.2 closely adheres to the methodology presented in [23]. Our near-optimal result is complicated due to its tightness. Thus, it is difficult to compare our result with the asymptotic bound in [23] analytically.
5 Privacy analysis of one-iteration DP-GD with random initialization
A significant challenge in the privacy analysis of the last-iteration model of DP-SGD lies in accounting for multiple randomization techniques used during iterations. This includes aspects like initialization, iterative steps, and sub-sampling. Since these techniques incorporate a mixture of random noise, the joint convexity of  $F$ -divergence becomes crucial in the privacy analysis of DP-SGD [45, 2]. Our Lemma [3.1] which provides a unified perspective on these convexity notations, has driven us to include it in the privacy analysis of DP-GD. Nevertheless, analyzing the trajectories from multi-step iterations remains complex. Therefore, our initial exploration is to investigate the effects of random initialization on a one-step iterate. It's noteworthy that in machine learning, training a deep neural network using (stochastic) gradient descent combined with random initialization is widely adopted [36]. The significance of random initialization in noisy gradient descent is also emphasized by [46] within the framework of Kullback-Leibler privacy.
Consider a dataset  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$  with  $x_i \in \mathbb{R}$  being the features and  $y_i \in \mathbb{R}$  being the labels. Let  $\ell(\theta, \mathcal{D})$  be a loss function and let  $g(\theta, \mathcal{D})$  be the gradient of  $\ell$  with respect to  $\theta$ . The output of one-step iteration of DP-GD initialized at  $\theta_0$  with step-size 1 is given by
$$
\theta (\mathcal {D}) = \theta_ {0} - \left(g \left(\theta_ {0}, \mathcal {D}\right) + \mathcal {N} \left(0, \sigma^ {2}\right)\right). \tag {1}
$$
In the setting of random initialization,  $\theta_0$  is chosen as a Gaussian random variable. Without loss of generality, we consider  $\theta_0 = I \sim \mathcal{N}(0,1)$  and rewrite  $\theta(\mathcal{D}) = s_I(\mathcal{D}) + \mathcal{N}(0,\sigma^2)$  with  $s_I(\mathcal{D}) = I - g(I,\mathcal{D})$ .  $\theta(\mathcal{D})$  is a Gaussian random variable when the initialization  $I$  is given, that is,  $\theta(\mathcal{D})|I = i \sim \mathcal{N}(s_i(\mathcal{D}),\sigma^2)$ . Thus, we can regard  $\theta(\mathcal{D})$  as an infinite mixture of Gaussian distributions with continuous Gaussian weights  $\{\varphi(i)\}_{i \in \mathbb{R}}$ , where  $\varphi$  is the pdf of  $I$  and the corresponding trade-off function  $T(\theta(\mathcal{D}_0),\theta(\mathcal{D}_1))$  can be bounded using the joint concavity.
For simplicity, we define  $\theta(\mathcal{D})|I$  as a random variable with a given initialization  $I$ . For two neighboring datasets  $\mathcal{D}_0$  and  $\mathcal{D}_1$ , it holds
$$
T \left(\left(\theta \left(\mathcal {D} _ {0}\right) \mid I, I\right), \left(\theta \left(\mathcal {D} _ {1}\right) \mid I, I\right)\right) = T \left(\left(X \mid I, I\right), \left(Y \mid I, I\right)\right)
$$
with  $X|I\sim \mathcal{N}(0,1)$  and  $Y|I\sim \mathcal{N}(\mu_I,1)$  for  $I\sim \mathcal{N}(0,1)$ , where  $\mu_{I} = (g(I,\mathcal{D}_{1}) - g(I,\mathcal{D}_{0})) / \sigma$
Theorem 5.1. Let  $\theta(\mathcal{D}_0)$  and  $\theta(\mathcal{D}_1)$  be defined in (I) for neighboring datasets  $\mathcal{D}_0$  and  $\mathcal{D}_1$ . Then, we have
$$
T \left(\theta \left(\mathcal {D} _ {0}\right), \theta \left(\mathcal {D} _ {1}\right)\right) (\alpha (t)) \geq \mathbb {E} _ {I} \left[ \Phi (- t _ {I} + \mu_ {I}) \cdot \mathbb {1} _ {[ \mu_ {I} \leq 0 ]} + \Phi (t _ {I} - \mu_ {I}) \cdot \mathbb {1} _ {[ \mu_ {I} > 0 ]} \right]
$$
with  $t_I = -\frac{t}{\mu_I} + \frac{\mu_I}{2}$  and  $\alpha(t) = \mathbb{E}_I\left[\Phi(t_I) \cdot \mathbb{1}_{[\mu_I \leq 0]} + \Phi(-t_I) \cdot \mathbb{1}_{[\mu_I > 0]}\right]$ . Here  $\Phi$  is the cumulative distribution function of  $\mathcal{N}(0,1)$  and the expectation is taken with respect to  $I$ .
Remark. Note that Theorem 5.1 is instance-based privacy guarantee as it relies on the datasets. To extend it to the worst case, we let  $\mu_I^{\mathrm{max}} = \max_{\mathcal{D}_0,\mathcal{D}_1}\{|g(I,\mathcal{D}_1^{\mathrm{max}}) - g(I,\mathcal{D}_0^{\mathrm{max}})| / \sigma \}$  be the sensitivity of the gradient with a given initialization  $I$ . As a result,  $\theta (\mathcal{D})$  output by one-step DP-GD is  $f$ -DP with  $f(\alpha (t)) = \mathbb{E}_I[\Phi (t_I^{\mathrm{max}} - \mu_I^{\mathrm{max}})]$ , where  $t_I^{\mathrm{max}} = -\frac{t}{\mu_I^{\mathrm{max}}} +\frac{\mu_I^{\mathrm{max}}}{2}$  and  $\alpha (t) = \mathbb{E}_I[\Phi (-t_I^{\mathrm{max}})]$ . The worst case trade-off function is bounded for strongly convex loss functions with a bounded data domain.
To numerically evaluate the trade-off function in Theorem 5.1 we consider an example  $\mathcal{D}_0 = \{(x_i, y_i)\}_{i=1}^n$  with  $y_i = ax_i$  and  $x_i^2 = 1$  for some constant  $a$  and we defined  $\mathcal{D}_1$  by removing an arbitrary element in  $\mathcal{D}_0$ . Moreover, we assume that  $\sigma = 1$ . Note that for this example without gradient clipping, the gradient is linear in  $I$  and  $\theta(\mathcal{D}_0)$  is the sum of two Gaussian random variables which is Gaussian. Thus, the trade-off function has a closed-form representation. In general, the output is non-Gaussian and we should adopt Theorem 5.1 For example, if we consider gradient clipping [11, 9] and replace  $g(\theta, \mathcal{D})$  by the clipped gradient
$$
g _ {c} (\theta , \mathcal {D}) = \sum_ {i = 1} ^ {n} \frac {g ^ {(i)} (\theta)}{\operatorname* {m a x} \left\{1 , \| g ^ {(i)} (\theta) \| _ {2} / c \right\}}, \qquad \text {w i t h} g ^ {(i)} (\theta) = (y _ {i} - \theta x _ {i}) (- x _ {i}),
$$
where the gradient of each data point  $g^{(i)}$  is cut off by some constant  $c > 0$ , then  $\mu_I^{\mathrm{max}}$  is given by
$$
\mu_ {I} ^ {\max } = \left\{ \begin{array}{c c} a - I, & | a - I | \leq c, \\ c, & a - I \geq c, \\ - c, & a - I \leq - c, \end{array} \right.
$$
which is not Gaussian. In this example  $g_{c}(\theta, \mathcal{D}) + \mathcal{N}(0,1)$  is considered as  $c$ -GDP if we disregard the effects of random initialization since the sensitivity of  $g_{c}$  is  $c$ .
We illustrate the trade-off function of Theorem 5.1 computed numerically in Figure 2 where we also compare it with  $c$ -GDP for  $a = 1$  and varying values of  $c$ . Overall, the figure suggests that random initialization can amplify the privacy of DP-GD, as our bounds outperform those of  $c$ -GDP, which does not take into account the randomness of initialization. Furthermore, we observe that as  $c$  increases, the amplification effect caused by random initialization becomes more significant, since the difference between  $T((X|I,I), (Y|I,I))$  and  $c$ -GDP also increases. This is reasonable, since the randomness resulting from initialization comes from  $I$  such that  $|a - I| \leq c$ , whereas for  $|a - I| > c$ ,  $\mu_I$  remains constant and no randomness is introduced. Thus, the random initialization introduces greater levels of randomness as  $c$  increases.
It is worth noting that in this example, without gradient clipping, we have  $\mu_I^{\mathrm{max}} = a - I$  and the dominate pair are two Gaussian distributions  $\mathcal{N}(0,1)$  and  $\mathcal{N}(0,2)$ . The Rényi DP fails to measure the privacy of initialization. In fact, it holds  $R_{\widetilde{\alpha}}(\mathcal{N}(0,1)\| \mathcal{N}(0,2)) = \infty$  for  $\widetilde{\alpha}$  large enough.
(a)  $c = 0.5$
(b)  $c = 2$
(c)  $c = 3$
Figure 2: Trade-off functions for linear models with  $a = 1$ .
6 Optimality of joint concavity and advanced joint concavity
In this section, we first explore the sufficient and necessary conditions under which Lemma 3.1 holds with equality. While Lemma 3.1 is generally not sharp, we introduce an  $f$ -DP analog of the advanced joint convexity of the hockey-stick divergence from [4], yielding tighter bounds in certain applications.
Recall the distributions  $P = (1 - w)P_{0} + wQ_{0}$  and  $Q = (1 - w)Q_{0} + wP_{0}$  that appear in the shuffled mechanisms. Bounding the trade-off function  $T(P,Q)$  directly using the joint concavity leads to a loose bound (cf., Figure 3b). For the scenarios where Lemma 3.1 is not tight, we introduce the  $f$ -DP analog of the advanced joint convexity of  $(\epsilon, \delta)$ -DP [4] that may lead to tighter bounds and we term it the "advanced joint concavity of trade-off functions".
The following proposition presents a necessary and sufficient condition for Lemma 3.1 to hold with equality.
Proposition 6.1. For  $m = 2$ , Lemma 3.1 holds with equality if and only if  $\frac{w_1p_1 + w_2p_2}{w_1q_1 + w_2q_2}(X) \stackrel{\mathbb{P}}{=} w_1\frac{p_1}{q_1}(X) + w_2\frac{p_2}{q_2}(X)$  with  $X \sim P_{\mathbf{w}}$ , where for  $p_i(X)/q_i(X) = 0/0$  and  $p_j(X)/q_j(X) \neq 0/0$  with  $i \neq j$ , we set  $p_i(X)/q_i(X) = p_j(X)/q_j(X)$ .
It is not difficult to see that  $P_0$  and  $Q_0$  in shuffling models satisfy this necessary and sufficient condition when  $n = 2$ .
As we discussed, Lemma 3.1 may not be sharp in general. The following lemma is about the advanced joint convexity of the hockey-stick divergence, which is a slight generalization of Theorem 2 in [4].
Lemma 6.2. For any non-negative  $\epsilon', \epsilon_0, \epsilon_1, \gamma$ , and  $\eta$  satisfying  $\exp(\epsilon') = (1 - w)\exp(\epsilon_0) + w\exp(\epsilon_1)$  and  $\exp(\epsilon_0)(1 - w)\gamma + \exp(\epsilon_1)w\eta = \exp(\epsilon')w$ , we have
$$
\begin{array}{l} H _ {e ^ {\epsilon^ {\prime}}} \left((1 - w) P _ {1} + w P _ {2} \| (1 - w) Q _ {1} + w Q _ {2}\right) \\ \leq (1 - w) H _ {e ^ {\epsilon_ {0}}} \left(P _ {1} \| (1 - \gamma) Q _ {1} + \gamma Q _ {2}\right) + w H _ {e ^ {\epsilon_ {1}}} \left(P _ {2} \| (1 - \eta) Q _ {1} + \eta Q _ {2}\right). \tag {2} \\ \end{array}
$$
Lemma 6.2 is reduced to the advanced joint convexity of the hockey-stick divergence in [4] when  $P_{1} = Q_{1}$ , by minimizing the right-hand-side of (2) with respect to  $\gamma, \eta, \epsilon_{0}$ , and  $\epsilon_{1}$ .
Recall the convex conjugate  $g^{*}$  of a function  $g$  defined by  $g^{*}(y) = \sup_{x}\{xy - g(x)\}$  and  $\mathcal{C}(f) = \min\{f, f^{-1}\}^{**}$  which is the symmetrization of  $f$ . We have the following advanced joint concavity of trade-off functions.
Lemma 6.3 (Advanced joint concavity). Suppose that  $T(P_i, Q_i)$  is symmetric for each  $i$ . Then, for  $0 \leq w \leq 1$ , we have
$$
\begin{array}{l} T \left(\left(1 - w\right) P _ {1} + w P _ {2}, (1 - w) Q _ {1} + w Q _ {2}\right) \\ \geq \mathcal {C} \left(\left((1 - w) (1 - \gamma) F _ {1, 1} ^ {*} + w (1 - \eta) F _ {2, 1} ^ {*} + (1 - w) \gamma F _ {1, 2} ^ {*} + w \eta F _ {2, 2} ^ {*}\right) ^ {*}\right) \\ \end{array}
$$
for arbitrary  $0 \leq \gamma < w < \eta \leq 1$ , where  $F_{i,j}(x)$  is given by  $F_{1,i}(x) := f_{1,i}\left(\frac{x(1 - w)(\eta - \gamma)}{(\eta - w)}\right)$ , and  $F_{2,i}(x) := f_{2,i}\left(\frac{xw(\eta - \gamma)}{(w - \gamma)}\right)$ , and the trade-off functions are defined as  $f_{i,j} = T(P_i, Q_j)$  for  $1 \leq i, j \leq 2$ . Moreover, for  $\gamma = \eta = w$ , it holds
$$
\begin{array}{l} T \left(\left(1 - w\right) P _ {1} + w P _ {2}, (1 - w) Q _ {1} + w Q _ {2}\right) \\ \geq \mathcal {C} \left((1 - w) T \left(P _ {1}, (1 - w) Q _ {1} + w Q _ {2}\right) + w T \left(P _ {2}, (1 - w) Q _ {1} + w Q _ {2}\right)\right). \\ \end{array}
$$
Determining the trade-off functions using advanced joint concavity can be challenging in many practical situations. In fact, to apply the advanced joint concavity, one need to specify the choice of  $\gamma$ ,  $\eta$  by maximizing the right-hand-side of Lemma 6.3. Therefore, in real-world applications, we often rely on both joint concavity and advanced joint concavity.
For  $P = (1 - w)P_{0} + wQ_{0}$  and  $Q = (1 - w)Q_{0} + wP_{0}$  in shuffling models, we have the following bound derived from Lemma 6.3
Proposition 6.4. For  $P = (1 - w)P_{0} + wQ_{0}$  and  $Q = (1 - w)Q_{0} + wP_{0}$  with some weight  $0 \leq w \leq 1/2$ , we have  $T(P, Q) \geq \mathcal{C}(2w\mathrm{Id} + (1 - 2w)T(P_{0}, Q_{0}))$ .
The equality in Proposition 6.4 does not hold exactly. However, this lower bound is almost the tightest closed-form expression. One may refer to Section E.1.1 in the appendix for the proof details.
This paper provides refined privacy bounds for mixture mechanisms, including shuffling models and DP-GD with random initialization. For shuffling models, we present a bound that is tighter than existing results based on  $(\epsilon, \delta)$ -DP. In the study of DP-GD, we demonstrate how random initialization can amplify privacy concerns. These bounds are derived using a unified  $f$ -DP approach based on the joint concavity and advanced joint concavity of trade-off functions. We also investigate the sharpness and other properties of these concavity notions.
In our future work, we plan to extend our analysis from one-step DP-GD to multi-step DP-SGD. For DP-SGD with multiple iterations, it is crucial to consider subsampling and privacy amplification by iteration in the privacy accountant, in addition to the randomness introduced by shuffling and random initialization. While there is an  $f$ -DP bound for subsampling provided in an independent work [42], as far as we know, there is limited research on  $f$ -DP results regarding privacy amplification by iteration.
Beyond DP-SGD, we intend to extend our theory to the privacy analysis of other key applications that involves various randomization techniques. These include the shuffled Gaussian mechanism for federated learning, as discussed in [25], and the composition of mixture mechanisms. For extending our theory to federated learning, we might adopt the  $f$ -DP framework outlined in [49]. Addressing the composition of mixture mechanisms demands examination of the tensor product between trade-off functions. This is a complex task, even when dealing with the simplest mixture mechanisms like sub-sampling, as highlighted in [51].
Weijie J. Su was supported in part by a Meta Research Award and NSF through CCF1934876.
Reza Shokri was supported by a Google PDPO Faculty Research Award, Intel within the www/privateai.org center, a Meta Faculty Research Award, the NUS Early Career Research Award (NUS ECRA award number NUS ECRA FY19 P16), and the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative.
[1] Martin Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Edgar R. Weippl, Stefan Katzenbeisser, Christopher Kruegel, Andrew C. Myers, and Shai Halevi, editors, Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016, pages 308-318. ACM, 2016.
[2] Jason M. Altschuler and Kunal Talwar. Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. In NeurIPS, 2022.
[3] Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive clipping. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 17455-17466, 2021.
[4] Borja Balle, Gilles Barthe, and Marco Gabordi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[5] Gilles Barthe and Federico Olmedo. Beyond differential privacy: Composition theorems and relational logic for f-divergences between probabilistic programs. In Automata, Languages, and Programming: 40th International Colloquium, ICALP 2013, Riga, Latvia, July 8-12, 2013, Proceedings, Part II 40, pages 49-60. Springer, 2013.
[6] Raef Bassily, Adam D. Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 55th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2014, Philadelphia, PA, USA, October 18-21, 2014, pages 464-473. IEEE Computer Society, 2014.
[43] Larry Wasserman and Shuheng Zhou. A statistical framework for differential privacy. J. Amer. Statist. Assoc., 105(489):375-389, 2010.
[44] Shirong Xu, Chendi Wang, Will Wei Sun, and Guang Cheng. Binary classification under local label differential privacy using randomized response mechanisms. Transactions on Machine Learning Research, 2023.
[45] Jiayuan Ye and Reza Shokri. Differentially private learning needs hidden state (or much faster convergence). In NeurIPS, 2022.
[46] Jiayuan Ye, Zhenyu Zhu, Fanghui Liu, Reza Shokri, and Volkan Cevher. Initialization matters: Privacy-utility analysis of overparameterized neural networks. In NeurIPS, 2023.
[47] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorch. CoRR, abs/2109.12298, 2021.
[48] Jun Zhang, Graham Cormode, Cecilia M. Procopiuč, Divesh Srivastava, and Xiaokui Xiao. PrivBayes: Private data release via Bayesian networks. ACM Trans. Database Syst., 42(4):25:1-25:41, 2017.
[49] Qinqing Zheng, Shuxiao Chen, Qi Long, and Weijie J. Su. Federated f-differential privacy. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pages 2251-2259. PMLR, 2021.
[50] Qinqing Zheng, Jinshuo Dong, Qi Long, and Weijie J. Su. Sharp composition bounds for Gaussian differential privacy via edgeworth expansion. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 11420-11435. PMLR, 2020.
[51] Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event, volume 151 of Proceedings of Machine Learning Research, pages 4782-4817. PMLR, 2022.
[52] Yuqing Zhu and Yu-Xiang Wang. Poisson subsampled Rényi differential privacy. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 7634-7642. PMLR, 2019.