SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section we discuss the (distinct) convergence analyses of PGD and Anti-PGD, and the suboptimal results derived by trying to apply the proof technique for one to the other. We focus on the convex setting for brevity, though these analyses can be directly extended to the non-convex setting.
PGD. The convergence of PGD (Example 3.1) is well-understood since it is a special case of SGD. One can show the following.
Proposition 4.4 (Adapted from Dekel et al. [9, Theorem 1]). Under Assumptions 4.1, 4.2 and 4.3, if  $\mathbf{B} = \mathbf{S}$  and  $\gamma < 1/2L$ , then the output of (7) satisfies
$$
\sum_ {t = 0} ^ {T} \frac {\mathbb {E} [ f (\mathbf {x} _ {t}) - f ^ {\star} ]}{T + 1} \leq \mathcal {O} \left(\frac {\| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \| ^ {2}}{\gamma T} + \gamma \sigma^ {2}\right). \tag {11}
$$
The proof follows from combining the update (9), standard facts about convex functions, and the fact that  $\gamma < 1/2L$ , to get the inequality
$$
\mathbb {E} _ {t} \left\| \mathbf {x} _ {t + 1} - \mathbf {x} ^ {\star} \right\| ^ {2} \leq \left\| \mathbf {x} _ {t} - \mathbf {x} ^ {\star} \right\| ^ {2} - \gamma \left(f (\mathbf {x} _ {t}) - f ^ {\star}\right) + \gamma^ {2} \sigma^ {2}.
$$
It is left to average over iterations  $0 \leq t \leq T$ .
Anti-PGD. For Anti-PGD (Example 3.2), one can show the following.
Proposition 4.5. Under Assumptions 4.1, 4.2 and 4.3, if  $\mathbf{B} = \mathbf{I}$  and  $\gamma < 1/2L$ , then the output of (7) satisfies
$$
\sum_ {t = 0} ^ {T} \frac {\mathbb {E} [ f (\mathbf {x} _ {t}) - f ^ {\star} ]}{T + 1} \leq \mathcal {O} \left(\frac {\| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \| ^ {2}}{\gamma T} + L \gamma^ {2} \sigma^ {2}\right) \tag {12}
$$
Since  $L\gamma < 1/2$ , the RHS of (12) is strictly smaller than the RHS of (11). While this result may be known, we were unable to find a reference, so we provide a complete proof in Appendix D. The proof utilizes perturbed iterate analysis [28]. We define a virtual sequence  $\{\tilde{\mathbf{x}}_t\}_{t=0}^T$  as follows:
$$
\tilde {\mathbf {x}} _ {t + 1} = \tilde {\mathbf {x}} _ {t} - \gamma \nabla f (\mathbf {x} _ {t}), \quad \tilde {\mathbf {x}} _ {0} = \mathbf {x} _ {0} \tag {13}
$$
The  $\tilde{\mathbf{x}}_t$  are the iterates of (7) when  $\mathbf{Z} = \mathbf{0}$ . We can then prove the following descent inequality:
$$
\left\| \tilde {\mathbf {x}} _ {t + 1} - \mathbf {x} ^ {\star} \right\| ^ {2} \leq \left\| \tilde {\mathbf {x}} _ {t} - \mathbf {x} ^ {\star} \right\| ^ {2} - \frac {\gamma}{2} (f (\mathbf {x} _ {t}) - f ^ {\star}) + 2 L \gamma \left\| \tilde {\mathbf {x}} _ {t} - \mathbf {x} _ {t} \right\| ^ {2}.
$$
Because of the anti-correlation in (10), the virtual iterates  $\tilde{\mathbf{x}}_t$  are close to the real iterates  $\mathbf{x}_t$ , as  $\mathbf{x}_t - \tilde{\mathbf{x}}_t = \gamma \mathbf{z}_t$ . Averaging over  $t$ , we recover (12). See Appendix D for details.
Tightness. The noise terms (those terms involving  $\sigma^2$ ) in (11), (12) are both tight. We show this in Appendix E on the objective  $f(\mathbf{x}) = (L / 2)\| \mathbf{x}\|^2$ .
Difficulties in a unified analysis. The proof techniques for PGD and Anti-PGD above are notably different, and as we explain in Appendix F, do not lead to favorable results when trying to use one of the two strategies to analyze both.
4.2 Main Results and Analytic Techniques
To unify the proof techniques above, we use a modified virtual sequence with restart iterations. For a parameter  $\tau = \tilde{\Theta}^{(1 / L_{\gamma})}$  (throughout,  $\tilde{\mathcal{O}}$  and  $\tilde{\Theta}$  hide poly-logarithmic factors), we define
$$
\tilde {\mathbf {x}} _ {t + 1} = \tilde {\mathbf {x}} _ {t} - \gamma \nabla f (\mathbf {x} _ {t}) \quad \text {i f} t + 1 \neq 0 \bmod \tau \tag {14}
$$
$$
\tilde {\mathbf {x}} _ {t + 1} = \mathbf {x} _ {t + 1} \quad \text {i f} t + 1 = 0 \bmod \tau .
$$
Similar to the virtual sequence in (13),  $\tilde{\mathbf{x}}_t$  incorporates only deterministic gradients  $\nabla f(\mathbf{x}_t)$ . However, every  $\tau$  iterations we reset  $\tilde{\mathbf{x}}_t$  to the real iterate  $\mathbf{x}_t$ . This allows us to control the divergence between the virtual sequence and the real sequence (enabling a tight analysis of PGD), while still capturing the convergence benefits of anti-correlated noise (enabling a tight analysis of Anti-PGD).
The parameter  $\tau$  is independent of  $\mathbf{B}$ , and depends only on the geometry of  $f$  and the stepsize  $\gamma$ . Using this machinery, we can prove convergence rates of (7) for any factorization  $\mathbf{S} = \mathbf{BC}$ . These rates involve  $\ell_2$  distances between the rows  $\mathbf{b}_t$  of the matrix  $\mathbf{B}$  (where  $\mathbf{b}_0 = \mathbf{0}$  for convenience).
Theorem 4.6 (non-convex). Suppose Assumptions 4.1 and 4.2 hold,  $\gamma \leq 1/4L$ , and  $\tau = 1/\gamma L$ . Then (7) produces iterates whose average error  $(T + 1)^{-1} \sum_{t=0}^{T} \mathbb{E} \left\| \nabla f(\mathbf{x}_t) \right\|^2$  is upper bounded by
$$
\mathcal {O} \left(\frac {(f (\mathbf {x} _ {0}) - f ^ {\star})}{\gamma T} + \frac {\sigma^ {2}}{T \tau} \times \left[ \frac {1}{\tau} \sum_ {t = 1} ^ {T} \left\| \mathbf {b} _ {t} - \mathbf {b} _ {\lfloor \frac {t}{\tau} \rfloor \tau} \right\| ^ {2} + \sum_ {t = 0 \mathrm {m o d} \tau} ^ {1 \leq t \leq T} \| \mathbf {b} _ {t} - \mathbf {b} _ {t - \tau} \| ^ {2} \right]\right).
$$
Theorem 4.7 (convex). Under Assumptions 4.1, 4.2, and 4.3, if  $\gamma \leq 1/4L$  and  $\tau = \tilde{\Theta}(1/\gamma L)$ , then (7) produces iterates with average error  $(T + 1)^{-1} \sum_{t=0}^{T} \mathbb{E}\left[f(\mathbf{x}_t) - f^\star\right]$  upper bounded by
$$
\tilde {\mathcal {O}} \left(\frac {\| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \| ^ {2}}{\gamma T} + \frac {\sigma^ {2}}{T L \tau} \times \left[ \frac {1}{\tau} \sum_ {t = 1} ^ {T} \left\| \mathbf {b} _ {t} - \mathbf {b} _ {\lfloor \frac {t}{\tau} \rfloor \tau} \right\| ^ {2} + \sum_ {\substack {1 \leq t \leq T \\ t = 0 \bmod \tau}} \| \mathbf {b} _ {t} - \mathbf {b} _ {t - \tau} \| ^ {2} + \left\| \mathbf {b} _ {\lfloor \frac {T}{\tau} \rfloor \tau} \right\| ^ {2} \right]\right).
$$
We give complete proofs in Appendix C. These convergence rates consist of two terms: The first term states how fast the function would converge in the absence of the noise. The second term, the noise term, is the focus of our paper, as it shows how the correlated noise  $\mathbf{BZ}$  affects convergence.
These rates involve only differences of rows of  $\mathbf{B}$  that are at most  $\tau$  iterations apart. Intuitively,  $\tau$  is a coarse indicator of whether an iterate  $\mathbf{x}_t$  is still sensitive to the noise injected at an iteration  $t' < t$ . If  $t > t' + \tau$ , then changes in the noise added at step  $t$  are effectively uncorrelated to iteration  $t'$ . As we detail in Appendix, applying Theorem 4.7 to the special cases in Examples 3.1, 3.2 recovers their tight convergence rates in (11), (12) correspondingly.
5 Finding Better Factorizations
We now draw on our results in Section 4 to develop better mechanisms for the MF-DP-FTRL framework. We modify the objective underlying the offline matrix factorization problem during the first stage of the MF-DP-FTRL workflow (Fig. 1). Specifically, observe that the noise term in Theorems 4.6 and 4.7 can be rewritten in matrix notation (up to multiplicative constants) as
$$
\left\| \mathbf {A} _ {\tau} \mathbf {B} \right\| _ {F} ^ {2} = \sum_ {t = 1} ^ {T} \left\| \boldsymbol {\lambda} _ {t} ^ {\top} \mathbf {B} \right\| ^ {2} = \sum_ {\substack {1 \leq t \leq T \\ t = 0 \bmod \tau}} \left\| \mathbf {b} _ {t} - \mathbf {b} _ {t - \tau} \right\| ^ {2} + \sum_ {\substack {1 \leq t \leq T \\ t \neq 0 \bmod \tau}} \left\| \frac {1}{\sqrt {\tau}} \left(\mathbf {b} _ {t} - \mathbf {b} _ {\lfloor \frac {t}{\tau} \rfloor \tau}\right) \right\| ^ {2} \tag{15}
$$
(a) Average gradient norm for varying learning rates.
Figure 2: Comparison of the average and last gradient norms for DP-MF and  $\mathrm{DP - MF^{+}}$  on a random non-strongly convex quadratic function with  $L = 10$ .
(b) Last gradient norm for vary-(c) Gradient norm over time for ing learning rates.  $\gamma = 10^{-2}$
where  $\pmb{\Lambda}_{\tau} = \left[\pmb{\lambda}_1^\top, \dots, \pmb{\lambda}_T^\top\right]^\top \in \mathbb{R}^{T \times T}$ , and we set the rows  $\pmb{\lambda}_t$  appropriately to select corresponding row differences of  $\mathbf{B}$  with either coefficient 1 or  $1 / \sqrt{\tau}$  depending on the index  $t$ . We give a precise definition of  $\pmb{\Lambda}_{\tau}$  and an explicit example when  $T = 12$ ,  $\tau = 3$  in Appendix B.
Recall that [10] minimize the Frobenius norm objective (6) based on their derived convergence bounds in (5). Since our derived convergence bounds are strictly tighter, we propose using Eq. (15) as the new objective function in (6). Intuitively, since  $\| \mathbf{\Lambda}_{\tau}\mathbf{B}\| _F^2$  is a better proxy for learning performance than  $\| \mathbf{B}\| _F^2$ , minimizing this quantity in the offline factorization problem should lead to ERM methods with better privacy-utility trade-offs.
We can solve our new offline matrix factorization problem in a straightforward manner. We can show that for  $\mathbf{A} = \mathbf{S}$ , we can solve this modified problem by first computing the solution  $\tilde{\mathbf{B}},\tilde{\mathbf{C}}$  using  $\mathrm{OPT}_F(\boldsymbol {\Lambda}_\tau \mathbf{A})$ . The solution to our modified objective is then  $\mathbf{C} = \tilde{\mathbf{C}},\mathbf{B} = \mathbf{AC}^{-1}$ . This implies we can use existing open-source solvers designed for (6) [51, 30, 10].
In this section, we evaluate the ERM performance of MF-DP-FTRL under different offline factorization objectives. We focus on the Frobenius norm objective (6), which we refer to as DP-MF [10, 7], and our modified objective (15), which we refer to as  $\mathrm{DP - MF^{+}}$ .
6.1 Validating Theoretical Results
We first validate our theoretical results above by comparing the convergence of DP-MF and  $\mathrm{DP - MF^{+}}$  on a random quadratic function that satisfies the assumptions of Theorem 4.7. Notably, we ensure the quadratic is not strongly convex. We treat  $\tau$  in (15) as a hyperparameter and tune it over a fixed grid. For complete details, please refer to Appendix H. We present the results in Fig. 2.
In Fig. 2(a) we plot  $\frac{1}{T}\sum_{t=0}^{T}\|\nabla f(\mathbf{x}_t)\|^2$ , as this quantity is proportional to the LHS of Theorem 4.7. For all learning rates, DP-MF $^+$  either matches or outperforms DP-MF. Moreover, the advantage of DP-MF $^+$  increases as the learning rate increases. This corresponds to our theory in Theorem 4.7. Indeed, the larger the stepsize  $\gamma$ , the smaller the optimal  $\tau$  (as  $\tau = \Theta(1/\gamma L)$ ), and the more often restarts are used in the analysis of Theorem 4.7.
Fig. 2(b) further depicts the last-iterate behaviours of DP-MF and  $\mathrm{DP - MF^{+}}$ , which is often more practically relevant. Interestingly, the last iterate behaviour is improved even in the cases where the average behaviour does not improve. Finally, in Fig. 2(c) we pick  $\gamma = 10^{-2}$ ,  $\tau = 50$  as the parameters for which both the average and the last-iterate behaviours are improved and plot the convergence curve over iterations.  $\mathrm{DP - MF^{+}}$  has regular oscillating behaviour, allowing it to achieve a good final-iterate performance. The period of these oscillations is exactly equal to  $\tau$ .
6.2 Practical DP Training Experiments
We now compare DP-MF,  $\mathrm{DP - MF^{+}}$ , and DP-SGD with privacy amplification [1] on the MNIST, CIFAR-10, and Stack Overflow datasets. We omit from comparison DP-FTRL [20] and DP-Fourier
(a) MNIST, 1 epoch
(b) MNIST, 16 epochs
Figure 3: Test set accuracy of various mechanisms on the MNIST and CIFAR-10 datasets.
(c) CIFAR-10, 20 epochs
[7] as these methods are strictly dominated by DP-MF. Unlike our theoretical analysis, we include clipping to derive formal  $(\varepsilon, \delta)$  privacy guarantees. To facilitate a fair comparison, we set  $\delta = 10^{-6}$  in all the settings, and compare against varying  $\varepsilon$ . We give complete experimental details in Appendix H
MNIST, logistic regression. We train for  $T = 2048$  iterations and either 1 or 16 epochs depending on the batch size, corresponding to a batch size of 29 and 469 respectively. We fix the clipping threshold at 1.0 and the learning rate at 0.5. We vary  $\tau$  in (15) over  $\{2,8,32,128,512,2048\}$ . The results are in Figs. 3(a) and 3(b). DP-MF+ improves monotonically with  $\tau$ , performing best when  $\tau = 2048 = T$ . For such  $\tau$ , DP-MF+ consistently out-performs DP-MF across all settings. Recall from (15) that this corresponds to the offline objective  $\| \boldsymbol{\Lambda}_T\mathbf{B}\| _F^2$  where  $\lambda_{ii} = 1 / \sqrt{T}$  for all  $i < T$  and  $\lambda_{TT} = 1$ . This objective strongly penalizes errors on the final iterate, which is the model used to compute test accuracy.
We also see that DP-MF $^{+}$  expands the number of settings in which we can beat DP-SGD. DP-MF only outperforms DP-SGD for sufficiently large  $\varepsilon$  ( $\varepsilon \geq 0.31$  for 1 epoch and  $\varepsilon \geq 31$  for 16 epochs). By contrast, DP-MF $^{+}$  outperforms DP-SGD in every setting except when  $\varepsilon = 0.01$  and 1 epoch. None of the mechanisms reached the accuracy levels obtained by the non-private baseline, even at  $\varepsilon = 100$ . We suspect this is due to the fact that we are using a fixed but aggressive clipping threshold of 1.0 across all experiments, which helps in the moderate privacy regime but hurts in very low privacy regime. Even though DP-MF $^{+}$  does not use privacy amplification, it outperforms DP-SGD, which uses privacy amplification. This is due to the efficient noise anti-correlations in DP-MF $^{+}$ . If amplification were not possible, performance of DP-SGD would degrade even further.
CIFAR-10, CNN. We follow the experimental setup from [7]. Specifically, we train all mechanisms for 20 epochs and  $T = 2000$  iterations, which corresponds to a batch size of  $500$ .<sup>3</sup> We tune the learning rate over a fixed grid. We fix  $\tau = T = 2000$  in DP-MF<sup>+</sup> as we found that worked best in the MNIST experiments. The results are given in Fig. 3(c). We see that DP-MF<sup>+</sup> ( $\tau = 2000$ ) offers a consistent improvement over DP-MF across all choices of  $\varepsilon$  considered. Both DP-MF and DP-MF<sup>+</sup> beat DP-SGD for  $\varepsilon > 4$ . This observation is consistent with prior work on DP-FTRL and DP-MF, where DP-SGD performs relatively better with smaller  $\varepsilon$  while DP-MF performs better with larger  $\varepsilon$ .
Stack Overflow, LSTM. In Appendix H, we compare DP-MF and  $\mathrm{DP - MF^{+}}$  on a federated learning task with user-level differential privacy. We do not compare DP-SGD on this task, as amplification techniques such as shuffling and subsampling are not possible in practical federated learning settings [20]. In this task, we train an LSTM network to do next-word prediction on the Stack Overflow dataset. To be consistent with the prior work [10] and to test if our proposed factorizations are compatible with the other types of workloads A from Eq. (2), we use momentum and learning rate decay. Our results are given in Table 2. We see that two methods perform comparably, verifying competitiveness of our method. Note that this task uses federated averaging [32] instead of gradient
In practice, one often trains small-scale models for many epochs, perhaps even using full-batch gradients, to improve the privacy/utility trade-off (at the cost of increased computation). We are interested in the relative performance for a fixed computation budget, so we train for a small number of epochs.
3While Choquette-Choo et al. [7] use momentum and learning rate decay, we omit the use of such techniques as they are orthogonal to our theoretical results.
descent. Developing offline factorization objectives specifically for federated learning remains an open problem.
In this work, we developed analytic techniques to study the convergence of gradient descent under linearly correlated noise that is motivated from a class of DP mechanisms. We derived tighter bounds than currently exist in the literature, and we use our novel theoretical understanding to design privacy mechanisms with improved convergence. Perhaps more importantly, our work highlights the wealth of stochastic optimization questions arising from recent advances in differentially private model training. As such, we distill and formalize various optimization problems arising from recent work on matrix mechanisms for DP. Our work raises a host of questions and open problems, including extending our analysis to include things such as clipping, shuffling, and momentum. Another key extension is to derive last-iterate convergence rates rather than average-iterate convergence rates, as in some settings it is only the final "released" model that needs formal privacy guarantees. Given the improved generalization properties of Anti-PGD [37], one could also investigate how to design more general linearly correlated noise mechanisms which improve both privacy and generalization.
The authors would like to thank Francesco D'Angelo, Nina Mainusch and Linara Adylova for their comments on the manuscript. The authors would also like to thank the reviewers for their helpful suggestions in improving the clarity of the writing.
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Oct 2016. doi: 10.1145/2976749.2978318. URL http://dx.doi.org/10.1145/2976749.2978318.
[2] Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper/2011/file/f0e52b27a7a5d6a1a87373ddbfa53dbe5-Paper.pdf.
[3] The TensorFlow Federated Authors. TensorFlow Federated Stack Overflow dataset, 2019. URL https://www.tensorflow.org/federated/api_docs/python/tff/simulation/datasets/stackoverflow/load_data.
[4] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464-473. IEEE, 2014.
[5] Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. Advances in neural information processing systems, 32, 2019.
[6] Sébastien Bubeck. Convex optimization: Algorithms and complexity. Found. Trends Mach. Learn., 8(3-4):231-357, nov 2015. ISSN 1935-8237. doi: 10.1561/2200000050. URL https://doi.org/10.1561/2200000050.
[7] Christopher A. Choquette-Choo, H. Brendan McMahan, Keith Rush, and Abhradeep Thakurta. Multi-epoch matrix factorization mechanisms for private machine learning, 2022. URL https://arxiv.org/abs/2211.06530.
[8] Rudrajit Das, Satyen Kale, Zheng Xu, Tong Zhang, and Sujay Sanghavi. Beyond uniform Lipschitz condition in differentially private optimization. arXiv preprint arXiv:2206.10713, 2022.
[40] Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22(3):400 - 407, 1951. doi: 10.1214/aoms/1177729586. URL https://doi.org/10.1214/aoms/1177729586.
[41] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In Annual Conference Computational Learning Theory, 2009.
[42] Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=S1g2JnRcFX.
[43] Sebastian U. Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for SGD with delayed gradients and compressed updates. J. Mach. Learn. Res., 21(1), jun 2022. ISSN 1532-4435.
[44] Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified SGD with memory. Advances in Neural Information Processing Systems, 31, 2018.
[45] Om Thakkar, Galen Andrew, and H. B. McMahan. Differentially private learning with adaptive clipping. In Advances in Neural Information Processing Systems, 2021.
[46] Hoang Tran and Ashok Cutkosky. Momentum aggregation for private non-convex erm, 2022.
[47] Harsh Vardhan and Sebastian U. Stich. Tackling benign nonconvexity with smoothing and stochastic gradients, 2022. URL https://arxiv.org/abs/2202.09052.
[48] Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems, 30, 2017.
[49] Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled Rényi differential privacy and analytical moments accountant. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1226–1235. PMLR, 2019.
[50] Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In International Conference on Machine Learning, pages 7184–7193. PMLR, 2019.
[51] Ganzhao Yuan, Yin Yang, Zhenjie Zhang, and Zhifeng Hao. Convex optimization for linear query processing under approximate differential privacy, 2016. URL https://arxiv.org/abs/1602.04302.
[52] Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. Advances in Neural Information Processing Systems, 33:5332-5344, 2020.
[53] Chulhee Yun, Shashank Rajput, and Suvrit Sra. Minibatch vs local SGD with shuffling: Tight convergence bounds and beyond. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=LdlwbBP2mlq.
[54] Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, and Tuo Zhao. Toward understanding the importance of noise in training neural networks. In International Conference on Machine Learning, pages 7594-7602. PMLR, 2019.
[55] Yuqing Zhu and Yu-Xiang Wang. Poisson subsampled Rényi differential privacy. In International Conference on Machine Learning, pages 7634-7642. PMLR, 2019.