SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The authors would like to thank Yaxi Hu and anonymous reviewers for helpful discussions on drafts of this paper. This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043), and the Swiss National Science Foundation (SNSF) under grant number 200021_205011, Google PDPO faculty research award, Intel within the www(private-ai.org center, Meta faculty research award, the NUS Early Career Research Award (NUS ECRA award number NUS ECRA FY19 P16), and the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
[1] Entropy, Relative Entropy, and Mutual Information, chapter 2, pages 13-55. John Wiley & Sons, Ltd, 2005. ISBN 9780471748823. doi: https://doi.org/10.1002/047174882X.ch2. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/047174882X.ch2.
[2] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[3] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
[4] Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pages 322-332. PMLR, 2019.
[5] Hilal Asi, Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: Optimal rates in 11 geometry. In International Conference on Machine Learning, pages 393-403. PMLR, 2021.
[6] Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in neural information processing systems, 31, 2018.
[7] Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1138-1156. IEEE, 2022.
[8] Rina Foygel Barber and John C Duchi. Privacy and statistical risk: Formalisms and minimax bounds. arXiv preprint arXiv:1412.4451, 2014.
[9] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464-473. IEEE, 2014.
[10] Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 1046-1059, 2016.
[11] Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. Advances in neural information processing systems, 32, 2019.
[12] Raef Bassily, Cristóbal Guzmán, and Michael Menart. Differentially private stochastic optimization: New results in convex and non-convex settings. Advances in Neural Information Processing Systems, 34:9317–9329, 2021.
[13] Raef Bassily, Mehryar Mohri, and Ananda Theertha Suresh. Differentially private learning with margin guarantees. arXiv preprint arXiv:2204.10376, 2022.
[49] Leonida Tonelli. Sull'integrazione per parti. Rend. Acc. Naz. Lincei, 5(18):246-253, 1909.
[50] Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Covariate shift in high-dimensional random feature regression. arXiv preprint arXiv:2111.08234, 2021.
[51] Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3797-3820, 2014.
[52] Santosh Vempala and Andre Wibisono. Rapid convergence of the unadjusted Langevin algorithm: Isoperimetry suffices. Advances in neural information processing systems, 32, 2019.
[53] Yu-Xiang Wang, Jing Lei, and Stephen E Fienberg. On-average kl-privacy and its equivalence to generalization for max-entropy mechanisms. In Privacy in Statistical Databases: UNESCO Chair in Data Privacy, International Conference, PSD 2016, Dubrovnik, Croatia, September 14–16, 2016, Proceedings, pages 121–134. Springer, 2016.
[54] Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, and Volkan Cevher. Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization). arXiv preprint arXiv:2209.07263, 2022.
[55] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes overparameterized deep relu networks. Machine learning, 109:467-492, 2020.
1 Introduction 1
1.1 Related Works 3
2 Problem and Methodology 3
2.1 Our objective and methodology 4
3 KL Privacy for Training Fully Connected ReLU Neural Networks 5
4 KL privacy bound for Linearized Network under over-parameterization 6
5 Numerical validation of our KL privacy bounds 7
6 Utility guarantees for Training Linearized Network 9
7 Conclusion 10
A Symbols and definitions 15
A.1 Additional notations 15
A.2 Relation between KL privacy Definition 2.3 to differential privacy 16
B Deferred proofs for Section 3 16
B.1 Deferred proofs for Theorem 3.1 16
B.2 Deferred proofs for Lemma 3.2 18
C Deferred proofs for Section 4 20
C.1 Bounding the gradient norm at initialization 20
C.2 Deferred proof for Theorem 4.1 22
D Deferred proofs for Section 6 23
D.1 Deferred proof for Lemma 6.1 on convergence of training linearized network 23
D.2 Deferred proof for Lemma 6.3 24
D.3 Deferred proof for Corollary 6.4 27
E Discussion on extending our results to Noisy GD with constant step-size 27
A Symbols and definitions
A.1 Additional notations
**Vecorization  $\operatorname{Vec}(\cdot)$ ** denotes the transformation that takes an input matrix  $A = (a_{ij})_{i \in [r], j \in [c]} \in \mathbb{R}^{r \times c}$  (with  $r$  rows and  $c$  columns) and outputs a  $rc$ -dimensional column vector:  $\operatorname{Vec}(A) = (a_{1,1}, \dots, a_{r,1}, a_{1,2}, \dots, a_{r,2}, \dots, a_{1,c}, \dots, a_{r,c})^{\top}$ .
Softmax function:  $\mathrm{softmax}(\pmb {y}) = \frac{e^{\pmb{y}^{[j]}}}{\sum_{j = 1}^{o}e^{\pmb{y}^{[j]}}}$  where  $o$  is the number of output classes.
$o$ : number of output classes for the neural network.
A.2 Relation between KL privacy Definition 2.3 to differential privacy
KL privacy is a more relaxed, yet closely connected privacy notion to  $(\varepsilon, \delta)$  differential privacy [22].
1. KL privacy and differential privacy are both worst-case privacy notions over all possible neighboring datasets, by requiring bounded distinguishability between the algorithm's output distributions on neighboring datasets in statistical divergence. The difference is that KL privacy requires bounded KL divergence, while  $(\varepsilon, \delta)$ -differential privacy is equivalent to bounded Hockey-stick divergence [6].
2. KL privacy and differential privacy are both definitions based on the privacy loss random variable  $\log \frac{P(A(D) = o)}{P(A(D') = o)}$ ,  $o \sim A(D)$  (following the definition in [2, Equation 1]). KL privacy implies that the privacy loss random variable has a bounded first order moment, while differential privacy requires a high probability argument that the privacy loss random variable is bounded by  $\varepsilon$  with probability  $1 - \delta$ . Therefore, KL privacy is generally a more relaxed notion than differential privacy.
3. Translation to each other: For  $\varepsilon = 0$ , KL privacy (bounded first-order moment of privacy loss random variable) implies  $(0, \delta)$ -differential privacy with  $\delta = \sqrt{\frac{\varepsilon}{2}}$  by Pinsker inequality. Higher order moments of the privacy loss random variable suffice to prove  $(\varepsilon, \delta)$ -differential privacy for  $\varepsilon > 0$ . Note that  $(\varepsilon, \delta)$ -DP with  $\delta > 0$  does not necessarily imply KL privacy, as the privacy loss random variable may be large at the tail event with  $\delta$  probability.
4. Due to the connection to the privacy loss random variable (which is closely connected to the likelihood ratio test for membership hypothesis testing), both differential privacy and KL privacy incur upper bound on the performance curve of inference attacks, such as the membership inference and attribute inference [39, 28], as we discuss in Footnote 3.
B Deferred proofs for Section 3
B.1 Deferred proofs for Theorem 3.1
To prove the new composition theorem, we will use the Girsanov's Theorem. Here we follow the presentation of [18, Theorem 6].
Theorem B.1 (Implication of Girsanov's theorem [18, Theorem 6]). Let  $(X_{t})_{t\in [0,\eta ]}$  and  $(\tilde{X}_t)_{t\in [0,\eta ]}$  be two continuous-time processes over  $\mathbb{R}^r$ . Let  $P_T$  be the probability measure that corresponds to the trajectory of  $(X_{t})_{t\in [0,\eta ]}$ , and let  $Q_{T}$  be the probability measure that corresponds to the trajectory of  $(\tilde{X}_{t})_{t\in [0,\eta ]}$ . Suppose that the process  $(X_{t})_{t\in [0,\eta ]}$  follows
$$
\mathrm {d} X _ {t} = b _ {t} \mathrm {d} t + \sigma \mathrm {d} B _ {t},
$$
where  $(B_{t})_{t\in [0,T]}$  is a standard Brownian motion over  $P_T$ , and the process  $(\hat{X}_t)_{t\in [0,\eta ]}$  follows
$$
\mathrm {d} \tilde {X} _ {t} = \tilde {b} _ {t} \mathrm {d} t + \sigma \mathrm {d} \tilde {B} _ {t},
$$
where  $(\tilde{B}_t)_{t\in [0,T]}$  is a standard Brownian motion over  $Q_{T}$  with  $\mathrm{d}\tilde{B}_t = \mathrm{d}B_t + \frac{1}{\sigma}\left(b_t - \tilde{b}_t\right)$ . Assume that  $\sigma$  is a  $r\times r$  symmetric positive definite matrix. Then, provided that Novikov's condition holds,
$$
\mathbb {E} _ {Q _ {T}} \exp \left(\frac {1}{2} \int_ {0} ^ {\eta} \| \sigma^ {- 1} \left(b _ {t} - \tilde {b} _ {t}\right) \| _ {2} ^ {2} \mathrm {d} t\right) <   \infty , \tag {12}
$$
$$
\frac {\mathrm {d} P _ {T}}{\mathrm {d} Q _ {T}} = \exp \left(\int_ {0} ^ {\eta} \sigma^ {- 1} (b _ {t} - \tilde {b} _ {t}) d \tilde {B} _ {t} - \frac {1}{2} \int_ {0} ^ {\eta} \| \sigma^ {- 1} (b _ {t} - \tilde {b} _ {t}) \| _ {2} ^ {2} \mathrm {d} t\right).
$$
We are now ready to apply Girsanov's theorem to prove the following new KL privacy composition theorem for Langevin diffusion processes on neighboring datasets  $\mathcal{D}$  and  $\mathcal{D}'$ . For ease of description, we repeat Theorem 3.1 as below.
Theorem 3.1. [KL composition under possibly unbounded gradient difference] The KL divergence between running Langevin diffusion (4) for DNN (2) on neighboring datasets  $\mathcal{D}$  and  $\mathcal{D}'$  satisfies
$$
\operatorname {K L} \left(\boldsymbol {W} _ {[ 0: T ]} \| \boldsymbol {W} _ {[ 0: T ]} ^ {\prime}\right) = \frac {1}{2 \sigma^ {2}} \int_ {0} ^ {T} \mathbb {E} \left[ \| \nabla \mathcal {L} \left(\boldsymbol {W} _ {t}; \mathcal {D}\right) - \nabla \mathcal {L} \left(\boldsymbol {W} _ {t}; \mathcal {D} ^ {\prime}\right) \| _ {2} ^ {2} \right] \mathrm {d} t. \tag {13}
$$
Proof. Recall that  $\mathbf{W}_t$  denotes the model parameter after running Langevin diffusion on dataset  $\mathcal{D}$  with time  $t$ , and  $\mathbf{W}_t'$  denotes the model parameter after running Langevin diffusion on dataset  $\mathcal{D}'$  with time  $t$ . To avoid confusion, we further denote  $p_{[t_1:t_2]}$  and  $p_{[t_1:t_2]}'$  as the distributions of model parameters trajectories  $\mathbf{W}_{[t_1:t_2]}$  and  $\mathbf{W}_{[t_1:t_2]}'$  during time interval  $[t_1,t_2]$  respectively. By definition of partial derivative, we have that
$$
\frac {\partial \mathrm {K L} \left(\boldsymbol {W} _ {[ 0 : t ]} \| \boldsymbol {W} _ {[ 0 : t ]} ^ {\prime}\right)}{\partial t} = \lim  _ {\eta \rightarrow 0} \frac {\mathrm {K L} \left(\boldsymbol {W} _ {[ 0 : t + \eta ]} \| \boldsymbol {W} _ {[ 0 : t + \eta ]} ^ {\prime}\right) - \mathrm {K L} \left(\boldsymbol {W} _ {[ 0 : t ]} \| \boldsymbol {W} _ {[ 0 : t ]} ^ {\prime}\right)}{\eta}. \tag {14}
$$
Now we compute the term  $\mathrm{KL}(\pmb{W}_{[0:t + \eta]}\| \pmb{W}_{[0:t + \eta]}^{\prime})$  as follows.
$$
\begin{array}{l} \operatorname {K L} \left(\boldsymbol {W} _ {[ 0: t + \eta ]} \| \boldsymbol {W} _ {[ 0: t + \eta ]} ^ {\prime}\right) = \mathbb {E} _ {w _ {[ 0: t + \eta ]} \sim p _ {[ 0: t + \eta ]}} \left[ \log \left(\frac {p _ {[ t : t + \eta ] | [ 0 : t ]} \left(w _ {[ t : t + \eta ]} \mid w _ {[ 0 : t ]}\right) p _ {[ 0 : t ]} (w _ {[ 0 : t ]})}{p _ {[ t : t + \eta ] | [ 0 : t ]} ^ {\prime} \left(w _ {[ t : t + \eta ]} \mid w _ {[ 0 : t ]}\right) p _ {[ 0 : t ]} ^ {\prime} (w _ {[ 0 : t ]})}\right) \right] \\ = \mathbb {E} _ {w _ {[ 0: t + \eta ]} \sim p _ {[ 0: t + \eta ]}} \left[ \log \left(\frac {p _ {[ t : t + \eta ] | [ 0 : t ]} \left(w _ {[ t : t + \eta ]} \mid w _ {[ 0 : t ]}\right)}{p _ {[ t : t + \eta ] | [ 0 : t ]} ^ {\prime} \left(w _ {[ t : t + \eta ]} \mid w _ {[ 0 : t ]}\right)}\right) + \log \left(\frac {p _ {[ 0 : t ]} \left(w _ {[ 0 : t ]}\right)}{p _ {[ 0 : t ]} ^ {\prime} , \left(w _ {[ 0 : t ]}\right)}\right) \right] \tag {15} \\ \end{array}
$$
where  $p_{[t:t + \eta] | [0:t]}(\cdot | w_{[0:t]})$  is the conditional distribution during running Langevin diffusion on dataset  $\mathcal{D}$ , given fixed values for model parameters trajectory  $W_{[0:t]} = w_{[0:t]}$ . Similarly,  $p_{[t:t + \eta] | [0:t]}'(\cdot | w_{[0:t]})$  is the conditional distribution during running Langevin diffusion on dataset  $\mathcal{D}'$ . Therefore by using the Markov property of the Langevin diffusion process and the definition of KL divergence in Eq. (15), we have that
$$
\mathrm {K L} \left(\boldsymbol {W} _ {[ 0: t + \eta ]} \| \boldsymbol {W} _ {[ 0: t + \eta ]} ^ {\prime}\right) = \mathbb {E} _ {w _ {t} \sim p _ {t}} \left[ \mathrm {K L} \left(p _ {[ t: t + \eta ] | t} (\cdot | w _ {t}) \| p _ {[ t: t + \eta ] | t} ^ {\prime} (\cdot | w _ {t})\right) \right] + \mathrm {K L} \left(p _ {t}, p _ {t} ^ {\prime}\right). \tag {16}
$$
Now to compute the term KL  $\left(p_{[t:t + \eta ]|t}(\cdot |w_t)\bigg\| p_{[t:t + \eta ]|t}'(\cdot |w_t)\right)$ , we only need to apply Theorem B.1 to the following two Langevin diffusion processes  $(\pmb{W}_{t + s|t})_{s\in [0,\eta ]}$  and  $(\pmb{W}_{t + s|t}')_{s\in [0,\eta ]}$ , conditioning on the observation  $\pmb{W}_{t|t} = \pmb{W}_{t|t}' = w_t$  at time  $t$ .
$$
\mathrm {d} \boldsymbol {W} _ {t + s | s} = - \nabla \mathcal {L} (\boldsymbol {W} _ {t + s | t}; \mathcal {D}) \mathrm {d} t + \sqrt {2 \sigma^ {2}} \mathrm {d} B _ {s}.
$$
$$
\mathrm {d} \pmb {W} _ {t + s | t} ^ {\prime} = - \nabla \mathcal {L} (\pmb {W} _ {t + s | t} ^ {\prime}; \mathcal {D} ^ {\prime}) \mathrm {d} t + \sqrt {2 \sigma^ {2}} \mathrm {d} \tilde {B} _ {s}.
$$
Note that when  $\eta$  is small enough, we have that the Novikov's condition in Eq. (12) holds because the exponent inside integration  $\frac{1}{2}\int_0^\eta \| \sigma_s^{-1}(b_s - \tilde{b}_s)\| _2^2\mathrm{d}s$  scales linearly with  $\eta$  that can be arbitrarily small. Therefore, by applying Girsanov's theorem, we have that
$$
\mathrm {K L} \left(p _ {[ t: t + \eta ] | t} (\cdot | w _ {t}) \Big \| p _ {[ t: t + \eta ] | t} ^ {\prime} (\cdot | w _ {t})\right) = \mathbb {E} \left[ \int_ {0} ^ {\eta} \sigma^ {- 1} (b _ {s} - \tilde {b} _ {s}) \mathrm {d} \tilde {B} _ {s} - \frac {1}{2} \int_ {0} ^ {T} \| \sigma^ {- 1} (b _ {s} - \tilde {b} _ {s}) \| _ {2} ^ {2} \mathrm {d} s \right],
$$
where  $b_{s} - \tilde{b}_{s} = -\nabla \mathcal{L}(\boldsymbol{W}_{t + s|t};\mathcal{D}) + \nabla \mathcal{L}(\boldsymbol{W}_{t + s|t};\mathcal{D}^{\prime})$ . By  $d\tilde{B}_s = \mathrm{d}B_s + \frac{1}{\sigma}\left(b_s - \tilde{b}_s\right)$  and Itô integration with regard to  $W_{[t:t + \eta ]|t}$ , we have that
$$
\operatorname {K L} \left(p _ {[ t: t + \eta ] | t} (\cdot | w _ {t}) \| p _ {[ t: t + \eta ] | t} ^ {\prime} (\cdot | w _ {t})\right) = \frac {\mathbb {E} \left[ \int_ {0} ^ {\eta} \| \nabla \mathcal {L} \left(\boldsymbol {W} _ {t + s | t} ; \mathcal {D}\right) - \nabla \mathcal {L} \left(\boldsymbol {W} _ {t + s | t} ; \mathcal {D} ^ {\prime}\right) \| _ {2} ^ {2} \mathrm {d} s \right]}{2 \sigma^ {2}}. \tag {17}
$$
By plugging Eq. (17) into Eq. (16), we have that
$$
\operatorname {K L} \left(p _ {[ t: t + \eta ]}, p _ {[ t: t + \eta ]} ^ {\prime}\right) = \frac {\mathbb {E} \left[ \int_ {0} ^ {\eta} \| \nabla \mathcal {L} \left(\boldsymbol {W} _ {t + s} ; \mathcal {D}\right) - \nabla \mathcal {L} \left(\boldsymbol {W} _ {t + s} ; \mathcal {D} ^ {\prime}\right) \| _ {2} ^ {2} \mathrm {d} s \right]}{2 \sigma^ {2}} + \operatorname {K L} \left(p _ {t}, p _ {t} ^ {\prime}\right). \tag {18}
$$
By plugging Eq. (18) into Eq. (14), and by exchanging the order of expectation and integration, we have that
$$
\begin{array}{l} \frac {\partial \mathrm {K L} (p _ {t} , p _ {t} ^ {\prime})}{\partial t} = \frac {1}{2 \sigma^ {2}} \lim  _ {\eta \rightarrow 0} \frac {\int_ {0} ^ {\eta} \mathbb {E} _ {p _ {t + s}} \left[ \| \nabla \mathcal {L} (\boldsymbol {W} _ {t + s} ; \mathcal {D}) - \nabla \mathcal {L} (\boldsymbol {W} _ {t + s} ; \mathcal {D} ^ {\prime}) \| _ {2} ^ {2} \right] d s}{\eta} \\ = \frac {1}{2 \sigma^ {2}} \mathbb {E} _ {p _ {t}} \left[ \| \nabla \mathcal {L} \left(\boldsymbol {W} _ {t}; \mathcal {D}\right) - \nabla \mathcal {L} \left(\boldsymbol {W} _ {t}; \mathcal {D} ^ {\prime}\right) \| _ {2} ^ {2} \right]. \tag {19} \\ \end{array}
$$
Note that here we exchange the order of expectation and integration by using the Tonelli's Theorem [49] for non-negative integrand function  $\| \nabla \mathcal{L}(\boldsymbol{W}_{t + s};\mathcal{D}) - \nabla \mathcal{L}(\boldsymbol{W}_{t + s};\mathcal{D}')\| _2^2$ . Integrating Eq. (19) on  $t\in [0,T]$  finishes the proof.
B.2 Deferred proofs for Lemma 3.2
Lemma 3.2. Let  $M_T$  be the subspace spanned by gradients  $\{\nabla \ell(\pmb{f}_{\pmb{W}_t}(\pmb{x}_i; \pmb{y}_i) : (\pmb{x}_i, \pmb{y}_i) \in \mathcal{D}, t \in [0, T]\}_{i=1}^n$  throughout Langevin diffusion  $(\pmb{W}_t)_{t \in [0, T]}$ . Denote  $\|\cdot\|_{M_T}$  as the  $\ell_2$  norm of the projected input vector onto  $M_T$ . Suppose that there exists constants  $c, \beta > 0$  such that for any  $\pmb{W}$ ,  $\pmb{W}'$  and  $(\pmb{x}, \pmb{y})$ , we have  $\|\nabla \ell(f_{\pmb{W}}(\pmb{x}); \pmb{y})) - \nabla \ell(f_{\pmb{W}'}(x); \pmb{y})\|_2 < \max\{c, \beta \| \pmb{W} - \pmb{W}'\|_{M_T}\}$ . Then running Langevin diffusion Eq. (4) with Gaussian initialization distribution (5) satisfies  $\varepsilon$ -KL privacy with
$$
\varepsilon = \frac {\operatorname* {m a x} _ {\mathcal {D} , \mathcal {D} ^ {\prime}} \int_ {0} ^ {T} \mathbb {E} \left[ \left\| \nabla \mathcal {L} (\boldsymbol {W} _ {t} ; \mathcal {D}) - \nabla \mathcal {L} (\boldsymbol {W} _ {t} ; \mathcal {D} ^ {\prime}) \right\| _ {2} ^ {2} \right] \mathrm {d} t}{2 \sigma^ {2}} w h e r e
$$
$$
\begin{array}{l} \int_ {0} ^ {T} \mathbb {E} \left[ \| \nabla \mathcal {L} (\boldsymbol {W} _ {t}; \mathcal {D}) - \nabla \mathcal {L} (\boldsymbol {W} _ {t}; \mathcal {D} ^ {\prime}) \| _ {2} ^ {2} \right] \mathrm {d} t \leq 2 T \cdot \underbrace {\mathbb {E} \left[ \| \nabla \mathcal {L} (\boldsymbol {W} _ {0} ; \mathcal {D}) - \nabla \mathcal {L} (\boldsymbol {W} _ {0} ; \mathcal {D} ^ {\prime}) \| _ {2} ^ {2} \right]} _ {\text {g r a d i e n t d i f f e r e n c e a t i n i t i z a t i o n}} \\ + \underbrace {\frac {2 \beta^ {2}}{n ^ {2} (2 + \beta^ {2})} \left(\frac {e ^ {(2 + \beta^ {2}) T} - 1}{2 + \beta^ {2}} - T\right) \cdot \left(\mathbb {E} \left[ \| \nabla \mathcal {L} (\boldsymbol {W} _ {0} ; \mathcal {D}) \| _ {2} ^ {2} \right] + 2 \sigma^ {2} r a n k (M _ {T}) + c ^ {2}\right)} _ {\text {g r a d i e n t d i f f e r e n c e f l u c t u a t i o n d u r i n g t r a n s i n g}} + \underbrace {\frac {2 c ^ {2} T}{n ^ {2}}} _ {\text {n o n - s m o o t h n e s s}}. \\ \end{array}
$$
Proof. By definition of the neighboring datasets  $\mathcal{D}$  and  $\mathcal{D}'$ , and the definition of empirical risk in Eq. (1), we have that for any  $\bar{W}$ , it satisfies that
$$
\left\| \nabla \mathcal {L} (\boldsymbol {W}; \mathcal {D}) - \nabla \mathcal {L} (\boldsymbol {W}; \mathcal {D} ^ {\prime}) \right\| _ {2} ^ {2} = \frac {1}{n ^ {2}} \left\| \ell \left(f _ {\boldsymbol {W}} (\boldsymbol {x}); \boldsymbol {y}\right)\right) - \nabla \ell \left(f _ {\boldsymbol {W}} \left(\boldsymbol {x} ^ {\prime}\right); \boldsymbol {y} ^ {\prime}\right) \| _ {2} ^ {2}, \tag {20}
$$
where  $(\pmb{x},\pmb{y})$  and  $(\pmb{x}^{\prime},\pmb{y}^{\prime})$  are the differing records between neighboring datasets  $\mathcal{D}$  and  $\mathcal{D}'$ . By the assumption that  $\| \nabla \ell (f_{\pmb{W}}(\pmb {x});\pmb {y})) - \nabla \ell (f_{\pmb{W}'}(\pmb {x});\pmb {y})\| _2 < \max \{c,\beta \| \pmb {W} - \pmb{W}'\|_{M_T}\}$ , and by the Cauchy-Schwarz inequality, we further have that for any  $\pmb{W}$  and  $\pmb{W}_t$ , it satisfies that
$$
\begin{array}{l} \left\| \nabla \ell \left(f _ {\mathbf {W} _ {t}} (\boldsymbol {x}); \boldsymbol {y}\right) - \nabla \ell \left(f _ {\mathbf {W} _ {t}} \left(\boldsymbol {x} ^ {\prime}\right); \boldsymbol {y} ^ {\prime}\right) \right\| _ {2} ^ {2} \leq 2 \left\| \nabla \ell \left(f _ {\mathbf {W} _ {0}} (\boldsymbol {x}); \boldsymbol {y}\right)\right) - \nabla \ell \left(f _ {\mathbf {W} _ {0}} \left(\boldsymbol {x} ^ {\prime}\right); \boldsymbol {y} ^ {\prime}\right) \| _ {2} ^ {2} \\ + 2 \beta^ {2} \| \boldsymbol {W} _ {t} - \boldsymbol {W} _ {0} \| _ {M _ {T}} ^ {2} + 2 c ^ {2}. \tag {21} \\ \end{array}
$$
The first term  $\| \nabla \ell (f_{\mathbf{W}_0}(\pmb {x});\pmb {y})) - \nabla \ell (f_{\mathbf{W}_0}(\pmb{x}');\pmb{y}')\| _2^2$  is constant during training (as it only depends on the initialization). Therefore, we only need to bound the second term  $\| \pmb {W}_t - \pmb {W}_0\|_{M_T}^2$ . For brevity, we denote the function  $\Phi (\pmb {W}) = \| \pmb {W} - \pmb {W}_0\|_{M_T}^2$ . Recall our definition,  $p_t$  as the distribution of model parameters after running Langevin diffusion on dataset  $\mathcal{D}$  with time  $t$ , and  $p_t^\prime$  as the distribution of model parameters after running Langevin diffusion on dataset  $\mathcal{D}'$  with time  $t$ . Then we have that
$$
\frac {\partial}{\partial t} \mathbb {E} _ {p _ {t}} [ \Phi (\boldsymbol {W}) ] = \lim  _ {\eta \rightarrow 0} \frac {\mathbb {E} _ {p _ {t + \eta}} [ \Phi (\boldsymbol {W}) ] - \mathbb {E} _ {p _ {t}} [ \Phi (\boldsymbol {W}) ]}{\eta}. \tag {22}
$$
Denote  $\Gamma_s$  as the following random operator on model parameters  $\theta$ .
$$
\Gamma_ {s} (\boldsymbol {W}) = \theta - s \nabla \mathcal {L} (\boldsymbol {W}; \mathcal {D}) + \sqrt {2 \sigma^ {2} s} Z,
$$
where  $Z\sim \mathcal{N}(0,\mathbb{I})$  . We first claim that the following equation holds.
$$
\lim  _ {\eta \rightarrow 0} \frac {\mathbb {E} _ {p _ {t + \eta}} [ \Phi (\boldsymbol {W}) ] - \mathbb {E} _ {p _ {t}} [ \Phi (\Gamma_ {\eta} (\boldsymbol {W})) ]}{\eta} = 0. \tag {23}
$$
This is by using Euler-Maruyama discretization method to approximate the solution  $\mathbf{W}_t$  of SDE Eq. (4). More specifically, the approximation error  $\mathbb{E}_{p_{t + \eta}}[\Phi (\mathbf{W})] - \mathbb{E}_{p_t}[\Phi (\Gamma_\eta (\mathbf{W}))]$  is of size  $O(r\eta^2)$  for small  $\eta$ , where  $r$  is the dimension of  $\mathbf{W}$ .
Therefore, by plugging Eq. (23) into Eq. (22), we have that
$$
\frac {\partial}{\partial t} \mathbb {E} _ {p _ {t}} [ \Phi (\boldsymbol {W}) ] = \lim _ {\eta \to 0} \frac {\mathbb {E} _ {p _ {t}} [ \Phi (\Gamma_ {\eta} (\boldsymbol {W})) ] - \mathbb {E} _ {p _ {t}} [ \Phi (\boldsymbol {W}) ]}{\eta}.
$$
Recall that  $\nabla^2\Phi (\pmb {W})$  exists almost everywhere with regard to  $\pmb {W}\sim p_t$ . Therefore we could approximate the term  $\mathbb{E}_{p_t}[\Phi (\Gamma_\eta (\pmb {W});\mathcal{D},\mathcal{D}')]$  via its second-order Taylor expansion at  $\pmb{W}$  as follows.