SlowGuess/NeurIPS_2023_OCR huggingface dataset:
to bound the gradient difference between datasets  $\mathcal{D}$  and  $\mathcal{D}'$  at time  $T$ , via the relaxed smoothness assumption of loss function (that is explained in Remark 3.5 in details). The complete proof is in Appendix B.2.
Remark 3.3 (Gradient difference at initialization). The first term and in our upper bound linearly scales with the difference between gradients on neighboring datasets  $\mathcal{D}$  and  $\mathcal{D}'$  at initialization. Under different initialization schemes, this gradient difference exhibits different dependency on the network depth and width, as we will prove theoretically in Theorem 4.1.
Remark 3.4 (Gradient difference fluctuation during training). The second term in Lemma 3.2 bounds the change of gradient difference during training, and is proportional to the the rank of a subspace  $M_T$  spanned by gradients of all training data. Intuitively, this fluctuation is because Langevin diffusion adds per-dimensional noise with variance  $\sigma^2$ , thus perturbing the training parameters away from the initialization at a scale of  $O(\sigma \sqrt{\mathrm{rank}(M_T)})$  in the expected  $\ell_2$  distance.
Remark 3.5 (Relaxed smoothness of loss function). The third term in Lemma 3.2 is due to the assumption  $\| \nabla \ell (f_{\mathbf{W}}(\mathbf{x});\mathbf{y})) - \nabla \ell (f_{\mathbf{W}'}(\mathbf{x});\mathbf{y})\| _2 < \max \{c,\beta \| \mathbf{W} - \mathbf{W}'\|_{M_T}\}$ . This assumption is similar to smoothness of loss function, but is more relaxed as it allows non-smoothness at places where the gradient is bounded by  $c$ . Therefore, this assumption is general to cover commonly-used smooth, non-smooth activation functions, e.g., sigmoid, ReLU.
Growth of KL privacy bound with increasing training time  $T$ . The first and third terms in our upper bound Lemma 3.2 grow linearly with the training time  $T$ , while the second term grows exponentially with regard to  $T$ . Consequently, for learning tasks that require a long training time to converge, the second term will become the dominating term and the KL privacy bound suffers from exponential growth with regard to the training time. Nevertheless, observe that for small  $T \to 0$ , the second component in Lemma 3.2 contains a small factor  $\frac{e^{(2 + \beta^2)T} - 1}{2 + \beta^2} - T = o(T)$  by Taylor expansion. Therefore, for small training time, the second component is smaller than the first and the third components in Lemma 3.2 that linearly scale with  $T$ , and thus does not dominate the privacy bound. Intuitively, this phenomenon is related to lazy training [19]. In Section 5 and Figure 2, we also numerically validate that the second component does not have a high effect on the KL privacy loss in the case of small training time.
Dependence of KL privacy bound on network over-parameterization. Under a fixed training time  $T$  and noise scale  $\sigma^2$ , Lemma 3.2 predicts that the KL divergence upper bound in Theorem 3.1 is dependent on the gradient difference and gradient norm at initialization, and the rank of gradient subspace  $\mathrm{rank}(M_T)$  throughout training. We now discuss the how these two terms change under increasing width and depth, and whether there are possibilities to improve them under over-parameterization.
1. The gradient norm at initialization crucially depends on how the per-layer variance in the Gaussian initialization distribution scales with the network width and depth. Therefore, it is possible to reduce the gradient difference at initialization (and thus improve the KL privacy bound) by using specific initialization schemes, as we later show in Section 4 and Section 5.
2. Regarding the rank of gradient subspace  $\mathrm{rank}(M_T)$ : when the gradients along the training trajectory span the whole optimization space,  $\mathrm{rank}(M_T)$  would equal the dimension of the learning problem. Consequently, the gradient fluctuation upper bound (and thus the KL privacy bound) worsens with increasing number of model parameters (over-parameterization) in the worst-case. However, if the gradients are low-dimensional [45, 32, 43] or sparse [37],  $\mathrm{rank}(M_T)$  could be dimension-independent and thus enables better bound for gradient fluctuation (and KL privacy bound). We leave this as an interesting open problem.
4 KL privacy bound for Linearized Network under over-parameterization
In this section, we focus on the training of linearized networks (3), which fosters a refined analysis on the interplay between KL privacy and over-parameterization (increasing width and depth). Analysis of DNNs via linearization is a commonly used technique in both theory [19] and practice [43, 41]. We hope our analysis for linearized network serves as an initial attempt that would open a door to theoretically understanding the relationship between over-parameterization and privacy.
To derive a composition-based KL privacy bound for training a linearized network, we apply Theorem 3.1 which requires an upper bound for the norm of gradient difference between the training
processes on neighboring datasets  $\mathcal{D}$  and  $\mathcal{D}'$  at any time  $t$ . Note that the empirical risk function for training linearized models enjoys convexity, and thus a relatively short amount of training time is enough for convergence. In this case, intuitively, the gradient difference between neighboring datasets does not change a lot during training, which allows for a tighter upper bound for the gradient difference norm for linearized networks (than Lemma 3.2).
In the following theorem, we prove that for a linearized network, the gradient difference throughout training has a uniform upper bound that only depends on the network width, depth and initialization.
Theorem 4.1 (Gradient Difference throughout training linearized network). Under Assumption 2.2, taking over the randomness of the random initialization and the Brownian motion, for any  $t \in [0,T]$ , running Langevin diffusion on a linearized network in Eq. (3) satisfies that
$$
\mathbb {E} \left[ \| \nabla \mathcal {L} \left(\boldsymbol {W} _ {t} ^ {\text {l i n}}; \mathcal {D}\right) - \mathcal {L} \left(\boldsymbol {W} _ {t} ^ {\text {l i n}}; \mathcal {D} ^ {\prime}\right) \| _ {2} ^ {2} \right] \leq \frac {4 B}{n ^ {2}}, \text {w h e r e} B := d \cdot o \cdot \left(\prod_ {i = 1} ^ {L - 1} \frac {\beta_ {i} m _ {i}}{2}\right) \sum_ {l = 1} ^ {L} \frac {\beta_ {L}}{\beta_ {l}}, \tag {7}
$$
where  $n$  is the training dataset size, and  $B$  is a constant that only depends on the data dimension  $d$ , the number of classes  $o$ , the network depth  $L$ , the per-layer network width  $\{m_i\}_{i=1}^L$ , and the per-layer variances  $\{\beta_i\}_{i=1}^L$  of the Gaussian initialization distribution.
Theorem 4.1 provides a precise analytical upper bound for the gradient difference during training linearized network, by tracking the gradient distribution for fully connected feed-forward ReLU network with Gaussian weight matrices. Our proof borrows some techniques from [3, 54] for computing the gradient distribution, refer to Appendix C.1 and C.2 for the full proofs. By plugging Eq. (7) into Theorem 3.1, we obtain the following KL privacy bound for training a linearized network.
Corollary 4.2 (KL privacy bound for training linearized network). Under Assumption 2.2 and neural networks (3) initialized by Gaussian distribution with per-layer variance  $\{\beta_i\}_{i=1}^L$ , running Langevin diffusion for linearized network with time  $T$  on any neighboring datasets satisfies that
$$
\mathrm {K L} \left(\boldsymbol {W} _ {[ 0: T ]} ^ {\text {l i n}} \| \boldsymbol {W} _ {[ 0: T ]} ^ {\prime \text {l i n}}\right) \leq \frac {2 B T}{n ^ {2} \sigma^ {2}}, \tag {8}
$$
where  $B$  is the constant that specifies the gradient norm upper bound, given by Eq. (7).
Over-parameterization affects privacy differently under different initialization. Corollary 4.2 and Theorem 4.1 prove the role of over-parameterization in our KL privacy bound, crucially depending on how the per-layer Gaussian initialization variance  $\beta_{i}$  scales with the per-layer network width  $m_{i}$  and depth  $L$ . We summarize our KL privacy bound for the linearized network under different width, depth and initialization schemes in Table 1, and elaborate the comparison below.
(1) LeCun initialization uses small, width-independent variance for initializing the first layer  $\beta_{1} = \frac{1}{d}$  (where  $d$  is the number of input features), and width-dependent variance  $\beta_{2} = \dots = \beta_{L} = \frac{1}{m}$  for initializing all the subsequent layers. Therefore, the second term  $\sum_{l=1}^{L} \frac{\beta_{L}}{\beta_{l}}$  in the constant  $B$  of Eq. (7) increases linearly with the width  $m$  and depth  $L$ . However, due to  $\frac{m_{l} \cdot \beta_{l}}{2} < 1$  for all  $l = 2, \dots, L$ , the first product term  $\prod_{l=1}^{L-1} \frac{\beta_{l} m_{l}}{2}$  in constant  $B$  decays with the increasing depth. Therefore, by combining the two terms, we prove that the KL privacy bound worsens with increasing width, but improves with increasing depth (as long as the depth is large enough). Similarly, under Xavier initialization  $\beta_{l} = \frac{2}{m_{l-1} + m_{l}}$ , we prove that the KL privacy bound (especially the constant  $B$  (7)) improves with increasing depth as long as the depth is large enough.
(2) NTK and He initializations use large per-layer variance  $\beta_{l} = \left\{ \begin{array}{ll} \frac{2}{m_{l}} & l = 1, \dots, L - 1 \\ \frac{1}{o} & l = L \end{array} \right.$  (for NTK) and  $\beta_{l} = \frac{2}{m_{l-1}}$  (for He). Consequently, the gradient difference under NTK or He initialization is significantly larger than that under LeCun initialization. Specifically, the gradient norm constant  $B$  in Eq. (7) grows linearly with the width  $m$  and the depth  $L$  under He and NTK initializations, thus indicating a worsening of KL privacy bound under increasing width and depth.
5 Numerical validation of our KL privacy bounds
To understand the relation between privacy and over-parameterization in practical DNNs training (and to validate our KL privacy bounds Lemma 3.2 and Corollary 4.2), we perform experiments for
DNNs training via noisy GD to numerically estimate the KL privacy loss. We will show that if the total training time is small, it is indeed possible to obtain numerical KL privacy bound estimates that does not grow with the total number of parameter (under carefully chosen initialization distributions).
Numerical estimation procedure. Theorem 3.1 proves that the exact KL privacy loss scales with the expectation of squared gradient norm during training. This could be estimated by empirically average of gradient norm across training runs. For training dataset  $\mathcal{D}$ , we consider all 'car' and 'plane' images of the CIFAR-10. For neighboring dataset, we consider all possible  $\mathcal{D}'$  that removes a record from  $\mathcal{D}$ , or adds a test record to  $\mathcal{D}$ , i.e., the standard "add-or remove-one" neighboring notion [2]. We run noisy gradient descent with constant step-size 0.01 for 50 epochs on both datasets.
Numerically validate the growth of KL privacy loss with regard to training time. Figure 1 shows numerical KL privacy loss under different initializations, for fully connected networks with width 1024 and depth 10. We observe that the KL privacy loss grows linearly at the beginning of training ( $< 10$  epochs), which validates the first and third term in the KL privacy bound Lemma 3.2. Moreover, the KL privacy loss under LeCun and Xavier initialization is close to zero at the beginning of training ( $< 10$  epochs). This shows LeCun and Xavier initialization induce small gradient norm at small training time, which is consistent with Theorem 4.1. However, when the number of epochs is large, the numerical KL privacy loss grows faster than linear accumulation under all initializations, thus validating the second term in Lemma 3.2.
Figure 1: Numerically estimated KL privacy loss for noisy GD with constant step-size 0.001 on deep neural network with width 1024 and depth 10. We report the mean and standard deviation across 6 training runs, taking worst-case over all neighboring datasets. The numerical KL privacy loss grows with the number of training epochs under all initializations. The growth rate is close to linear at beginning of training (epochs  $< 10$ ) and is faster than linear at epochs  $\geq 10$
(a) Privacy vs depth (20 epochs,  $\sigma = 0.01$ , width  $= 1024$
(b) Privacy vs depth (50 epochs,  $\sigma = 0.01$  width  $= 1024$
(c) Privacy vs width (50 epochs,  $\sigma = 0.01$ , depth  $= 10$
Figure 2: Numerically estimated KL privacy loss for noisy GD with constant step-size on fully connected ReLU network with different width, depth and initializations. We report the mean and standard deviation across 6 training runs, taking worst-case over all neighboring datasets. Under increasing width, the KL privacy loss always grows under all evaluated initializations. Under increasing depth, at the beginning of training (20 epochs), the KL privacy loss worsens with depth under He initialization, but first worsens with depth  $(\leq 8)$  and then improves with depth  $(\geq 8)$  under Xavier and LeCun initializations. At later phases of the training (50 epochs), KL privacy worsens (increases) with depth under all evaluated initializations.
Numerically validate the dependency of KL privacy loss on network width, depth and initializations. Figure 2 shows the numerical KL privacy loss under different network depth, width and initializations, for a fixed training time. In Figure 2c, we observe that increasing width and training time always increases KL privacy loss. This is consistent with Theorem 4.1, which shows that increasing width worsens the gradient norm at initialization (given fixed depth), thus harming KL privacy bound Lemma 3.2 at the beginning of training. We also observe that the relationship between KL privacy
and network depth depends on the initialization distributions and the training time. Specifically, in Figure 2a, when the training time is small (20 epochs), for LeCun and Xavier initializations, the numerical KL privacy loss improves with increasing depth when depth  $>8$ . Meanwhile, when the training time is large (50 epochs) in Figure 2b, KL privacy loss worsens with increasing depth under all initializations. This shows that given small training time, the choice of initialization distribution affects the dependency of KL privacy loss on increasing depth, thus validating Lemma 3.2 and Theorem 4.1.
6 Utility guarantees for Training Linearized Network
Our privacy analysis suggests that training linearized network under certain initialization schemes (such as LeCun initialization) allows for significantly better privacy bounds under overparameterization by increasing depth. In this section, we further prove utility bounds for Langevin diffusion under initialization schemes and investigate the effect of over-parameterization on the privacy utility trade-off. In other words, we aim to understand whether there is any utility degradation for training linearized networks when using the more privacy-preserving initialization schemes.
Convergence of training linearized network. We now prove convergence of the excess empirical risk in training linearized network via Langevin diffusion. This is a well-studied problem in the literature for noisy gradient descent. We extend the convergence theorem to continuous-time Langevin diffusion below and investigate factors that affect the convergence under over-parameterization. The proof is deferred to Appendix D.1.
Lemma 6.1 (Extension of [42, Theorem 2] and [45, Theorem 3.1]). Let  $\mathcal{L}_0^{lin}(\pmb {W};\mathcal{D})$  be the empirical risk function of a linearized network in Eq. (3) expanded at initialization vector  $\pmb{W}_{0}^{lin}$ . Let  $\pmb{W}_{0}^{*}$  be an  $\alpha$ -near-optimal solution for the ERM problem such that  $\mathcal{L}_0^{lin}(\pmb{W}_0^*;\mathcal{D}) - \min_{\pmb{W}}\mathcal{L}_0^{lin}(\pmb {W};\mathcal{D})\leq \alpha$ . Let  $\mathcal{D} = \{\pmb {x}_i\}_{i = 1}^n$  be an arbitrary training dataset of size  $n$ , and denote  $M_0 = \left(\nabla f_{\pmb{W}_0^{lin}}(\pmb {x}_1),\dots ,\nabla f_{\pmb{W}_0^{lin}}(\pmb {x}_n)\right)^{\top}$  as the NTK feature matrix at initialization. Then running Langevin diffusion (4) on  $\mathcal{L}_0^{lin}(\pmb {W})$  with time  $T$  and initialization vector  $\pmb{W}_{0}^{lin}$  satisfies
$$
\mathbb {E} [ \mathcal {L} _ {0} ^ {l i n} (\bar {\boldsymbol {W}} _ {T} ^ {l i n}) ] - \min _ {\boldsymbol {W}} \mathcal {L} _ {0} ^ {l i n} (\boldsymbol {W}; \mathcal {D}) \leq \alpha + \frac {R}{2 T} + \frac {1}{2} \sigma^ {2} r a n k (M _ {0}),
$$
where the expectation is over Brownian motion  $B_{T}$  in Langevin diffusion in Eq. (4),  $\bar{\mathbf{W}}_T^{lin} = \frac{1}{T}\int \bar{\mathbf{W}}_t^{lin}\mathrm{d}t$  is the average of all iterates, and  $R = \| \pmb{W}_0^{lin} - \pmb{W}_0^*\|_{M_0}^2$  is the gap between initialization parameters  $\pmb{W}_0^{lin}$  and solution  $\pmb{W}_0^*$ .
Remark 6.2. The excess empirical risk bound in Lemma 6.1 is smaller if data is low-rank, e.g., image data, then  $\mathrm{rank}(M_0)$  is small. This is consistent with the prior dimension-independent private learning literature [32, 33, 37] and shows the benefit of low-dimensional gradients on private learning.
Lemma 6.1 highlights that the excess empirical risk scales with the gap  $R$  between initialization and solution (denoted as lazy training distance), the rank of the gradient subspace, and the constant  $B$  that specifies upper bound for expected gradient norm during training. Specifically, the smaller the lazy training distance  $R$  is, the better is the excess risk bound given fixed training time  $T$  and noise variance  $\sigma^2$ . We have discussed how over-parameterization affects the gradient norm constant  $B$  and the gradient subspace rank  $\mathrm{rank}(M_0)$  in Section 3. Therefore, we only still need to investigate how the lazy training distance  $R$  changes with the network width, depth, and initialization, as follows.
Lazy training distance  $R$  decreases with model over-parameterization. It is widely observed in the literature [19, 55, 38] that under appropriate choices of initializations, gradient descent on fully connected neural network falls under a lazy training regime. That is, with high probability, there exists a (nearly) optimal solution for the ERM problem that is close to the initialization parameters in terms of  $\ell_2$  distance. Moreover, this lazy training distance  $R$  is closely related to the smallest eigenvalue of the NTK matrix, and generally decreases as the model becomes increasingly overparameterized. In the following proposition, we compute a near-optimal solution via the pseudo inverse of the NTK matrix, and prove that it has small distance to the initialization parameters via existing lower bounds for the smallest eigenvalue of the NTK matrix [40].
Lemma 6.3 (Bounding lazy training distance via smallest eigenvalue of the NTK matrix). Under Assumption 2.4 and single-output linearized network Eq. (3) with  $o = 1$ , assume that the per-layer network widths  $m_0, \dots, m_L = \tilde{\Omega}(n)$  are large. Let  $\mathcal{L}_0^{lin}(\boldsymbol{W})$  be the empirical risk Eq. (1) for
linearized network expanded at initialization vector  $\mathbf{W}_0^{lin}$ . Then for any  $\mathbf{W}_0^{lin}$ , there exists a corresponding solution  $\mathbf{W}_0^{\frac{1}{n^2}}$ , s.t.  $\mathcal{L}_0^{lin}(\mathbf{W}_0^{\frac{1}{n^2}}) - \min_{\mathbf{W}} \mathcal{L}_0^{lin}(\mathbf{W}; \mathcal{D}) \leq \frac{1}{n^2}$ ,  $\text{rank}(M_0) = n$  and
$$
R \leq \tilde {\mathcal {O}} \left(\max  \left\{\frac {1}{d \beta_ {L} \left(\prod_ {i = 1} ^ {L - 1} \beta_ {i} m _ {i}\right)}, 1 \right\} \frac {n}{\sum_ {l = 1} ^ {L} \beta_ {l} ^ {- 1}}\right), \tag {9}
$$
with high probability over training data sampling and random initialization Eq. (5), where  $\tilde{\mathcal{O}}$  ignores logarithmic factors with regard to  $n$ ,  $m$ ,  $L$ , and tail probability  $\delta$ .
The full proof is deferred to Appendix D.2. By using Lemma 6.3, we provide a summary of bounds for  $R$  under different initializations in Table 1. We observe that the lazy training distance  $R$  decreases with increasing width and depth under LeCun, He and NTK initializations, while under Xavier initialization  $R$  only decreases with increasing depth.
Privacy & Excess empirical risk tradeoffs for Langevin diffusion under linearized network. We now use the lazy training distance  $R$  to prove empirical risk bound and combine it with our KL privacy bound Section 4 to show the privacy utility trade-off under over-parameterization.
Corollary 6.4 (Privacy utility trade-off for linearized network). Assume that all conditions in Lemma 6.3 hold. Let  $B$  be the gradient norm constant in Eq. (7), and let  $R$  be the lazy training distance bound in Lemma 6.3. Then for  $\sigma^2 = \frac{2BT}{\varepsilon n^2}$  and  $T = \sqrt{\frac{\varepsilon nR}{2B}}$ , releasing all iterates of Langevin diffusion with time  $T$  satisfies  $\varepsilon$ -KL privacy, and has empirical excess risk upper bounded by
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} _ {0} ^ {l i n} \left(\bar {\boldsymbol {W}} _ {T} ^ {l i n}\right) \right] - \min  _ {\boldsymbol {W}} \mathcal {L} _ {0} ^ {l i n} (\boldsymbol {W}; \mathcal {D}) \leq \tilde {\mathcal {O}} \left(\frac {1}{n ^ {2}} + \sqrt {\frac {B R}{\varepsilon n}}\right) (10) \\ = \tilde {\mathcal {O}} \left(\frac {1}{n ^ {2}} + \sqrt {\frac {\operatorname* {m a x} \left\{1 , d \beta_ {L} \prod_ {l = 1} ^ {L - 1} \beta_ {l} m _ {l} \right\}}{2 ^ {L - 1} \varepsilon}}\right) (11) \\ \end{array}
$$
with high probability over random initialization Eq. (5), where the expectation is over Brownian motion  $B_{T}$  in Langevin diffusion, and  $\tilde{O}$  ignores logarithmic factors with regard to width  $m$ , depth  $L$ , number of training data  $n$  and tail probability  $\delta$ .
See Appendix D.3 for the full proof. Corollary 6.4 proves that the excess empirical risk worsens in the presence of a stronger privacy constraint, i.e., a small privacy budget  $\varepsilon$ , thus contributing to a trade-off between privacy and utility. However, the excess empirical risk also scales with the lazy training distance  $\bar{R}$  and the gradient norm constant  $B$ . These constants depend on network width, depth and initialization distributions, and we prove privacy utility trade-offs for training linearized network under commonly used initialization distributions, as summarized in Table 1.
We would like to highlight that our privacy utility trade-off bound under LeCun and Xavier initialization strictly improves with increasing depth as long as the data satisfy Assumption 2.4 and the hidden-layer width is large enough. To our best knowledge, this is the first time that a strictly improving privacy utility trade-off under over-parameterization is shown in literature. This shows the benefits of precisely bounding the gradient norm (Appendix C.1) in our privacy and utility analysis.
We prove new KL privacy bound for training fully connected ReLU network (and its linearized variant) using the Langevin diffusion algorithm, and investigate how privacy is affected by the network width, depth and initialization. Our results suggest that there is a complex interplay between privacy and over-parameterization (width and depth) that crucially relies on what initialization distribution is used and the how much the gradient fluctuates during training. Moreover, for a linearized variant of fully connected network, we prove KL privacy bounds that improve with increasing depth under certain initialization distributions (such as LeCun and Xavier). We further prove excess empirical risk bounds for linearized network under KL privacy, which similarly improve as depth increases under LeCun and Xavier initialization. This shows the gain of our new privacy analysis for capturing the effect of over-parameterization. We leave it as an important open problem as to whether our privacy utility trade-off results for linearized network could be generalized to deep neural networks.