Title: Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: overparameterized neural network

privacy
Abstract: We analytically investigate how over-parameterization of models in randomized machine learning algorithms impacts the information leakage about their training data. Specifically, we prove a privacy bound for the KL divergence between model distributions on worst-case neighboring datasets, and explore its dependence on the initialization, width, and depth of fully connected neural networks. We find that this KL privacy bound is largely determined by the expected squared gradient norm relative to model parameters during training. Notably, for the special setting of linearized network, our analysis indicates that the squared gradient norm (and therefore the escalation of privacy loss) is tied directly to the per-layer variance of the initialization distribution. By using this analysis, we demonstrate that privacy bound improves with increasing depth under certain initializations (LeCun and Xavier), while degrades with increasing depth under other initializations (He and NTK). Our work reveals a complex interplay between privacy and depth that depends on the chosen initialization distribution. We further prove excess empirical risk bounds under a fixed KL privacy budget, and show that the interplay between privacy utility trade-off and depth is similarly affected by the initialization.

Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks

Abstract

We analytically investigate how over-parameterization of models in randomized machine learning algorithms impacts the information leakage about their training data. Specifically, we prove a privacy bound for the KL divergence between model distributions on worst-case neighboring datasets, and explore its dependence on the initialization, width, and depth of fully connected neural networks. We find that this KL privacy bound is largely determined by the expected squared gradient norm relative to model parameters during training. Notably, for the special setting of linearized network, our analysis indicates that the squared gradient norm (and therefore the escalation of privacy loss) is tied directly to the per-layer variance of the initialization distribution. By using this analysis, we demonstrate that privacy bound improves with increasing depth under certain initializations (LeCun and Xavier), while degrades with increasing depth under other initializations (He and NTK). Our work reveals a complex interplay between privacy and depth that depends on the chosen initialization distribution. We further prove excess empirical risk bounds under a fixed KL privacy budget, and show that the interplay between privacy utility trade-off and depth is similarly affected by the initialization.

1 Introduction

Deep neural networks (DNNs) in the over-parameterized regime (i.e., more parameters than data) perform well in practice but the model predictions can easily leak private information about the training data under inference attacks such as membership inference attacks [44] and reconstruction attacks [17, 7, 29]. This leakage can be mathematically measured by the extent to which the algorithm's output distribution changes if DNNs are trained on a neighboring dataset (differing only in a one record), following the differential privacy (DP) framework [23].

To train differential private model, a typical way is to randomly perturb each gradient update in the training process, such as stochastic gradient descent (SGD), which leads to the most widely applied DP training algorithm in the literature: DP-SGD [2]. To be specific, in each step, DP-SGD employs gradient clipping, adds calibrated Gaussian noise, and yields differential privacy guarantee that scales with the noise multiplier (i.e., per-dimensional Gaussian noise standard deviation divided by the clipping threshold) and number of training epochs. However, this privacy bound [2] is overly general as its gradient clipping artificially neglects the network properties (e.g., width and depth) and training schemes (e.g., initializations). Accordingly, a natural question arises in the community:

How does the over-parameterization of neural networks (under different initializations) affect the privacy bound of the training algorithm over worst-case datasets?

To answer this question, we circumvent the difficulties of analyzing gradient clipping, and instead algorithmically focus on analyzing privacy for the Langevin diffusion algorithm without gradient clipping nor Lipschitz assumption on loss function. 2 It avoids an artificial setting in DP-SGD [2] where a constant sensitivity constraint is enforced for each gradient update and thus makes the privacy bound insensitive to the network over-parameterization. Theoretically , we prove that the KL privacy loss for Langevin diffusion scales with the expected gradient difference between the training on any two worst-case neighboring datasets (Theorem 3.1). 3 By proving precise upper bounds on the expected ℓ 2 -norm of this gradient difference, we thus obtain KL privacy bounds for fully connected neural network (Lemma 3.2) and its linearized variant (Corollary 4.2) that changes with the network width, depth and per-layer variance for the initialization distribution. We summarized the details of our KL privacy bounds in Table 1, and highlight our key observations below.

Width always worsen privacy, under all the considered initialization schemes. Meanwhile, the interplay between network depth and privacy is much more complex and crucially depends on which initialization scheme is used and how long the training time is.

Regarding the specific initialization schemes, under small per-layer variance in initialization (e.g. in LeCun and Xavier), if the depth is large enough, our KL privacy bound for training fully connected network (with a small amount of time) as well as linearized network (with finite time) decays exponentially with increasing depth. To the best of our knowledge, this is the first time that an improvement of privacy bound under over-parameterization is observed.

We further perform numerical experiments (Section 5) on deep neural network trained via noisy gradient descent to validate our privacy analyses. Finally, we analyze the privacy utility trade-off for training linearized network, and prove that the excess empirical risk bound (given any fixed KL privacy budget) scales with a lazy training distance bound R (i.e., how close is the initialization to a minimizer of the empirical risk) and a gradient norm constant B throughout training (Corollary 6.4). By analyzing these two terms precisely, we prove that under certain initialization distributions (such as LeCun and Xavier), the privacy utility trade-off strictly improves with increasing depth for linearized network (Table 1). To our best knowledge, this is the first time that such a gain in privacy-utility trade-off due to over-parameterization (increasing depth) is shown. Meanwhile, prior results only prove (nearly) dimension-independent privacy utility trade-off for such linear models in the literature [45, 32, 37]. Our improvement demonstrates the unique benefits of our algorithmic framework and privacy-utility analysis in understanding the effect of over-parameterization.

1.1 Related Works

over-parameterization in DNNs and NTK . Theoretical demonstration on the benefit of overparameterization in DNNs occurs in global convergence [3, 21] and generalization [4, 16]. Under proper initialization, the training dynamics of over-parameterized DNNs can be described by a kernel function, termed as neural tangent kernel (NTK) [31], which stimulates a series of analysis in DNNs. Accordingly, over-parameterization has been demonstrated to be beneficial/harmful to several topics in deep learning, e.g., robustness [15, 54], covariate shift [50]. However, the relationship between over-parameterization and privacy (based on the differential privacy framework) remains largely an unsolved problem, as the training dynamics typically change [14] after adding new components in the privacy-preserving learning algorithm (such as DP-SGD [2]) to enforce privacy constraints.

Membership inference privacy risk under over-parameterization . A recent line of works [47, 48] investigates how over-parameterization affects the theoretical and empirical privacy in terms of membership inference advantage, and proves novel trade-off between privacy and generalization error. These literature are closet to our objective of investigating the interplay between privacy and overparameterization. However, Tan et al. [47, 48] focus on proving upper bounds for an average-case privacy risk defined by the advantage (relative success) of membership inference attack on models trained from randomly sampled training dataset from a population distribution. By contrast, our KL privacy bound is heavily based on the strongest adversary model in the differential privacy definition, and holds under an arbitrary worst-case pair of neighboring datasets, differing only in one record. Our model setting (e.g., fully connected neural networks) is also quite different from that of Tan et al. [47, 48]. The employed analysis tools are accordingly different.

Differentially private learning in high dimension . Standard results for private empirical risk minimization [9, 46] and private stochastic convex optimization [11, 12, 5] prove that there is an unavoidable factor d in the empirical risk and population risk that depends on the model dimension. However, for unconstrained optimization, it is possible to seek for the dimension-dependency in proving risk bounds for certain class of problems (such as generalized linear model [45]). Recently, there is a growing line of works that proves dimension-independent excess risk bounds for differentially private learning, by utilizing the low-rank structure of data features [45] or gradient matrices [32, 37] during training. Several follow-up works [33, 13] further explore techniques to enforce the low-rank property (via random projection) and boost privacy utility trade-off. However, all the works focus on investigating a general high-dimensional problem for private learning, rather than separating the study for different network choices such as width, depth and initialization. Instead, our study focuses on the fully connected neural network and its linearized variant, which enables us to prove more precise privacy utility trade-off bounds for these particular networks under over-parameterization.

2 Problem and Methodology

We consider the following standard multi-class supervised learning setting. Let D = ( z 1 , · · · , z n ) be an input dataset of size n , where each data record z i = ( x i , y i ) contains a d -dimensional feature vector x i ∈ R d and a label vector y i ∈ Y = {-1 , 1 } o on o classes. We aim to learn a neural network output function f W ( · ) : X → Y parameterized by W via empirical risk minimization (ERM)

where ℓ ( f W ( x i ); y i ) is a loss function that reflects the approximation quality of model prediction f W ( x i ) compared to the ground truth label y i . For simplicity, throughout our analysis, we employ the cross-entropy loss ℓ ( f W ( x ); y ) = -⟨ y , log softmax ( f W ( x )) ⟩ for multi-class network with o ≥ 2 , and ℓ ( f W ( x ); y ) = log(1 + exp( -yf W ( x )) for single-output network with o = 1 .

Fully Connected Neural Networks . We consider the L -layer, multi-output, fully connected, deep neural network (DNN) with ReLU activation. Denote the width of hidden layer l as m l for l = 1 , · · · , L -1 . For consistency, we also denote m 0 = d and m L = o . The network output f W ( x ) := h L ( x ) is defined recursively as follows.

where h l ( x ) denotes the post-activation output at l -th layer, and { W l ∈ R m l × m l -1 : l = 1 , . . . , L } denotes the set of per-layer weight matrices of DNN. For brevity, we denote the vector W := ( Vec ( W 1 ) , . . . , Vec ( W L )) ∈ R m 1 · d + m 2 · m 1 + ··· + o · m L -1 , i.e., the the concatenation of vectorizations for weight matrices of all layers, as the model parameter.

Linearized Network . We also analyze the following linearized network , which is used in prior works [35, 3, 41] as an important tool to (approximately and qualitatively) analyze the training dynamics of DNNs. Formally, the linearized network f lin, 0 W ( x ) is a first-order Taylor expansion of the fully connected ReLU network at initialization parameter W lin 0 , as follows.

where f W lin 0 ( x ) is the output function of the fully connected ReLU network (2) at initialization W lin 0 . We denote L lin 0 ( W ; D ) = 1 n ∑ n i =1 ℓ ( f W lin 0 ( x i ) + ∂ f W ( x ) ∂ W | W = W lin 0 ( W -W lin 0 ); y i ) as the empirical loss function for training linearized network, by plugging (3) into (1).

Langevin Diffusion . Regarding the optimization algorithm, we focus on the Langevin diffusion algorithm [36] with per-dimensional noise variance σ 2 . Note that we aim to avoid gradient clipping while still proving KL privacy bounds. After initializing the model parameters W 0 at time zero, the model parameters W t at subsequent time t evolves as the following stochastic differential equation.

Initialization Distribution . The initialization of parameters W 0 crucially affects the convergence of Langevin diffusion, as observed in prior literatures [52, 25, 24]. In this work, we investigate the following general class of Gaussian initialization distributions with different (possibly depthdependent) variances for the parameters in each layer. For any layer l = 1 , · · · , L , we have

where β 1 , · · · , β L > 0 are the per-layer variance for Gaussian initialization. By choosing different variances, we recover many common initialization schemes in the literature, as summarized in Table 1.

2.1 Our objective and methodology

Weaim to understand the relation between privacy, utility and over-parameterization (depth and width) for the Langevin diffusion algorithm (under different initialization distributions). For privacy analysis, we prove a KL privacy bound for running Langevin diffusion on any two worst-case neighboring datasets. Below we first give the definition for neighboring datasets.

Definition 2.1. We denote D and D ′ as neighboring datasets if they are of same size and only differ in one record. For brevity, we also denote the differing records as ( x , y ) ∈ D and ( x ′ , y ′ ) ∈ D ′ .

Assumption 2.2 (Bounded Data) . For simplicity, we assume bounded data, i.e., ∥ x ∥ 2 ≤ d .

√

We now give the definition for KL privacy, which is a more relaxed, yet closely connected privacy notion to the standard ( ε, δ ) differential privacy [22], see Appendix A.2 for more discussions. KL privacy and its relaxed variants are commonly used in previous literature [8, 10, 53].

Definition 2.3 (KL privacy) . Arandomized algorithm A satisfies ε -KL privacy if for any neighboring datasets D and D ′ , we have that the KL divergence KL( A ( D ) ∥A ( D ′ )) ≤ ε , where A ( D ) denotes the algorithm's output distribution on dataset D .

In this paper, we prove KL privacy upper bound for max D , D ′ KL( W [0: T ] ∥ W ′ [0: T ] ) when running Langevin diffusion on any worst-case neighboring datasets. For brevity, here (and in the remaining paper), we abuse the notations and denote W [0: T ] and W ′ [0: T ] as the distributions of model parameters trajectory during Langevin diffusion processes Eq. (4) with time T on D and D ′ respectively.

For utility analysis, we prove the upper bound for the excess empirical risk given any fixed KL divergence privacy budget for a single-output neural network under the following additional assumption (it is only required for utility analysis and not needed for our privacy bound).

̸

Assumption 2.4 ([40, 20, 21]) . The training data x 1 , · · · , x n are i.i.d. samples from a distribution P x that satisfies E [ x ] = 0 , ∥ x ∥ 2 = √ d for x ∼ P x , and with probability one for any i = j , x i ∦ x j .

Our ultimate goal is to precisely understand how the excess empirical risk bounds (given fixed KL privacy budget) are affected by increasing width and depth under different initialization distributions.

3 KL Privacy for Training Fully Connected ReLU Neural Networks

In this section, we perform the composition-based KL privacy analysis for Langevin Diffusion given random Gaussian initialization distribution under Eq. (5) for fully connected ReLU network. More specifically, we prove upper bound for the KL divergence between distribution of output model parameters when running Langevin diffusion on an arbitrary pair of neighboring datasets D and D ′ .

Our first insight is that by a Bayes rule decomposition for density function, KL privacy under a relaxed gradient sensitivity condition can be proved (that could hold without gradient clipping).

Theorem 3.1 (KL composition under possibly unbounded gradient difference) . The KL divergence between running Langevin diffusion (4) for DNN (2) on neighboring datasets D and D ′ satisfies

Proof sketch . We compute the partial derivative of KL divergence with regard to time t , and then integrate it over t ∈ [0 , T ] to compute the KL divergence during training with time T . For computing the limit of differentiation, we use Girsanov's theorem to compute the KL divergence between the trajectory of Langevin diffusion processes on D and D ′ . The complete proof is in Appendix B.1.

Theorem 3.1 is an extension of the standard additivity [51] of KL divergence (also known as chain rule [1]) for a finite sequence of distributions to continuous time processes with (possibly) unbounded drift difference. The key extension is that Theorem 3.1 does not require bounded sensitivity between the drifts of Langevin Diffusion on neighboring datasets. Instead, it only requires finite second-order moment of drift difference (in the ℓ 2 -norm sense) between neighboring datasets D , D ′ , which can be proved by the following Lemma. We prove that this expectation of squared gradient difference incurs closed-form upper bound under deep neural network (under mild assumptions), for running Langevin diffusion (without gradient clipping) on any neighboring dataset D and D ′ .

Lemma 3.2 (Drift Difference in Noisy Training) . Let M T be the subspace spanned by gradients {∇ ℓ ( f W t ( x i ; y i ) : ( x i , y i ) ∈ D , t ∈ [0 , T ] } n i =1 throughout Langevin diffusion ( W t ) t ∈ [0 ,T ] . Denote ∥·∥ M T as the ℓ 2 norm of the projected input vector onto M T . Suppose that there exists constants c, β > 0 such that for any W , W ′ and ( x , y ) , we have ∥∇ ℓ ( f W ( x ); y )) -∇ ℓ ( f W ′ ( x ); y ) ∥ 2 < max { c , β ∥ W -W ′ ∥ M T } . Then running Langevin diffusion Eq. (4) with Gaussian initialization

Proof sketch . The key is to reduce the problem of upper bounding the gradient difference at any training time T , to analyzing its two subcomponents: ∥∇ ℓ ( f W t ( x ); y )) -∇ ℓ ( f W t ( x ′ ); y ′ ) ∥ 2 2 ≤ 2 ∥∇ ℓ ( f W 0 ( x ); y )) -∇ ℓ ( f W 0 ( x ′ ); y ′ ) ∥ 2 2 ︸ ︷︷ ︸ gradient difference at initialization +2 β 2 ∥ W t -W 0 ∥ 2 M T ︸ ︷︷ ︸ parameters' change after time T +2 c 2 , where ( x , y ) and ′ ′ ′

( x , y ) are the differing data between neighboring datasets D and D . This inequality is by the Cauchy-Schwartz inequality. In this way, the second term in Lemma 3.2 uses the change of parameters

to bound the gradient difference between datasets D and D ′ at time T , via the relaxed smoothness assumption of loss function (that is explained in Remark 3.5 in details). The complete proof is in Appendix B.2.

Remark 3.3 (Gradient difference at initialization) . The first term and in our upper bound linearly scales with the difference between gradients on neighboring datasets D and D ′ at initialization. Under different initialization schemes, this gradient difference exhibits different dependency on the network depth and width, as we will prove theoretically in Theorem 4.1.

Remark 3.4 (Gradient difference fluctuation during training) . The second term in Lemma 3.2 bounds the change of gradient difference during training, and is proportional to the the rank of a subspace M T spanned by gradients of all training data. Intuitively, this fluctuation is because Langevin diffusion adds per-dimensional noise with variance σ 2 , thus perturbing the training parameters away from the initialization at a scale of O ( σ √ rank ( M T )) in the expected ℓ 2 distance.

Remark 3.5 (Relaxed smoothness of loss function) . The third term in Lemma 3.2 is due to the assumption ∥∇ ℓ ( f W ( x ); y )) -∇ ℓ ( f W ′ ( x ); y ) ∥ 2 < max { c , β ∥ W -W ′ ∥ M T } . This assumption is similar to smoothness of loss function, but is more relaxed as it allows non-smoothness at places where the gradient is bounded by c . Therefore, this assumption is general to cover commonly-used smooth, non-smooth activation functions, e.g., sigmoid, ReLU.

Growth of KL privacy bound with increasing training time T . The first and third terms in our upper bound Lemma 3.2 grow linearly with the training time T , while the second term grows exponentially with regard to T . Consequently, for learning tasks that requires a long training time to converge, the second term will become the dominating term and the KL privacy bound suffers from exponential growth with regard to the training time. Nevertheless, observe that for small T → 0 , the second component in Lemma 3.2 contains a small factor e (2+ β 2 ) T -1 2+ β 2 -T = o ( T ) by Taylor expansion.Therefore, for small training time, the second component is smaller than the first and the third components in Lemma 3.2 that linearly scale with T , and thus does not dominate the privacy bound. Intuitively, this phenomenon is related to lazy training [19]. In Section 5 and Figure 2, we also numerically validate that the second component does not have a high effect on the KL privacy loss in the case of small training time.

Dependence of KL privacy bound on network over-parameterization . Under a fixed training time T and noise scale σ 2 , Lemma 3.2 predicts that the KL divergence upper bound in Theorem 3.1 is dependent on the gradient difference and gradient norm at initialization, and the rank of gradient subspace rank ( M T ) throughout training. We now discuss the how these two terms change under increasing width and depth, and whether there are possibilities to improve them under over-parameterization.

The gradient norm at initialization crucially depends on how the per-layer variance in the Gaussian initialization distribution scales with the network width and depth. Therefore, it is possible to reduce the gradient difference at initialization (and thus improve the KL privacy bound) by using specific initialization schemes, as we later show in Section 4 and Section 5.

Regarding the rank of gradient subspace rank ( M T ) : when the gradients along the training trajectory span the whole optimization space, rank ( M T ) would equal the dimension of the learning problem. Consequently, the gradient fluctuation upper bound (and thus the KL privacy bound) worsens with increasing number of model parameters (over-parameterization) in the worst-case. However, if the gradients are low-dimensional [45, 32, 43] or sparse [37], rank ( M T ) could be dimension-independent and thus enables better bound for gradient fluctuation (and KL privacy bound). We leave this as an interesting open problem.

4 KL privacy bound for Linearized Network under over-parameterization

In this section, we focus on the training of linearized networks (3), which fosters a refined analysis on the interplay between KL privacy and over-parameterization (increasing width and depth). Analysis of DNNs via linearization is a commonly used technique in both theory [19] and practice [43, 41]. We hope our analysis for linearized network serves as an initial attempt that would open a door to theoretically understanding the relationship between over-parameterization and privacy.

To derive a composition-based KL privacy bound for training a linearized network, we apply Theorem 3.1 which requires an upper bound for the norm of gradient difference between the training

processes on neighboring datasets D and D ′ at any time t . Note that the empirical risk function for training linearized models enjoys convexity, and thus a relatively short amount of training time is enough for convergence. In this case, intuitively, the gradient difference between neighboring datasets does not change a lot during training, which allows for a tighter upper bound for the gradient difference norm for linearized networks (than Lemma 3.2).

In the following theorem, we prove that for a linearized network, the gradient difference throughout training has a uniform upper bound that only depends on the network width, depth and initialization.

Theorem 4.1 (Gradient Difference throughout training linearized network) . Under Assumption 2.2, taking over the randomness of the random initialization and the Brownian motion, for any t ∈ [0 , T ] , running Langevin diffusion on a linearized network in Eq. (3) satisfies that

where n is the training dataset size, and B is a constant that only depends on the data dimension d , the number of classes o , the network depth L , the per-layer network width { m i } L i =1 , and the per-layer variances { β i } L i =1 of the Gaussian initialization distribution.

Theorem 4.1 provides a precise analytical upper bound for the gradient difference during training linearized network, by tracking the gradient distribution for fully connected feed-forward ReLU network with Gaussian weight matrices. Our proof borrows some techniques from [3, 54] for computing the gradient distribution, refer to Appendix C.1 and C.2 for the full proofs. By plugging Eq. (7) into Theorem 3.1, we obtain the following KL privacy bound for training a linearized network.

Corollary 4.2 (KL privacy bound for training linearized network) . Under Assumption 2.2 and neural networks (3) initialized by Gaussian distribution with per-layer variance { β i } L i =1 , running Langevin diffusion for linearized network with time T on any neighboring datasets satisfies that

where B is the constant that specifies the gradient norm upper bound, given by Eq. (7) .

Over-parameterization affects privacy differently under different initialization . Corollary 4.2 and Theorem 4.1 prove the role of over-parameterization in our KL privacy bound, crucially depending on how the per-layer Gaussian initialization variance β i scales with the per-layer network width m i and depth L . We summarize our KL privacy bound for the linearized network under different width, depth and initialization schemes in Table 1, and elaborate the comparison below.

1

(1) LeCun initialization uses small, width-independent variance for initializing the first layer β 1 = d (where d is the number of input features), and width-dependent variance β 2 = · · · = β L = 1 m for initializing all the subsequent layers. Therefore, the second term ∑ L l =1 β L β l in the constant B of Eq. (7) increases linearly with the width m and depth L . However, due to m l · β l 2 < 1 for all l = 2 , · · · , L , the first product term ∏ L -1 l =1 β l m l 2 in constant B decays with the increasing depth. Therefore, by combining the two terms, we prove that the KL privacy bound worsens with increasing width, but improves with increasing depth (as long as the depth is large enough). Similarly, under Xavier initialization β l = 2 m l -1 + m l , we prove that the KL privacy bound (especially the constant B (7)) improves with increasing depth as long as the depth is large enough.

NTK) and β l = 2 m l -1 (for He). Consequently, the gradient difference under NTK or He initialization is significantly larger than that under LeCun initialization. Specifically, the gradient norm constant B in Eq. (7) grows linearly with the width m and the depth L under He and NTK initializations, thus indicating a worsening of KL privacy bound under increasing width and depth.

5 Numerical validation of our KL privacy bounds

To understand the relation between privacy and over-parameterization in practical DNNs training (and to validate our KL privacy bounds Lemma 3.2 and Corollary 4.2), we perform experiments for

DNNs training via noisy GD to numerically estimate the KL privacy loss. We will show that if the total training time is small, it is indeed possible to obtain numerical KL privacy bound estimates that does not grow with the total number of parameter (under carefully chosen initialization distributions).

Numerical estimation procedure . Theorem 3.1 proves that the exact KL privacy loss scales with the expectation of squared gradient norm during training. This could be estimated by empirically average of gradient norm across training runs. For training dataset D , we consider all 'car' and 'plane' images of the CIFAR-10. For neighboring dataset, we consider all possible D ′ that removes a record from D , or adds a test record to D , i.e., the standard "add-or remove-one" neighboring notion [2]. We run noisy gradient descent with constant step-size 0 . 01 for 50 epochs on both datasets.

Numerically validate the growth of KL privacy loss with regard to training time . Figure 1 shows numerical KL privacy loss under different initializations, for fully connected networks with width 1024 and depth 10 . We observe that the KL privacy loss grows linearly at the beginning of training ( < 10 epochs), which validates the first and third term in the KL privacy bound Lemma 3.2. Moreover, the KL privacy loss under LeCun and Xavier initialization is close to zero at the beginning of training ( < 10 epochs). This shows LeCun and Xavier initialization induce small gradient norm at small training time, which is consistent with Theorem 4.1. However, when the number of epochs is large, the numerical KL privacy loss grows faster than linear accumulation under all initializations, thus validating the second term in Lemma 3.2.

2

4

6

8

10

12

0

.

1

0

.

2

depth

Numerical KL privacy loss

He Initialization

Xavier Initialization

(a) Privacy vs depth (20 epochs, σ = 0 . 01 , width = 1024 )

0

10

20

30

40

50

0

0

.

1

0

.

2

0

.

3

0

.

4

Epochs

Numerical KL privacy loss

He Initialization

Xavier Initialization

2

4

6

8

10

12

0

.

1

0

.

2

depth

0

200

400

600

800

1

,

000

0

.

1

0

.

2

width

(b) Privacy vs depth (50 epochs, σ = 0 . 01 , width = 1024 )

(c) Privacy vs width (50 epochs, σ = 0 . 01 , depth = 10 )

Numerically validate the dependency of KL privacy loss on network width, depth and initializations . Figure 2 shows the numerical KL privacy loss under different network depth, width and initializations, for a fixed training time. In Figure 2c, we observe that increasing width and training time always increases KL privacy loss. This is consistent with Theorem 4.1, which shows that increasing width worsens the gradient norm at initialization (given fixed depth), thus harming KL privacy bound Lemma 3.2 at the beginning of training. We also observe that the relationship between KL privacy

and network depth depends on the initialization distributions and the training time. Specifically, in Figure 2a, when the training time is small (20 epochs), for LeCun and Xavier initializations, the numerical KL privacy loss improves with increasing depth when depth > 8 . Meanwhile, when the training time is large (50 epochs) in Figure 2b, KL privacy loss worsens with increasing depth under all intializations. This shows that given small training time, the choice of initialization distribution affects the dependency of KL privacy loss on increasing depth, thus validating Lemma 3.2 and Theorem 4.1.

6 Utility guarantees for Training Linearized Network

Our privacy analysis suggests that training linearized network under certain initialization schemes (such as LeCun initialization) allows for significantly better privacy bounds under overparameterization by increasing depth. In this section, we further prove utility bounds for Langevin diffusion under initialization schemes and investigate the effect of over-parameterization on the privacy utility trade-off. In other words, we aim to understand whether there is any utility degradation for training linearized networks when using the more privacy-preserving initialization schemes.

Convergence of training linearized network . We now prove convergence of the excess empirical risk in training linearized network via Langevin diffusion. This is a well-studied problem in the literature for noisy gradient descent. We extend the convergence theorem to continuous-time Langevin diffusion below and investigate factors that affect the convergence under over-parameterization. The proof is deferred to Appendix D.1.

Lemma 6.1 (Extension of [42, Theorem 2] and [45, Theorem 3.1]) . Let L lin 0 ( W ; D ) be the empirical risk function of a linearized network in Eq. (3) expanded at initialization vector W lin 0 . Let W ∗ 0 be an α -near-optimal solution for the ERM problem such that L lin 0 ( W ∗ 0 ; D ) -min W L lin 0 ( W ; D ) ≤ α . Let D = { x i } n i =1 be an arbitrary training dataset of size n , and denote M 0 = ( ∇ f W lin 0 ( x 1 ) , · · · , ∇ f W lin 0 ( x n ) ) ⊤ as the NTK feature matrix at initialization. Then running Langevin diffusion (4) on L lin 0 ( W ) with time T and initialization vector W lin 0 satisfies

where the expectation is over Brownian motion B T in Langevin diffusion in Eq. (4) , ¯ W lin T = 1 T ∫ ¯ W lin t d t is the average of all iterates, and R = ∥ W lin 0 -W ∗ 0 ∥ 2 M 0 is the gap between initialization parameters W lin 0 and solution W ∗ 0 .

Remark 6.2 . The excess empirical risk bound in Lemma 6.1 is smaller if data is low-rank, e.g., image data, then rank ( M 0 ) is small. This is consistent with the prior dimension-independent private learning literature [32, 33, 37] and shows the benefit of low-dimensional gradients on private learning.

Lemma 6.1 highlights that the excess empirical risk scales with the gap R between initialization and solution (denoted as lazy training distance), the rank of the gradient subspace, and the constant B that specifies upper bound for expected gradient norm during training. Specifically, the smaller the lazy training distance R is, the better is the excess risk bound given fixed training time T and noise variance σ 2 . We have discussed how over-parameterization affects the gradient norm constant B and the gradient subspace rank rank ( M 0 ) in Section 3. Therefore, we only still need to investigate how the lazy training distance R changes with the network width, depth, and initialization, as follows.

Lazy training distance R decreases with model over-parameterization . It is widely observed in the literature [19, 55, 38] that under appropriate choices of initializations, gradient descent on fully connected neural network falls under a lazy training regime. That is, with high probability, there exists a (nearly) optimal solution for the ERM problem that is close to the initialization parameters in terms of ℓ 2 distance. Moreover, this lazy training distance R is closely related to the smallest eigenvalue of the NTK matrix, and generally decreases as the model becomes increasingly overparameterized. In the following proposition, we compute a near-optimal solution via the pseudo inverse of the NTK matrix, and prove that it has small distance to the initialization parameters via existing lower bounds for the smallest eigenvalue of the NTK matrix [40].

Lemma 6.3 (Bounding lazy training distance via smallest eigenvalue of the NTK matrix) . Under Assumption 2.4 and single-output linearized network Eq. (3) with o = 1 , assume that the per-layer network widths m 0 , · · · , m L = ˜ Ω( n ) are large. Let L lin 0 ( W ) be the empirical risk Eq. (1) for

linearized network expanded at initialization vector W lin 0 . Then for any W lin 0 , there exists a corresponding solution W 1 n 2 0 , s.t. L lin 0 ( W 1 n 2 0 ) -min W L lin 0 ( W ; D ) ≤ 1 n 2 , rank ( M 0 ) = n and

with high probability over training data sampling and random initialization Eq. (5) , where ˜ O ignores logarithmic factors with regard to n , m , L , and tail probability δ .

The full proof is deferred to Appendix D.2. By using Lemma 6.3, we provide a summary of bounds for R under different initializations in Table 1. We observe that the lazy training distance R decreases with increasing width and depth under LeCun, He and NTK initializations, while under Xavier initialization R only decreases with increasing depth.

Privacy & Excess empirical risk tradeoffs for Langevin diffusion under linearized network . We now use the lazy training distance R to prove empirical risk bound and combine it with our KL privacy bound Section 4 to show the privacy utility trade-off under over-parameterization.

Corollary 6.4 (Privacy utility trade-off for linearized network) . Assume that all conditions in Lemma 6.3 holds. Let B be the gradient norm constant in Eq. (7) , and let R be the lazy training distance bound in Lemma 6.3. Then for σ 2 = 2 BT εn 2 and T = √ εnR 2 B , releasing all iterates of Langevin diffusion with time T satisfies ε -KL privacy, and has empirical excess risk upper bounded by

with high probability over random initiailization Eq. (5) , where the expectation is over Brownian motion B T in Langevin diffusion, and ˜ O ignores logarithmic factors with regard to width m , depth L , number of training data n and tail probability δ .

See Appendix D.3 for the full proof. Corollary 6.4 proves that the excess empirical risk worsens in the presence of a stronger privacy constraint, i.e., a small privacy budget ε , thus contributing to a trade-off between privacy and utility. However, the excess empirical risk also scales with the lazy training distance R and the gradient norm constant B . These constants depend on network width, depth and initialization distributions, and we prove privacy utility trade-offs for training linearized network under commonly used initialization distributions, as summarized in Table 1.

We would like to highlight that our privacy utility trade-off bound under LeCun and Xavier initialization strictly improves with increasing depth as long as the data satisfy Assumption 2.4 and the hidden-layer width is large enough. To our best knowledge, this is the first time that a strictly improving privacy utility trade-off under over-parameterization is shown in literature. This shows the benefits of precisely bounding the gradient norm (Appendix C.1) in our privacy and utility analysis.

7 Conclusion

We prove new KL privacy bound for training fully connected ReLU network (and its linearized variant) using the Langevin diffusion algorithm, and investigate how privacy is affected by the network width, depth and initialization. Our results suggest that there is a complex interplay between privacy and over-parameterization (width and depth) that crucially relies on what initialization distribution is used and the how much the gradient fluctuates during training. Moreover, for a linearized variant of fully connected network, we prove KL privacy bounds that improve with increasing depth under certain initialization distributions (such as LeCun and Xavier). We further prove excess empirical risk bounds for linearized network under KL privacy, which similarly improve as depth increases under LeCun and Xavier initialization. This shows the gain of our new privacy analysis for capturing the effect of over-parameterization. We leave it as an important open problem as to whether our privacy utility trade-off results for linearized network could be generalized to deep neural networks.

Acknowledgments and Disclosure of Funding

The authors would like to thank Yaxi Hu and anonymous reviewers for helpful discussions on drafts of this paper. This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043), and the Swiss National Science Foundation (SNSF) under grant number 200021_205011, Google PDPO faculty research award, Intel within the www.private-ai.org center, Meta faculty research award, the NUS Early Career Research Award (NUS ECRA award number NUS ECRA FY19 P16), and the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

References

Entropy, Relative Entropy, and Mutual Information , chapter 2, pages 13-55. John Wiley & Sons, Ltd, 2005. ISBN 9780471748823. doi: https://doi.org/10.1002/047174882X.ch2. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/047174882X.ch2 .

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning , pages 242-252. PMLR, 2019.

Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pages 322-332. PMLR, 2019.

Hilal Asi, Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: Optimal rates in l1 geometry. In International Conference on Machine Learning , pages 393-403. PMLR, 2021.

Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in neural information processing systems , 31, 2018.

Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1138-1156. IEEE, 2022.

Rina Foygel Barber and John C Duchi. Privacy and statistical risk: Formalisms and minimax bounds. arXiv preprint arXiv:1412.4451 , 2014.

Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science , pages 464-473. IEEE, 2014.

Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 1046-1059, 2016.

Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. Advances in neural information processing systems , 32, 2019.

Raef Bassily, Cristóbal Guzmán, and Michael Menart. Differentially private stochastic optimization: New results in convex and non-convex settings. Advances in Neural Information Processing Systems , 34:9317-9329, 2021.

Raef Bassily, Mehryar Mohri, and Ananda Theertha Suresh. Differentially private learning with margin guarantees. arXiv preprint arXiv:2204.10376 , 2022.

Zhiqi Bu, Hua Wang, and Qi Long. On the convergence and calibration of deep learning with differential privacy. arXiv preprint arXiv:2106.07830 , 2021.

Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. Journal of the ACM , 70(2):1-18, 2023.

Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in neural information processing systems , 32, 2019.

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium , volume 6, 2021.

Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215 , 2022.

Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems , 32, 2019.

Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054 , 2018.

Simon S. Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 1675-1685. PMLR, 2019.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer, 2006.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. , 9(3-4):211-407, 2014.

Yoav Freund, Yi-An Ma, and Tong Zhang. When is the convergence time of langevin algorithms dimension independent? a composite optimization viewpoint. Journal of Machine Learning Research , 23(214):1-32, 2022.

Arun Ganesh and Kunal Talwar. Faster differentially private samplers via rényi divergence analysis of discretized langevin mcmc. Advances in Neural Information Processing Systems , 33:7222-7233, 2020.

Arun Ganesh, Abhradeep Thakurta, and Jalaj Upadhyay. Langevin diffusion: An almost universal algorithm for private euclidean (convex) optimization. arXiv preprint arXiv:2204.01585 , 2022.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

Chuan Guo, Alexandre Sablayrolles, and Maziar Sanjabi. Analyzing privacy leakage in machine learning via multiple hypothesis testing: A lesson from fano. arXiv preprint arXiv:2210.13662 , 2022.

Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. arXiv preprint arXiv:2206.07758 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Peter Kairouz, Monica Ribero Diaz, Keith Rush, and Abhradeep Thakurta. (nearly) dimension independent private erm with adagrad rates via publicly estimated subspaces. In Conference on Learning Theory , pages 2717-2746. PMLR, 2021.

Shiva Prasad Kasiviswanathan. Sgd with low-dimensional gradients with applications to private and distributed learning. In Uncertainty in Artificial Intelligence , pages 1905-1915. PMLR, 2021.

Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade , pages 9-50. Springer, 2002.

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems , 32, 2019.

DS Lemons, A Gythiel, and Paul Langevin's. Sur la théorie du mouvement brownien [on the theory of brownian motion]. CR Acad. Sci.(Paris) , 146:530-533, 1908.

Xuechen Li, Daogao Liu, Tatsunori B Hashimoto, Huseyin A Inan, Janardhan Kulkarni, Yin-Tat Lee, and Abhradeep Guha Thakurta. When does differentially private learning not suffer in high dimensions? Advances in Neural Information Processing Systems , 35:28616-28630, 2022.

Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. Phase diagram for two-layer relu neural networks at infinite-width limit. The Journal of Machine Learning Research , 22(1): 3327-3373, 2021.

Saeed Mahloujifar, Alexandre Sablayrolles, Graham Cormode, and Somesh Jha. Optimal membership inference bounds for adaptive composition of sampled gaussian mechanisms. arXiv preprint arXiv:2204.06106 , 2022.

Quynh Nguyen, Marco Mondelli, and Guido F Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference on Machine Learning , pages 8119-8129. PMLR, 2021.

Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can linearized neural networks actually say about generalization? Advances in Neural Information Processing Systems , 34:8998-9010, 2021.

Ohad Shamir and Tong Zhang. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In International conference on machine learning , pages 71-79. PMLR, 2013.

Vaishaal Shankar, Alex Fang, Wenshuo Guo, Sara Fridovich-Keil, Jonathan Ragan-Kelley, Ludwig Schmidt, and Benjamin Recht. Neural kernels without tangents. In International Conference on Machine Learning , pages 8614-8623. PMLR, 2020.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

Shuang Song, Thomas Steinke, Om Thakkar, and Abhradeep Thakurta. Evading the curse of dimensionality in unconstrained private glms. In International Conference on Artificial Intelligence and Statistics , pages 2638-2646. PMLR, 2021.

Kunal Talwar, Abhradeep Thakurta, and Li Zhang. Private empirical risk minimization beyond the worst case: The effect of the constraint set geometry. arXiv preprint arXiv:1411.5417 , 2014.

Jasper Tan, Blake Mason, Hamid Javadi, and Richard Baraniuk. Parameters or privacy: A provable tradeoff between overparameterization and membership inference. Advances in Neural Information Processing Systems , 35:17488-17500, 2022.

Jasper Tan, Daniel LeJeune, Blake Mason, Hamid Javadi, and Richard G Baraniuk. A blessing of dimensionality in membership inference through regularization. In International Conference on Artificial Intelligence and Statistics , pages 10968-10993. PMLR, 2023.

Leonida Tonelli. Sull'integrazione per parti. Rend. Acc. Naz. Lincei , 5(18):246-253, 1909.

Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Covariate shift in high-dimensional random feature regression. arXiv preprint arXiv:2111.08234 , 2021.

Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory , 60(7):3797-3820, 2014.

Santosh Vempala and Andre Wibisono. Rapid convergence of the unadjusted langevin algorithm: Isoperimetry suffices. Advances in neural information processing systems , 32, 2019.

Yu-Xiang Wang, Jing Lei, and Stephen E Fienberg. On-average kl-privacy and its equivalence to generalization for max-entropy mechanisms. In Privacy in Statistical Databases: UNESCO Chair in Data Privacy, International Conference, PSD 2016, Dubrovnik, Croatia, September 14-16, 2016, Proceedings , pages 121-134. Springer, 2016.

Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, and Volkan Cevher. Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization). arXiv preprint arXiv:2209.07263 , 2022.

Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes overparameterized deep relu networks. Machine learning , 109:467-492, 2020.

Contents

A Symbols and definitions

A.1 Additional notations

Vecorization Vec ( · ) denotes the transformation that takes an input matrix A = ( a ij ) i ∈ [ r ] ,j ∈ [ c ] ∈ R r × c (with r rows and c columns) and outputs a rc -dimensional column vector: Vec ( A ) = ( a 1 , 1 , · · · , a r, 1 , a 1 , 2 , · · · , a r, 2 , · · · , a 1 ,c , · · · , a r,c ) ⊤ .

Softmax function: softmax ( y ) = e y [ j ] ∑ o j =1 e y [ j ] where o is the number of output classes.

o : number of output classes for the neural network.

A.2 Relation between KL privacy Definition 2.3 to differential privacy

KL privacy is a more relaxed, yet closely connected privacy notion to ( ε, δ ) differential privacy [22].

KL privacy and differential privacy are both worst-case privacy notions over all possible neighboring datasets, by requiring bounded distinguishability between the algorithm's output distributions on neighboring datasets in statistical divergence. The difference is that KL privacy requires bounded KL divergence, while ( ε, δ ) -differential privacy is equivalent to bounded Hockey-stick divergence [6].

KL privacy and differential privacy are both definitions based on the privacy loss random variable log P ( A ( D )= o ) P ( A ( D ′ )= o ) , o ∼ A ( D ) (following the definition in [2, Equation 1]). KL privacy implies that the privacy loss random variable has a bounded first order moment, while differential privacy requires a high probability argument that the privacy loss random variable is bounded by ε with probability 1 -δ . Therefore, KL privacy is generally a more relaxed notion than differential privacy.

Translation to each other: For ε = 0 , KL privacy (bounded first-order moment of privacy loss random variable) implies (0 , δ ) -differential privacy with δ = √ ε 2 by Pinsker inequality. Higher order moments of the privacy loss random variable suffice to prove ( ε, δ ) -differential privacy for ε > 0 . Note that ( ε, δ ) -DP with δ > 0 does not necessarily imply KL privacy, as the privacy loss random variable may be large at the tail event with δ probability.

Due to the connection to the privacy loss random variable (which is closely connected to the likelihood ratio test for membership hypothesis testing), both differential privacy and KL privacy incur upper bound on the performance curve of inference attacks, such as the membership inference and attribute inference [39, 28], as we discuss in Footnote 3.

B Deferred proofs for Section 3

B.1 Deferred proofs for Theorem 3.1

To prove the new composition theorem, we will use the Girsanov's Theorem. Here we follow the presentation of [18, Theorem 6].

Theorem B.1 (Implication of Girsanov's theorem [18, Theorem 6]) . Let ( X t ) t ∈ [0 ,η ] and ( ˜ X t ) t ∈ [0 ,η ] be two continuous-time processes over R r . Let P T be the probability measure that corresponds to the trajectory of ( X t ) t ∈ [0 ,η ] , and let Q T be the probability measure that corresponds to the trajectory of ( ˜ X t ) t ∈ [0 ,η ] . Suppose that the process ( X t ) t ∈ [0 ,η ] follows

where ( B t ) t ∈ [0 ,T ] is a standard Brownian motion over P T , and the process ( ˜ X t ) t ∈ [0 ,η ] follows

where ( ˜ B t ) t ∈ [0 ,T ] is a standard Brownian motion over Q T with d ˜ B t = d B t + 1 σ ( b t -˜ b t ) . Assume that σ is a r × r symmetric positive definite matrix. Then, provided that Novikov's condition holds,

we have that

We are now ready to apply Girsanov's theorem to prove the following new KL privacy composition theorem for Langevin diffusion processes on neighboring datasets D and D ′ . For ease of description, we repeat Theorem 3.1 as below.

Theorem 3.1 . [KL composition under possibly unbounded gradient difference] The KL divergence between running Langevin diffusion (4) for DNN (2) on neighboring datasets D and D ′ satisfies

Proof. Recall that W t denotes the model parameter after running Langevin diffusion on dataset D with time t , and W ′ t denotes the model parameter after running Langevin diffusion on dataset D ′ with time t . To avoid confusion, we further denote p [ t 1 : t 2 ] and p ′ [ t 1 : t 2 ] as the distributions of model parameters trajectories W [ t 1 : t 2 ] and W ′ [ t 1 : t 2 ] during time inteval [ t 1 , t 2 ] respectively. By definition of partial derivative, we have that

Now we compute the term KL( W [0: t + η ] ∥ W ′ [0: t + η ] ) as follows.

where p [ t : t + η ] | [0: t ] ( · | w [0: t ] ) is the conditional distribution during running Langevin diffusion on dataset D , given fixed values for model parameters trajectory W [0: t ] = w [0: t ] . Similarly, p ′ [ t : t + η ] | [0: t ] ( · | w [0: t ] ) is the conditional distribution during running Langevin diffusion on dataset D ′ .

Therefore by using the Markov property of the Langevin diffusion process and the definition of KL divergence in Eq. (15), we have that

Nowto compute the term KL ( p [ t : t + η ] | t ( · | w t ) ∥ ∥ ∥ p ′ [ t : t + η ] | t ( · | w t ) ) , we only need to apply Theorem B.1 to the following two Langevin diffusion processes ( W t + s | t ) s ∈ [0 ,η ] and ( W ′ t + s | t ) s ∈ [0 ,η ] , conditioning on the observation W t | t = W ′ t | t = w t at time t .

Note that when η is small enough, we have that the Novikov's condition in Eq. (12) holds because the exponent inside integration 1 2 ∫ η 0 ∥ σ -1 s ( b s -˜ b s ) ∥ 2 2 d s scales linearly with η that can be arbitrarily small. Therefore, by applying Girsanov's theorem, we have that

where b s -˜ b s = -∇L ( W t + s | t ; D ) + ∇L ( W t + s | t ; D ′ ) . By d ˜ B s = d B s + 1 σ ( b s -˜ b s ) and Itô integration with regard to W [ t : t + η ] | t , we have that

By plugging Eq. (17) into Eq. (16), we have that

By plugging Eq. (18) into Eq. (14), and by exchanging the order of expectation and integration, we have that

Note that here we exchange the order of expectation and integration by using the Tonelli's Theorem [49] for non-negative integrand function ∥∇L ( W t + s ; D ) -∇L ( W t + s ; D ′ ) ∥ 2 2 . Integrating Eq. (19) on t ∈ [0 , T ] finishes the proof.

B.2 Deferred proofs for Lemma 3.2

Lemma 3.2 . Let M T be the subspace spanned by gradients {∇ ℓ ( f W t ( x i ; y i ) : ( x i , y i ) ∈ D , t ∈ [0 , T ] } n i =1 throughout Langevin diffusion ( W t ) t ∈ [0 ,T ] . Denote ∥·∥ M T as the ℓ 2 norm of the projected input vector onto M T . Suppose that there exists constants c, β > 0 such that for any W , W ′ and ( x , y ) , we have ∥∇ ℓ ( f W ( x ); y )) -∇ ℓ ( f W ′ ( x ); y ) ∥ 2 < max { c , β ∥ W -W ′ ∥ M T } . Then running Langevin diffusion Eq. (4) with Gaussian initialization distribution (5) satisfies ε -KL privacy with max D , D′ ∫ T 0 E [ ∥ ∇L ( W t ; D ) -∇L ( W t ; D ′ ) ∥ 2 ] d t

Proof. By definition of the neighboring datasets D and D ′ , and the definition of empirical risk in Eq. (1), we have that for any W , it satisfies that

where ( x , y ) and ( x ′ , y ′ ) are the differing records between neighboring datasets D amd D ′ . By the assumption that ∥∇ ℓ ( f W ( x ); y )) - ∇ ℓ ( f W ′ ( x ); y ) ∥ 2 < max { c , β ∥ W -W ′ ∥ M T } , and by the Cauchy-Schwarz inequality, we further have that for any W and W t , it satisfies that

The first term ∥∇ ℓ ( f W 0 ( x ); y )) -∇ ℓ ( f W 0 ( x ′ ); y ′ ) ∥ 2 2 is constant during training (as it only depends on the initialization). Therefore, we only need to bound the second term ∥ W t -W 0 ∥ 2 M T . For brevity, we denote the function Φ( W ) = ∥ W -W 0 ∥ 2 M T . Recall our definition, p t as the distribution of model parameters after running Langevin diffusion on dataset D with time t , and p ′ t as the distribution of model parameters after running Langevin diffusion on dataset D ′ with time t . Then we have that

Denote Γ s as the following random operator on model parameters θ .

where Z ∼ N (0 , I ) . We first claim that the following equation holds.

This is by using Euler-Maruyama discretization method to approximate the solution W t of SDE Eq. (4). More specifically, the approximation error E p t + η [Φ( W )] -E p t [Φ(Γ η ( W ))] is of size O ( rη 2 ) for small η , where r is the dimension of W .

Therefore, by plugging Eq. (23) into Eq. (22), we have that

Recall that ∇ 2 Φ( W ) exists almost everywhere with regard to W ∼ p t . Therefore we could approximate the term E p t [Φ(Γ η ( W ); D , D ′ )] via its second-order Taylor expansion at W as follows.

By plugging Φ( W ) = ∥ W -W 0 ∥ 2 M T into Eq. (24), we have that

where the last inequality is by using the Cauchy-schwartz inequality. By plugging the assumption that ∥∇ ℓ ( f W ( x ); y )) -∇ ℓ ( f W ′ ( x ); y ) ∥ 2 < max { c , β ∥ W -W ′ ∥ 2 } into the above inequality, we have that

By solving the above ordinary differential inequality on t ∈ [0 , T ] , we have that

By plugging Eq. (29) into Eq. (21) and Eq. (20), followed by integration over time t ∈ [0 , T ] , we have that

which concludes our proof.

C Deferred proofs for Section 4

C.1 Bounding the gradient norm at initialization

To bound the moment of ℓ 2 norm of the gradient ∂f ( x ) ∂ W l of network output function, we need the following (extended) lemmas from Zhu et al. [54].

Lemma C.1 ([54, Lemma 1]) . Let w ∼ N (0 , σ 2 I n ) , then for two fixed non-zero vectors h 1 , h 2 ∈ R n whose correlation is unknown, define two random variables X = ( w ⊤ h 1 1 { w ⊤ h 2 ≥ 0 } ) 2 and Y = s ( w ⊤ h 1 ) 2 , where s ∼ Ber (1 , 1 / 2) follows a Bernoulli distribution with 1 trial and 1 2 success rate, and s and w are independent random variables. Then X and Y have the same distribution.

Lemma C.2 (Extension of [54, Lemma 2]) . Given a fixed non-zero matrix H 1 ∈ R p × r and a fixed non-zero vector h 2 ∈ R p and , let W ∈ R q × p be a random matrix with i.i.d. entries W ij ∼ N (0 , β ) and a matrix (or vector) V = ϕ ′ ( Wh 2 ) WH 1 ∈ R q × r , then, we have E [ ∥ V ∥ 2 F ∥ H 1 ∥ 2 F ] = qβ 2 .

Proof. According to the definition of V = ϕ ′ ( Wh 2 ) WH 1 ∈ R q × r , we have:

where D i,i = 1 {⟨ W [ i ] , h 2 ⟩≥ 0 } , W [ i ;] is the i -th row of W , and H [; j ] 1 is the j -th column vector of H 1 . Therefore by Lemma C.1, with i.i.d. Bernoulli random variable ρ 1 , · · · , ρ q ∼ Ber (1 , 1 / 2) , we have

where ˜ w ij = ⟨ W [ i ;] , H [; j ] 1 ⟩ / (√ β ∥ H [; j ] 1 ∥ 2 2 ) . By the fact that W [ i ;] has i.i.d. Gaussian entries, for any fixed j , we have that ˜ w ij ∼ N (0 , 1) , i = 1 , · · · , q independently. Therefore, we have

Now, we are ready to prove output gradient expectation at random initialization as follows.

Lemma C.3 (Output Gradient Expectation Bound at Random Initialization) . Fix any data record x , then over the randomness of the initialization distributions for W 1 , · · · , W L , i.e., W l ∼ N (0 , β l I ) for l = 1 , · · · , L -1 , it satisfies that

Proof. We use Vec ( W l ) to denote the concatenation of all row vector of the parameter matrix W l . By chain rule, for l = 1 , · · · , L -1 , we have that

Similarly, for the L -th layer, we have that

By properties of ReLU activation ϕ , we have ϕ ′ L -i = diag [ sgn ( W L -i h L -1 -i )] , where sgn ( x ) = { 1 x > 0 0 x ≤ 0 operates coordinate-wise with regard to the input matrix. Therefore, we have that for l = 1 , · · · , L -1

For notational simplicity, we introduce the notation of t l ′ l for l = 1 , · · · , L -1 and l ≤ l ′ < L as follows.

Then by definition, we have that

By rotational invariance of Gaussian column vectors, we prove that for any possible value of t L -1 l (which is completely determined by W 1 , · · · , W L -1 and x ), for any l = 1 , · · · , L -1 , we have that

By Lemma C.2, for any l = 1 , · · · , L -2 and l ≤ l ′ ≤ L -2 , we have that

We now bound the last term E W 1 , ··· , W l [ ∥ t l l ∥ 2 F ] for l = 1 , · · · , L -1 . By definition, we have that

To bound the E W 1 , ··· , W l [ ∥ h l -1 ∥ 2 2 ] , note that by Lemma C.2, we prove that for any l = 1 , · · · , L -1

.

Therefore, for any l = 1 , · · · , L , we have that

By plugging (42) into (39), we have that

By combining (37), (38) and (43), we have for any l = 1 , · · · , L -1

On the other hand, by plugging Eq. (42) (under ℓ = L ) into Eq. (36), we have that

Therefore,

which suffices to prove Eq. (33).

C.2 Deferred proof for Theorem 4.1

Finally, we prove that the gradient difference between two training datasets under linearized network is bounded by a constant throughout training (which only depends on the network width, depth and initialization distribution).

Theorem 4.1 . Under Assumption 2.2, taking over the randomness of the random initialization and the Brownian motion, for any t ∈ [0 , T ] , running Langevin diffusion on linearized network Eq. (3) satisfies that

where n is the training dataset size, and B is a constant that only depends on the data dimension d , the number of classes o , the network depth L , the per-layer network width { m i } L i =1 , and the per-layer variances { β i } L i =1 of the Gaussian initialization distribution as follows.

Proof. Denote W as the initialization parameters and denote W lin t as the parameters for linearized network after training time t . Then the gradient difference under linearized network and cross-entropy loss function is as follows.

Plugging Lemma C.3 into the above equation with data Assumption 2.2 suffice to prove the result.

D Deferred proofs for Section 6

D.1 Deferred proof for Lemma 6.1 on convergence of training linearized network

In this section, we prove empirical risk bound for the average of all iterates of Langevin diffusion, building on standard results [42, Theorem 2] and [45, Theorem 3.1] for the (discrete time) stochastic gradient descent algorithm.

Lemma 6.1 . (Extension of [42, Theorem 2] and [45, Theorem 3.1]) Let L lin 0 ( W ; D ) be the empirical risk function of linearized newtork Eq. (3) expanded at initialization vector W lin 0 . Let W ∗ 0 be an α -near-optimal solution for the ERM problem such that L lin 0 ( W ∗ 0 ; D ) -min W L lin 0 ( W ; D ) ≤ α . Let D = x 1 , · · · , x n be an arbitrary training dataset of size n , and denote M 0 = ( ∇ f W lin 0 ( x 1 ) , · · · , ∇ f W lin 0 ( x n ) ) ⊤ as the NTK feature matrix at initialization. Then running Langevin diffusion (4) on L lin 0 ( W ) with time T and initialization vector W lin 0 satisfies

where the expectation is over Brownian motion B T in Langevin diffusion Eq. (4) , ¯ W lin T = 1 T ∫ ¯ W lin t d t is the average of all iterates, and R = ∥ W lin 0 -W ∗ 0 ∥ 2 M 0 is the gap between initialization parameters W lin 0 and solution W ∗ 0 .

Proof. Our proofs are heavily based on the idea in [45, Theorem 3.1] to work only in the parameter space spanned by the input feature vectors. And our proof serves as an extension of their bound to the continuous-time Langevin diffusion algorithm. We begin by using convexity of the empirical loss function L lin 0 ( W ; D ) for linearized network to prove the following standard results

Denote M 0 = ( ∇ f W lin 0 ( x 1 ) · · · ∇ f W lin 0 ( x n ) ) and compute the gradient under cross entropy loss and linearized network, we have ∇L lin 0 ( W lin T ; D ) lies in the column space of M 0 . Denote Π M 0 as the projection operator to the column space of M 0 , then (46) can be rewritten as

By taking expectation over the randomness of Brownian motion in Langevin diffusion, we have

We now rewrite E [ ⟨ Π M 0 ( ¯ W lin t -W ∗ 0 ) , ∇L lin 0 ( W lin t ; D ) ⟩ ] by computing ∂ ∂t E [ ∥ W lin t -W ∗ 0 ∥ 2 M 0 ] , where ∥ W lin t -W ∗ 0 ∥ 2 M 0 = Π M 0 ( W lin t -W ∗ 0 ) ⊤ Π M 0 ( W lin t -W ∗ 0 ) . By applying (24) with p t being the distribution for W lin t in Langevin diffusion for linearized network starting from point initialization W lin 0 , and with function d ( W ) = ∥ W -W ∗ 0 ∥ 2 M 0 , we have that

Therefore by plugging (49) into (48), we have that

D.2 Deferred proof for Lemma 6.3

To bound lazy training distance of training linearized network, we would need the following auxiliary Lemma about high probability upper bound for the final layer output of linearized network at initialization.

Lemma D.1. Fix any data record x , then with high probability 1 -δ over random initialization Eq. (5) of model weight matrices W 1 , · · · , W L for layer 1 , · · · , L , i.e., W l ∼ N (0 , β l I ) , it satisfies that

where ˜ O ignores logarithmic terms with regard to width m , depth L and tail probability δ .

Proof. To bound the term ∥ f W lin 0 ( x i ) ∥ 2 2 , by definition Eq. (2), for any x , we have that

We now bound the terms in the right-hand-side of Eq. (53) one by one.

Regarding the first term ∥ h L ( x ) ∥ 2 2 ∥ h L -1 ( x ) ∥ 2 2 in Eq. (53), observe that by the network output definition Eq. (2), we have that

where ˜ w ∼ N (0 , 1) and the last equality is by rotaional invariance of Gaussian distribution used for initializing L -th layer weight matrix W L ∈ R 1 × m . Therefore, by tail probabilty expression for standard Gaussian random variable, we have that with high probability 1 -δ L over random initialization of W L ∈ R 1 × m , it satisfies that

Regarding the terms ∥ h l ( x ) ∥ 2 2 ∥ h l -1 ( x ) ∥ 2 2 in Eq. (53) for layer l = 1 , · · · , L -1 , by setting H 1 = h 2 = h l -1 in Eq. (32), we immediately prove that over random initialization of weight matrix W l ∈ R m l × m l -1 , it satisfies that

where ρ 1 , · · · , ρ m l i.i.d. ∼ Ber (1 , 1 2 ) and ˜ w 1 , · · · , ˜ w m l i.i.d. ∼ N (0 , 1) and ρ i and ˜ w i are independent.

By tail probabilty expression for Gaussian random variable ˜ w i , we have that P ( ˜ w 2 i ≥ t ) ≤ e -t/ 2 for any t > 0 . By union bound over i = 1 , · · · , m l , we prove that for any layer l = 1 , · · · , L -1 , with high probability 1 -δ 2 L over random initialization of weight matrix W l ∈ R m l × m l -1 , it satisfies that

Moreover, by applying Hoeffding's inequality to i.i.d. Bernoulli r.v.s ρ 1 , · · · , ρ m , we prove with high probability 1 -δ 2 L , it satisfies that ∑ m i =1 ρ i ≤ m l 2 (1 + log(2 L/δ )) . By combining it with Eq. (57) via union bound, and plugging the result into Eq. (59), we prove for any l = 1 , · · · , L -1 , it satisfies with high probability 1 -δ L over random initialization of weight matrix W l ∈ R m l × m l -1 that

By using union bound over Eq. (58) for layer l = 1 , · · · , L -1 and Eq. (55), we have that with high probability 1 -δ over random initialization Eq. (5), it satisfies that

where ˜ O ignores logarithmic factors with regard to m , L and δ .

By plugging Eq. (59) into Eq. (53), we prove that with high probability 1 -δ over initialization Eq. (5), the following bound holds.

where ˜ O ignores logarithmic terms with regard to width m , depth L and tail probability δ .

Lemma 6.3 . (Bounding lazy training distance via smallest eigenvalue of the NTK matrix) Under Assumption 2.4 and single-output linearized network Eq. (3) with o = 1 , assume that the per-layer network widths m 0 , · · · , m L = ˜ Ω( n ) are large. Let L lin 0 ( W ) be the empirical risk Eq. (1) for linearized network expanded at initialization vector W lin 0 . Then for any W lin 0 , there exists a corresponding solution W 1 n 2 0 , s.t. L lin 0 ( W 1 n 2 0 ) -min W L lin 0 ( W ; D ) ≤ 1 n 2 , rank ( M 0 ) = n and

with high probability over training data sampling and random initialization Eq. (5) , where ˜ O ignores logarithmic factors with regard to n , m , L , and tail probability δ .

Proof. Given an arbitrary initialization parameter vector W lin 0 , we first construct an solution W 1 n 2 0 that is nearly optimal for the ERM problem over L lin 0 ( W ) , as follows.

is the gradient matrix at initialization and † denotes pseudo-inverse.

We now prove that the solution W 1 n 2 0 is close to the initialization parameters W lin 0 in ℓ 2 distance with high probability. By applying the holder inequality in (62), we have that

where λ 0 is the smallest non-zero eigenvalue of the PSD matrix M 0 M ⊤ 0 . When the data regularity assumption Assumption 2.4 holds and the per-layer width m 0 , · · · , m L = ˜ Ω( n ) , by applying existing bounds for the smallest eigenvalue of the NTK matrix M 0 M ⊤ 0 for single-output network in [40,

Theorem 4.1], we prove that with high probability over data sampling and random initialization Eq. (5), it satisfies that

where we have set the auxiliary variables ξ l = 1 in [40, Theorem 4.1] because the per-layer width m l = ˜ Ω( n ) are large enough for l = 0 , · · · , L (where n is the number of training data).

Therefore, by using Eq. (65), we have that with high probability over data sampling and random initialization Eq. (5), it satisfies that

where m l is the width of layer l and n is the number of training data.

For the second term in Eq. (64), by Cauchy-Schwarz inequality, we have that

By Lemma D.1, we have that ∥ f W lin 0 ( x ) ∥ 2 ≤ ˜ O ( d · β L ( ∏ L -1 i =1 β i m i )) with high probability over the random initialization Eq. (5). By plugging this result and y i ∈ {-1 , 1 } into Eq. (67), we have that with high probability over random initialization Eq. (5), it satisfies that

where ˜ O ingores logarithmic factors with regard to n , m , L , and tail probability δ . Therefore, by combining Eq. (66) and Eq. (68) with union bound, and by plugging the result into Eq. (64), we have that with high probability over data sampling and random initialization Eq. (5), it satisfies that

where m l is the width of layer l , n is the number of training data, and ˜ O ignores logarithmic factors with regard to n , m , L , and tail probability δ .

We finally prove that W 1 n 2 0 is a 1 n 2 -near-optimal solution. Note that Eq. (65) implies that with high probability M 0 M ⊤ 0 is full rank, i.e., rank ( M 0 M ⊤ 0 ) = n . Therefore M † 0 = M ⊤ 0 ( M 0 M ⊤ 0 ) -1

random initialization Eq. (5). By plugging it into the cross-entropy loss for the single-output network

defined below Eq. (1), we have that with high probability over the training data sampling and random initialization Eq. (5), the solution W 1 n 2 0 satisfies

D.3 Deferred proof for Corollary 6.4

Corollary 6.4 . (Privacy utility trade-off for linearized network) Assume that all conditions in Lemma 6.3 holds. Let B = d ( ∏ L -1 i =1 β i m i 2 ) ∑ L l =1 β L β l be the gradient norm constant proved in Eq. (7) , and let R ≤ ˜ O ( max { 1 dβ L ( ∏ L -1 i =1 β i m i ) , 1 } n ∑ L l =1 β -1 l ) be the lazy training distance bound proved in Lemma 6.3. Then for σ 2 = 2 BT εn 2 and T = √ εnR 2 B , releasing all iterates of Langevin diffusion with time T satisfies ε -KL privacy, and has empirical excess risk upper bounded by

with high probability over random initiailization Eq. (5) , where the expectation is over Brownian motion B T in Langevin diffusion, and ˜ O ignores logarithmic factors with regard to width m , depth L , number of training data n and tail probability δ . A summary of our excess empirical risk bounds under different initializations is in Table 1.

Proof. By setting W ∗ 0 = W 1 n 2 0 in Lemma 6.1 with W 1 n 2 0 constructed as Lemma 6.3, we have with high probability over random initialization Eq. (5), we have

where R ≤ ˜ O ( max { 1 dβ L ( ∏ L -1 i =1 β i m i ) , 1 } n ∑ L l =1 β -1 l ) by Lemma 6.3.

Meanwhile, to ensure ε -KL privacy, by Corollary 4.2, we only need to set σ 2 = 2 BT εn 2 where B = d ( ∏ L -1 i =1 β i m i 2 ) ∑ L l =1 β L β l by for single-output network with o = 1 . By plugging σ 2 = 2 BT εn 2 into (72), we prove that

Setting T = √ εnR 2 B in (73) and elaborating the computations suffice to prove the result.

E Discussion on extending our results to Noisy GD with constant step-size

In this section, we discuss how to extend our privacy analyses to noisy GD with constant step-size. Specifically, we only need to extend the KL composition theorem under possibly unbounded gradient difference, i.e., Theorem 3.1, to the noisy GD algorithm.

Theorem E.1 (KL composition for noisy GD under possibly unbounded gradient difference) . Let the iterative update in noisy GD algorithm be defined by: W ( k +1) = W ( k ) -η ∇L ( W ( k ) ; D ) +

√ 2 ησ 2 Z k , where Z k ∼ N (0 , I ) . Then the KL divergence between running noisy GD for DNN (2) on neighboring datasets D and D ′ satisfies

Proof. Denote p ( k ) as the distribution of model parameters after running noisy GD on dataset D with k steps, and similarly denote p ′ ( k ) as the distribution of model parameters after running noisy GD on dataset D ′ with k steps. Similarly, denote p (1: k ) as the joint distribution of ( W (1) , · · · , W ( k ) ) , and denote p ′ (1: k ) as the joint distribution of ( W ′ (1) , · · · , W ′ ( k ) ) . Now we expand the term KL ( p (1 ,k +1) , p ′ (1 ,k +1) ) by the Bayes rule as follows.

Observe that conditioned on fixed model parameters W (1: k ) at iteration 1 , · · · , k , the distributions p ( k +1) | (1: k ) , p ′ ( k +1) | (1: k ) are Gaussian with per-dimensional variance σ 2 . Therefore, by computing the KL divergence between two multivariate Gaussians, we have that

Therefore, by plugging Eq. (77) into Eq. (76), we have that

By summing (78) over k = 0 , · · · , K -1 and observing that KL ( p (0) , p ′ (0) ) = 0 (as the initialization distribution is the same between noisy GD on D and D ′ ), we finish the proof for Eq. (74).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to connect the structure of a neural network and its initialization scheme to the privacy guarantees it can offer. KL divergence is used as privacy metric, under the settings of Langevine diffusion.\n\nSTRENGTHS:\nThe paper aims to solve a very important problem, i.e. the effects of overparameterization on privacy for overparametrized deep learning models. The analysis through KL divergence allows the problem to be treated analytically, and to connect the structure of the network to privacy bounds.\n\nWEAKNESSES:\nfifth line of caption of Table 1: $n$ and $\\phi$ appear here for the first time. It looks like they represent the number of training samples and your regularity condition parameter respectively. Probably better to mention this in the caption as well.\n\nsame line: $m = \\Omega(n^{14} L^{16} \\phi^{-4})$. To me this assumption looks severely irrealistic for any deep learning model.\n\nThe trade-offs in the table are simply upper-bounds. It is unclear to me how these upper-bounds can be considered tight, therefore telling anything about a possible comparison between different initializations / widths.\n\nActivation function and regularity parameter have the same notation $\\phi$\n\nAssumption 2.1 enforces the samples to respect $\\| x_i\\|_2 \\leq 1$. This scaling is quite innatural considering your initializations schemes. which are designed to renormalize the preactivations when the input norms scale with $\\sqrt d$ (more natural assumption).\n\nEnd of line 151, are the subscripts a typo here?\n\nIt looks that the main bound in Lemma 4.1 derives from a worst case, per-layer computation, under the assumption of ReLU activation. This suggests the bound to be non tight, and therefore to not be informative about how different initializations really affect the preservation of privacy.\n\nLack of experimental validation on the claims in the paper.\n\nQUESTIONS:\nthird line of caption of figure 1: you write $d = m$ first, and then you mention that you assume $m \\geq d$. Is this a typo?\n\nwould your results still hold if we consider a more natural scaling of the data $\\| x_i\\|_2 \\leq \\sim d$?\n\nYour results are valid only in the case of ReLU networks. Is the analysis generalizable?\n\nIn the discussion after Lemma 3.2 you mention that there are two important keys to control privacy: initialization and training time. You focus your analysis on the first one, and you motivate why the second component is not relevant in the case of small training time. Correct? If yes, how do you verify this hypothesis? Is it a consequence of lazy training?\n\nYour Lemma 4.1 looks to exploit a worst-case bound over the layers of the network, plus the property of the ReLU activation. Can you confirm?\n\nLIMITATIONS:\nNo limitations discussion needed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the KL privacy bound for training fully connected networks and its linearized variant using the Langevin diffusion algorithm. The study delves into the impact of network width, depth, and initialization on privacy. Notably, the paper reveals that, with specific parameter initialization, the KL privacy bound can be enhanced by increasing the depth of the network. Additionally, the author provides a proof that the privacy-utility trade-off of linearized networks can be improved with depth when employing LeCun initialization.\n\nSTRENGTHS:\n1.The author investigates an intriguing problem that is likely to capture significant interest within the community.\n\n2.The results presented in the paper are not only interesting but also coherent, aligning well with the research topic.\n\n3.The paper is skillfully written, allowing for easy comprehension and smooth flow of ideas.\n\nWEAKNESSES:\nThe paper lacks numerical experiments to validate the disparity between empirical KL privacy leakage and the privacy bound proposed in this paper. This would shed light on the tightness of the bound. Given that the arguments are based on the upper bound, providing numerical experiments demonstrating the evolution of empirical KL privacy leakage with varying depths and widths, aligned with the theoretical findings, would enhance the persuasiveness of the research.\n\nQUESTIONS:\nQuestions:\n1. In Lemma 3.2, Equation (7) suggests that for relatively large T, the dominant factor in the KL privacy bound is the fluctuation term of gradient differences, which appears to be independent of the differences between neighboring datasets $D$ and $D'$. Could the author provide further clarification on this matter?\n2. Additionally, I am curious about the evolution of the rank of $M_T$ with respect to time T. The author briefly touches upon this aspect, but a more detailed explanation would be appreciated.\n\nSome suggestions and corrections for the paper:\n1. In Assumption 2.2, it would be preferable to use a commonly recognized constant symbol such as 'C' instead of '\\phi' to denote a constant.\n2. There seems to be some confusion in the notations in lines 151 and 152. Is it intended that x_1 = x, y = y_1, and y' = y_2?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically analyzes how the width, depth, and initialization of deep neural networks affect the privacy utility trade-off. For the special setting of linearized networks, the paper shows that gradient norm and privacy loss hinge on the per-layer variance of the initialization distributions. Thus, the privacy trade-off can be governed by width, depth, and initialization. Interestingly, for LeCun's initialization, the utility can be amplified by increasing depth which is against common intuition that overparameterized models will worsen the privacy utility trade-off.\n\nSTRENGTHS:\n1. The theoretical finding on the connection between depth and privacy-utility trade-off is interesting, that the deeper network can achieve better tradeoff than the shallow ones given LeCun's initialization.\n2. It is novel that the network properties, e.g., initialization is analyzed through the Lagevian dynamics on linearized networks.\n\nWEAKNESSES:\n1. The presentation coould be improved. Some typos in the formulations make the paper hard to follow.\n   * KL privacy bound is used before the definition or explanation in Line 38. Especially, what is the relation between the KL privacy bound and differential privacy bound?\n   * The definition of the non-lazy training regime in Table 1 is not defined.\n   * For the definition of LeCun initialization, the $\\beta$ is layer-independent in Line 262 as $\\beta_2=\\cdots =\\beta_L={1\\over m}$. But in Table 1, the $\\beta$ is layer-dependent as $\\beta_l = \\frac{1}{m_{l-1}}$.\n   * In Line 24, `is randomze` should be `is randomizing`.\n2. The difference between LeCun's initialization $\\beta_l=1/m_{l-1}$ and others $\\beta_l=2/m_{l-1}$ is mainly in the factor, `2`. The authors could give a more intuitive explanation for why the factor is crucial for privacy. \n3. More generally regarding the factor in initialization, analyzing the best factor (probably some value between 0 and 2) may lead to a counterintuitive trade-off between privacy and utility. If I follow the derivations in Line 264-265, the excess empirical risk for LeCun's initialization with $\\beta_t=\\frac{a}{m_{l-1}}$  with $a\\in (0,2)$ could be proportional to $\\frac{a}{2}^{L/2}$. This means that the risk bound will be greatly reduced when $a$ approaches zero. However, this is counterintuitive, as a smaller $a$ implies smaller $\\beta_t$, and therefore model will be initialized almost as zero. As common knowledge, zero parameters are not favored for the initialization of a deep network. In brief, smaller initial parameters will quickly vanish the activations by layer and therefore the model cannot converge or of low utility. Unless I missed some assumptions, the derivation is problematic due to the above counterintuitive results.\n\nQUESTIONS:\n1. What is the relation between the KL privacy bound and the differential privacy bound? What is the motivation to study the KL privacy bound?\n2. Can you explain the counterintuitive derivation given a variable factor in $\\beta_l$? See Weakness 3 for details.\n\n----- Update after reading rebuttals ---\nThe authors have addressed most of my concerns.\n\nLIMITATIONS:\nThe authors did not discuss the limitations or potential social impacts of the paper. \n1. The paper presents some promising privacy improvement through proper initialization and also demonstrate some increasing risks if networks are not properly initialized. The results could be an important signal for people to treat the initialization seriously. I suggest the authors make the social impact clearly in the paper.\n2. The paper has some inherent limitations due to the analysis tools of Lagevian diffusion algorithms, for example, the requirement for linearized networks, large depths, etc. The limitations are hard to address in the scope of the paper but is worth to mention for readers to under the proper use cases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
The authors propose a KL privacy bound for DP-SGD that depend on the width, depth, and initialisation of the neural network. They show that the privacy bound improves with increasing depth for the LeCun and Xavier initialisation while it degrades with increasing depth.

STRENGTHS:
1. The authors came up with an original idea exploring a topic that has so far received little attention in the literature
2. the findings are somewhat surprising and unintuitive

WEAKNESSES:
1. Often there are mistakes in privacy proofs. Having some experiments with privacy auditing to back up the surprising results would make the results more credible. 

2. There are many typos in the middle of the paper. Especially related to the use of articles (articles missing) or repetitive words. 

3. The authors motivate their contribution by the loose privacy bounds of DP-SGD. But the privacy bounds are then not used to improve the privacy accounting of the neural network. 

4. Limitations not discussed in depth

QUESTIONS:
1. The result about the improving privacy bound is really surprising. Do the authors have an intuition why this is the case?

LIMITATIONS:
Limitations are not explicitly mentioned. The authors touch upon the restrictive results on linearised networks but this limitation is mostly underplayed. It would be interesting to discuss more when this simplification can lead to worse results.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09