Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descent

1 Introduction . Machine learning models leak a significant amount of information about their training data , through their parameters and predictions [ 21 , 18 , 5 ] . Iterative randomized training algorithms can limit this information leakage and bound the differential privacy loss of the learning process [ 3 , 1 , 8 , 9 ] . The strength of this certified defense is determined by an upper bound on the ( Rényi ) divergence between the probability distributions of model parameters learned on any pair of neighboring datasets . The general method to compute the differential privacy bound for gradient perturbation-based learning algorithms is to view the process as a number of ( identical ) differential privacy mechanisms , and to compute the composition of their bounds . However , this over-estimates the privacy loss of the released model [ 13 , 19 ] , and results in a loose differential privacy bound . This is because composition bounds also accounts for the leakage of all intermediate gradient updates , even though only the final model parameters are observable to adversary . Feldman et al . [ 8 , 9 ] address this issue for the privacy analysis of gradient computations over one single training epoch , for smooth and convex loss functions . However , in learning a model over multiple training epochs , such a guarantee is quantitatively similar to composition bounds of privacy amplification by sub-sampling [ 8 ] . The open challenge , that we tackle in this paper , is to provide an analysis that can tightly bound the privacy loss of the released model after K training epochs , for any K. We present a novel analysis for privacy dynamics of noisy gradient descent with smooth and strongly convex loss functions . We construct a pair of continuous-time Langevin diffusion [ 20 ] processes that trace the probability distributions over the model parameters of noisy GD . Subsequently , we derive differential inequalities bounding the rate of privacy loss ( worst case Rényi divergence between the coupled stochastic processes associated with neighboring datasets ) throughout the training process . * Equal contribution . Alphabetical Order . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We then prove an exponentially-fast converging privacy bound for noisy GD : ( simplified theorem ) Under 1-strongly convex and β-smooth loss function ` ( θ ; x ) with total gradient sensitivity 1 , the noisy GD Algorithm 1 , with initial parameter vector θ0 ∼ ΠC ( N ( 0 , 2σ2Id ) ) and step size η < 1β , satisfies ( α , ε ) -Rényi DP with ε = O ( α σ2n2 ( 1− e −ηK2 ) ) , where n is the size of the training set . This guarantee shows that the privacy loss converges exponentially in the number of iterations K , instead of growing proportionally with K as in the composition-based analysis of the same algorithms . Our bound captures a strong privacy amplification due to the dynamics ( and convergence ) of differential privacy over the noisy gradient descent algorithm with private internal state . We analyze the tightness of the bound , the utility of the randomized algorithm under the computed differential privacy bound , as well as its gradient complexity ( number of required gradient computations ) . We prove the tightness guarantee for our bound by showing that there exist a loss function and neighboring datasets such that the divergence between corresponding model parameter distributions matches our privacy bound . For Lipschitz , smooth , and strongly convex loss functions , we prove that noisy GD achieves optimal utility under differential privacy with an error of order O ( dn2ε2 ) , with a small gradient complexity of order O ( n log ( n ) ) . This improves over the prior utility results for noisy SGD algorithms [ 3 ] . Our analysis results in a significantly smaller gradient complexity by a factor of n/ log ( n ) , and a slightly better utility by a factor of polylog ( n ) . We anticipate that our work will have a positive societal impact , by paving the way for building accurate and privacy preserving machine learning systems for sensitive personal data . 2 Preliminaries on differential privacy . Let X be the data universe , and a dataset D contain n records from it : D = ( x1 , x2 , · · · , xn ) ∈ Xn . We refer to a dataset pair D , D′ as neighboring if they differ in one data record . A measure ν is said to be absolutely continuous with respect to another measure ν′ on same space ( denoted as ν ν′ ) if for all measurable set S , ν ( S ) = 0 whenever ν′ ( S ) = 0 . Definition 2.1 ( [ 17 ] Rényi differential privacy ) . Let α > 1 . A randomized algorithm A : Xn → Rd satisfies ( α , ε ) -Rényi Differential Privacy ( RDP ) , if for any two neighboring datasets D , D′ ∈ Xn , the α Rényi divergence Rα ( A ( D ) ‖A ( D′ ) ) ≤ ε . For a pair of measures ν , ν′ over the same space with ν ν′ , Rα ( ν‖ν′ ) is defined as Rα ( ν‖ν′ ) = 1 α− 1 logEα ( ν‖ν′ ) , where Eα ( ν‖ν′ ) = ∫ ( dν dν′ ) α dν′ . ( 1 ) We refer to Rα ( A ( D ) ‖A ( D′ ) ) also as the Rényi privacy loss of algorithm A on datasets D , D′ . An RDP guarantee can be converted to ( ε , δ ) -DP guarantee [ 17 , Proposition 5 ] . Definition 2.2 ( [ 22 ] Rényi information ) . Let α > 1 . For any two measures ν , ν′ over Rd with µ ν′ and corresponding probability density functions p , p′ , if p ( θ ) p′ ( θ ) is differentiable , the α-Rényi Information of ν with respect to ν′ is Iα ( ν‖ν′ ) = 4 α2 E θ∼p′ [ ∥∥∥∥∇ p ( θ ) α2p′ ( θ ) α2 ∥∥∥∥2 2 ] = E θ∼p′ [ p ( θ ) α−2 p′ ( θ ) α−2 ∥∥∥∥∇ p ( θ ) p′ ( θ ) ∥∥∥∥2 2 ] . ( 2 ) See the Appendix B for a comprehensive presentation of preliminaries . 3 Privacy analysis of noisy gradient descent . Let D = ( x1 , x2 , · · · , xn ) be a dataset of size n with records taken from a universe X . For a given machine learning algorithm , let ` ( θ ; x ) : X × Rd → R be a loss function of a parameter vector θ ∈ C on the data point x , where C is a closed convex set ( can be Rd ) . A generic formulation of the optimization problem to learn the model parameters , is in the form of empirical risk minimization ( ERM ) with the following objective , where LD ( θ ) is the empirical loss of the model , with parameter vector θ , on a dataset D. θ∗ = arg min θ∈C LD ( θ ) , where LD ( θ ) = 1 n ∑ x∈D ` ( θ ; x ) . ( 3 ) Releasing this optimization output ( i.e. , θ∗ ) can leak information about the dataset D , hence violating data privacy . To mitigate this risk , there exist randomized algorithms to ensure that the ( α-Rényi ) privacy loss of the ERM algorithm is upper-bounded by ε , i.e. , the algorithm satisfies ( α , ε ) -RDP . Algorithm 1 ANoisy-GD : Noisy Gradient Descent Input : Dataset D = ( x1 , x2 , · · · , xn ) , loss function ` ( θ ; x ) , closed convex set C ⊆ Rd , learning rate η , noise variance σ2 , initial parameter vector θ0 . 1 : for k = 0 , 1 , · · · , K − 1 do 2 : g ( θk ; D ) = ∑n i=1∇ ` ( θk ; xi ) 3 : θk+1 = ΠC ( θk − ηng ( θk ; D ) + √ 2ηN ( 0 , σ2Id ) ) 4 : Output θK In this paper , our objective is to analyze privacy loss of Noisy Gradient Descent ( Algorithm 1 ) , which is a randomized ERM algorithm . Let θk , θ′k be the parameter vectors at the k ’ th iteration of ANoisy-GD on neighboring datasets D and D′ , respectively . We denote by Θηk and Θ′ηk the corresponding random variables that model θk and θ′k . We abuse notation to also denote their probability distributions by Θηk and Θ′ηk . In this paper , our objective is to model and analyze the dynamics of differential privacy of this algorithm . More precisely , we focus on the following . 1 . Compute an RDP bound ( i.e. , the worst case Rényi divergence Rα ( ΘK‖Θ′K ) between the output distributions of two neighboring datasets ) for Algorithm 1 , and analyze its tightness . 2 . Compute the contribution of each iteration to the privacy loss . As we go from step k = 1 to K in Algorithm 1 , we investigate how the algorithm ’ s privacy loss changes as it runs the k ’ th iteration ( computed as Rα ( Θηk ∥∥∥Θ′ηk ) −Rα ( Θη ( k−1 ) ∥∥∥Θ′η ( k−1 ) ) ) . In the end , we aim to provide a RDP bound that is tight , thus facilitating optimal utility [ 3 ] . We emphasize that our goal is to construct a theoretical framework for analyzing privacy loss of releasing the output θK of the algorithm , assuming private internal states ( i.e. , θ1 , · · · , θK−1 ) . 3.1 Tracing diffusion for Noisy GD . To analyze the privacy loss of Noisy GD , which is a discrete-time stochastic process , we first interpolate each discrete update from θk to θk+1 with a piece-wise continuously differentiable diffusion process . Let D and D′ be a pair of arbitrarily chosen neighboring datasets . Given step-size η and initial parameter vector θ0 = θ′0 , the respective k ’ th discrete updates in Algorithm 1 on neighboring datasets D and D′ are { θk+1 = ΠC ( θk − η∇LD ( θk ) + √ 2ησ2Zk ) , θ′k+1 = ΠC ( θ ′ k − η∇LD ( θ′k ) + √ 2ησ2Zk ) , with Zk ∼ N ( 0 , Id ) . ( 4 ) These two discrete jumps can be interpolated with two stochastic processes Θt and Θ′t over time ηk ≤ t ≤ η ( k + 1 ) respectively . At the start of each step , t = ηk , the random variables Θηk and Θ′ηk model the distribution of the θk and θ ′ k in the noisy GD processes respectively . During time ηk < t < η ( k + 1 ) , we model the respective gradient updates on D and D′ with the following stochastic processes . { Θt = Θηk − η · U1 ( Θηk ) − ( t− ηk ) · U2 ( Θηk ) + √ 2 ( t− ηk ) σ2Zk Θ′t = Θ ′ ηk − η · U1 ( Θ′ηk ) + ( t− ηk ) · U2 ( Θ′ηk ) + √ 2 ( t− ηk ) σ2Zk ( 5 ) where the vectors U1 ( θ ) = 12 ( ∇LD ( θ ) +∇LD′ ( θ ) ) and U2 ( θ ) = 1 2 ( ∇LD ( θ ) −∇LD′ ( θ ) ) represent the average and difference between gradients on neighboring datasets D and D′ respectively . At the end of step , i.e . at t→ η ( k + 1 ) , we project Θt and Θ′t onto convex set C , and obtain Θη ( k+1 ) = ΠC ( lim t→η ( k+1 ) − Θt ) , Θ′η ( k+1 ) = ΠC ( lim t→η ( k+1 ) − Θ′t ) . ( 6 ) By plugging ( 5 ) into ( 6 ) , we compute that the projected random variable Θη ( k+1 ) and Θ′η ( k+1 ) have the same distributions as the parameters θk+1 and θ′k+1 at k + 1 th step of noisy GD respectively . Repeating the construction for k = 0 , · · · , K − 1 , we define two piece-wise continuous diffusion processes { Θt } t≥0 and { Θ′t } t≥0 whose distributions at time t = ηk are consistent with θk and θ′k in the noisy GD processes ( 4 ) for any k ∈ { 0 , · · · , K − 1 } . Definition 3.1 ( Coupled tracing diffusions ) . Let Θ0 = Θ′0 be two identically distributed random variables . We refer to the stochastic processes { Θt } t≥0 and { Θ′t } t≥0 that evolve along diffusion processes ( 5 ) in ηk < t < η ( k + 1 ) and undergo projection steps ( 6 ) at the end of step t = η ( k + 1 ) , as coupled tracing diffusions for noisy GD on neighboring datasets D , D′ . The Rényi divergence Rα ( ΘηK‖Θ′ηK ) reflects the Rényi privacy loss of Algorithm 1 with K steps . Conditioned on observing θk and θ′k , the processes { Θt } ηk < t < η ( k+1 ) and { Θ′t } ηk < t < η ( k+1 ) in ( 5 ) are Langevin diffusions along vector fields −U2 ( θk ) and U2 ( θ′k ) respectively , for duration η . Therefore , conditioned on observing θk and θ′k , the diffusion processes in ( 5 ) have the following stochastic differential equations ( SDEs ) respectively . dΘt = −U2 ( θk ) dt+ √ 2σ2dWt , dΘ ′ t = U2 ( θ ′ k ) dt+ √ 2σ2dWt , ( 7 ) where dWt ∼ √ dtN ( 0 , Id ) describe the Wiener processes on Rd . Therefore , the conditional probability density functions pt|ηk ( θ|θk ) and p′t|ηk ( θ|θ ′ k ) follow the following Fokker-Planck equation . For brevity , we use pt|ηk ( θ|θk ) and p′t|ηk ( θ|θ ′ k ) to represent the conditional probability density function p ( Θt = θ|Θηk = θk ) and p ( Θ′t = θ|Θ′ηk = θ′k ) respectively . ∂pt|ηk ( θ|θk ) ∂t = ∇ · ( pt|ηk ( θ|θk ) U2 ( θk ) ) + σ2∆pt|ηk ( θ|θk ) ∂p′t|ηk ( θ|θ ′ k ) ∂t = −∇ · ( p′t|ηk ( θ|θ ′ k ) U2 ( θ ′ k ) ) + σ2∆pt|ηk ( θ|θ′k ) ( 8 ) By taking expectations over probability density function pηk ( θk ) or p′ηk ( θ ′ k ) on both sides of ( 8 ) , we obtain the partial differential equation that models the evolution of ( unconditioned ) probability density function pt ( θ ) and p′t ( θ ) in the coupled tracing diffusions . Lemma 1 . For coupled tracing diffusion processes ( 5 ) in time ηk < t < η ( k + 1 ) , the equivalent Fokker-Planck equations are { ∂pt ( θ ) ∂t = ∇ · ( pt ( θ ) Vt ( θ ) ) + σ 2∆pt ( θ ) ∂p′t ( θ ) ∂t = ∇ · ( p ′ t ( θ ) V ′ t ( θ ) ) + σ 2∆p′t ( θ ) , ( 9 ) where Vt ( θ ) = −V ′t ( θ ) = E θk∼pηk|t [ U2 ( θk ) |θ ] are time-dependent vector fields on Rd , and U2 ( θ ) = 1 2 [ ∇LD ( θ ) −∇LD′ ( θ ) ] is the difference between gradients on neighboring datasets . By this density evolution equation , we model the noisy gradient descent updates with coupled tracing diffusions . The tracing diffusion process is similar to Langevin diffusion . Therefore , we first study the privacy dynamics in coupled tracing ( Langevin ) diffusions .

Summary: This paper investigates the information leakage of noisy gradient descent (GD) algorithms for smooth and strongly convex loss functions. The authors analyze the dynamics of (Renyi) privacy loss during training (over K iterations) when the internal state of the training algorithm is not observable. This leads to an upper bound on the rate of privacy loss of the released model by way of the Renyi divergence between a pair of distributions over model parameters learned on neighboring datasets. The bound turns out to be much better than that obtained using composition-based approaches.