On the Convergence and Calibration of Deep Learning with Differential Privacy

1 Introduction . Deep learning has achieved tremendous success in many applications that involve crowdsourced information , e.g. , face image , emails , financial status , and medical records . However , using such sensitive data raises severe privacy concerns on a range of image recognition , natural language processing and other tasks ( Cadwalladr & Graham-Harrison , 2018 ; Rocher et al. , 2019 ; Ohm , 2009 ; De Montjoye et al. , 2013 ; 2015 ) . For concrete examples , there are multiple successful privacy attacks on deep learning models , in which members in the dataset can be re-identified using the location or the purchase record , via the membership inference attack ( Shokri et al. , 2017 ; Carlini et al. , 2019 ) . In another example , the attackers can extract a person ’ s name , email address , phone number , and physical address from the billion-parameter GPT-2 ( Radford et al. , 2019 ) via the extraction attack ( Carlini et al. , 2020 ) . Therefore , many studies have applied differential privacy ( DP ) ( Dwork et al. , 2006 ; Dwork , 2008 ; Dwork et al. , 2014 ; Mironov , 2017 ; Duchi et al. , 2013 ; Dong et al. , 2019 ) , a mathematically rigorous approach , to protect against leakage of private information ( Abadi et al. , 2016 ; McSherry & Talwar , 2007 ; McMahan et al. , 2017 ; Geyer et al. , 2017 ) . To achieve this gold standard of privacy guarantee , since the seminal work ( Abadi et al. , 2016 ) , DP optimizers are applied to train the neural networks while preserving the accuracy of prediction . To name a few , researchers have proposed DP-SGD ( Abadi et al. , 2016 ; Bassily et al. , 2014 ) , DP-Adam ( Bu et al. , 2019 ) , DP-SGLD ( Wang et al. , 2015 ; Li et al. , 2019 ) , DP-FTRL ( Kairouz et al. , 2021 ) , DP-FedSGD and DP-FedAvg ( McMahan et al. , 2017 ) . Algorithmically speaking , DP optimizers generally have two extra steps in comparison to non-DP standard optimizers : the per-sample clipping and the random noise addition , so that DP optimizers descend in the direction of the averaged , clipped , noisy gradient ( see Figure 1 ) . These extra steps protect the resulting models against privacy attacks via the Gaussian mechanism ( Dwork et al. , 2014 , Theorem A.1 ) , at the expense of an empirical performance degradation compared to the non-DP deep learning , in terms of much slower convergence and lower utility . For example , state-of-the-art CIFAR10 accuracy with DP is ≈ 70 % without pre-training ( Papernot et al. , 2020 ) ( while the same non-DP networks can achieve 90 % accuracy ) and similar performance drops have been observed on facial images , tweets , and many other datasets ( Bagdasaryan et al. , 2019 ) . Empirically , many works have evaluated the effects of noise scale , batch size , clipping norm , learning rate , and network architecture on the privacy-accuracy trade-off ( Abadi et al. , 2016 ; Papernot et al. , 2020 ) . However , despite the prevalent usage of DP optimizers , little is known about its convergence behavior from a theoretical viewpoint , which is necessary to understand and improve the deep learning with differential privacy . We notice one previous attempt by ( Chen et al. , 2020 ) , analyzing the DP-SGD convergence with an assumption of symmetric gradient distribution , which can be unrealistic and inapplicable to real datasets . Our Contributions • We explicitly characterize the general training dynamics of deep learning with DP gradient methods ( e.g. , DP-GD and DP-Adam ) . We show a fundamental influence of the DP training on the NTK matrix , which causes the convergence to worsen . This analysis leads to a convergence theory for the DP deep learning . • We propose a novel principle for designing the DP optimizers and thus develop a new global clipping method that provably enjoys desirable convergence behaviors . • We demonstrate via numerous experiments that DP optimizers with global clipping significantly improve the loss convergence . Interestingly , our clipping further effectively mitigates the calibration issue of existing DP classifiers , which usually exacebates the “ over-confidence ” in non-DP models . • Our global clipping has the same privacy guarantee and computational efficiency as the local clipping , which leads to a mix-up training strategy where local and global clippings are applied interchangeably . • Our global clipping is easy-to-code ( see Appendix D ) and generalizable to arbitrary optimizers , network architectures , loss functions , and tasks including federated learning . A quick preview of the comparison among the DP optimizers with the local and the global clipping is as follows : 2 Warmup : Convergence of Non-Private Gradient Descent . We start by reviewing the standard , non-DP Gradient Descent ( GD ) for arbitrary neural network and arbitrary loss , before we dive into the analysis of DP optimizers . In particular , we analyze the training dynamics using the neural tangent kernel ( NTK ) matrix1 . Suppose a neural network f is governed by weights w , with samples xi and labels yi ( i = 1 , ... , n ) . Denote the prediction by fi = f ( xi , w ) , and the per-sample loss by ` i = ` ( f ( xi , w ) , yi ) for some loss function ` . We define the objective function L to be the average of per-sample losses L ( w ) = 1n ∑n i=1 ` ( f ( xi , w ) , yi ) . The discrete gradient descent with a step size η , and the corresponding gradient flow2 are : w ( k + 1 ) = w ( k ) − η ∂L ∂w > , and ẇ ( t ) = − ∂L ∂w > = − 1 n ∑ i ∇w ` i ( t ) . 1We emphasize that our analysis works on any neural networks , not limited to the infinitely wide or over-parameterized ones . Put differently , we don ’ t assume the NTK matrix H to be deterministic nor nearly time-independent , as was the case in ( Arora et al. , 2019a ; Lee et al. , 2019 ; Du et al. , 2018 ; Allen-Zhu et al. , 2019 ; Zou et al. , 2020 ; Fort et al. , 2020 ; Arora et al. , 2019b ) . 2I.e. , the ordinary differential equation ( ODE ) describing the weight updates with infinitely small step size η → 0 in the continuous time . Applying the chain rules , we obtain the following general dynamics of the loss L , L̇ = ∂L ∂w ẇ = − ∂L ∂w ∂L ∂w > = −∂L ∂f ∂f ∂w ∂f ∂w > ∂L ∂f > = −∂L ∂f H ( t ) ∂L ∂f > , ( 2.1 ) where ∂L∂f = 1 n ( ∂ ` 1 ∂f1 , ... , ∂ ` n∂fn ) ∈ R 1×n , and the Gram matrix H ( t ) : = ∂f∂w ∂f ∂w > ∈ Rn×n is known as the NTK matrix , which is positive semi-definite and crucial to analyzing the convergence behavior . To give a concrete example , let ` be the MSE loss ` i ( w ) = ( f ( xi , w ) − yi ) 2 and LMSE = 1 n ∑ i ( fi − yi ) 2 , then L̇MSE = −4 ( f − y ) > H ( t ) ( f − y ) /n2 . Furthermore , if H ( t ) is positive definite , the MSE loss LMSE → 0 exponentially fast ( Du et al. , 2018 ; Allen-Zhu et al. , 2019 ; Zou et al. , 2020 ) , the cross-entropy loss LCE → 0 at rate O ( 1/t ) and any loss convex in the prediction L = ∑ i ` i/n converges to 0 ( Allen-Zhu et al. , 2019 ) . 3 Differentially Private Gradient Methods and Global Clipping . We now introduce the DP optimizers ( Google ; Facebook ) to train the DP neural networks . One popular optimizer is the DP-SGD ( Song et al. , 2013 ; Chaudhuri et al. , 2011 ; Abadi et al. , 2016 ; Bu et al. , 2019 ) in Algorithm 1 and more optimizers such as DP-Adam and DP-FedAvg ( McMahan et al. , 2017 ) for federated learning can be found in Appendix F. In contrast to the standard SGD , the DP-SGD has two unique steps : the per-sample clipping ( to bound the sensitivity of per-sample gradients ) and the random noise addition ( to guarantee the privacy of models ) , both are discussed in details via the Gaussian mechanism in Lemma B.1 . Algorithm 1 DP-SGD ( with local or global flat per-sample clipping ) Parameters : initial weights w0 , learning rate ηt , subsampling probability p , number of iterations T , noise scale σ , gradient norm bound R , maximum norm bound Z ≥ R. for t = 0 , . . . , T − 1 do Subsample a batch It ⊆ { 1 , . . . , n } from training set with probability p for i ∈ It do v ( i ) t ← ∇w ` ( f ( xi , wt ) , yi ) Option 1 : Clocal , i = min { 1 , R/‖v ( i ) t ‖2 } . Local clipping factor ( existing ) Option 2 : Cglobal , i ≡ { R/Z if ‖v ( i ) t ‖2 ≤ Z 0 if ‖v ( i ) t ‖2 > Z . Global clipping factor ( ours ) v̄ ( i ) t ← Ci · v ( i ) t . Clip the gradient V̄t ← ∑ i∈It v̄ ( i ) t . Sum over batch wt+1 ← wt − ηt|It| ( V̄t + σR · N ( 0 , I ) ) . Apply Gaussian mechanism and descend Although the per-sample clipping is widely applied in DP deep learning , its effect on convergence remains a mystery . Empirical observations have found that optimizers with the per-sample clipping ( even when no noise is present ) have much worse convergence and accuracy ( Abadi et al. , 2016 ; Bagdasaryan et al. , 2019 ) . In fact , the current form of clipping is heuristic and lacks theoretical understanding , especially when the noise addition is present . In what follows , we use C to denote Clocal or Cglobal when it is clear from the context . We propose and analyze a new clipping , namely the global clipping ( see Option 2 in Algorithm 1 ) , where the clipping operation takes place on all per-sample gradients that pass the screening procedure . From this viewpoint , the global clipping is a batch clipping instead of an individual clipping ( see Appendix F.6 for comparison with local clipping ) . More precisely , in local clipping , each per-sample gradient ∇w ` i compare its length to R and multiplied with a sample-specific clipping factor 0 < Ci ≤ 1 . In global clipping , only ∇w ` i with norm smaller than Z is used ( otherwise Ci = 0 ) and multiplied with a common clipping factor R/Z , which guarantees the sensitivity to be R as in local clipping . At a high level , the idea of global clipping is to preserve the gradient direction ( i.e . to remove the gradient bias ) while bounding the sensitivity during the clipping , which will guarantee the positive semi-definiteness of the NTK matrix via Theorem 2 .

Summary: The paper proposes a new clipping method for DP-SGD. It is shown that this new clipping method performs favorably empirically. Furthermore, the paper analyzes the clipping method using NTK.