On the Convergence and Calibration of Deep Learning with Differential Privacy

1 Introduction . Deep learning has achieved tremendous success in many applications that involve crowdsourced information , e.g. , face image , emails , financial status , and medical records . However , using such sensitive data raises severe privacy concerns on a range of image recognition , natural language processing and other tasks ( Cadwalladr & Graham-Harrison , 2018 ; Rocher et al. , 2019 ; Ohm , 2009 ; De Montjoye et al. , 2013 ; 2015 ) . For concrete examples , there are multiple successful privacy attacks on deep learning models , in which members in the dataset can be re-identified using the location or the purchase record , via the membership inference attack ( Shokri et al. , 2017 ; Carlini et al. , 2019 ) . In another example , the attackers can extract a person ’ s name , email address , phone number , and physical address from the billion-parameter GPT-2 ( Radford et al. , 2019 ) via the extraction attack ( Carlini et al. , 2020 ) . Therefore , many studies have applied differential privacy ( DP ) ( Dwork et al. , 2006 ; Dwork , 2008 ; Dwork et al. , 2014 ; Mironov , 2017 ; Duchi et al. , 2013 ; Dong et al. , 2019 ) , a mathematically rigorous approach , to protect against leakage of private information ( Abadi et al. , 2016 ; McSherry & Talwar , 2007 ; McMahan et al. , 2017 ; Geyer et al. , 2017 ) . To achieve this gold standard of privacy guarantee , since the seminal work ( Abadi et al. , 2016 ) , DP optimizers are applied to train the neural networks while preserving the accuracy of prediction . To name a few , researchers have proposed DP-SGD ( Abadi et al. , 2016 ; Bassily et al. , 2014 ) , DP-Adam ( Bu et al. , 2019 ) , DP-SGLD ( Wang et al. , 2015 ; Li et al. , 2019 ) , DP-FTRL ( Kairouz et al. , 2021 ) , DP-FedSGD and DP-FedAvg ( McMahan et al. , 2017 ) . Algorithmically speaking , DP optimizers generally have two extra steps in comparison to non-DP standard optimizers : the per-sample clipping and the random noise addition , so that DP optimizers descend in the direction of the averaged , clipped , noisy gradient ( see Figure 1 ) . These extra steps protect the resulting models against privacy attacks via the Gaussian mechanism ( Dwork et al. , 2014 , Theorem A.1 ) , at the expense of an empirical performance degradation compared to the non-DP deep learning , in terms of much slower convergence and lower utility . For example , state-of-the-art CIFAR10 accuracy with DP is ≈ 70 % without pre-training ( Papernot et al. , 2020 ) ( while the same non-DP networks can achieve 90 % accuracy ) and similar performance drops have been observed on facial images , tweets , and many other datasets ( Bagdasaryan et al. , 2019 ) . Empirically , many works have evaluated the effects of noise scale , batch size , clipping norm , learning rate , and network architecture on the privacy-accuracy trade-off ( Abadi et al. , 2016 ; Papernot et al. , 2020 ) . However , despite the prevalent usage of DP optimizers , little is known about its convergence behavior from a theoretical viewpoint , which is necessary to understand and improve the deep learning with differential privacy . We notice one previous attempt by ( Chen et al. , 2020 ) , analyzing the DP-SGD convergence with an assumption of symmetric gradient distribution , which can be unrealistic and inapplicable to real datasets . Our Contributions • We explicitly characterize the general training dynamics of deep learning with DP gradient methods ( e.g. , DP-GD and DP-Adam ) . We show a fundamental influence of the DP training on the NTK matrix , which causes the convergence to worsen . This analysis leads to a convergence theory for the DP deep learning . • We propose a novel principle for designing the DP optimizers and thus develop a new global clipping method that provably enjoys desirable convergence behaviors . • We demonstrate via numerous experiments that DP optimizers with global clipping significantly improve the loss convergence . Interestingly , our clipping further effectively mitigates the calibration issue of existing DP classifiers , which usually exacebates the “ over-confidence ” in non-DP models . • Our global clipping has the same privacy guarantee and computational efficiency as the local clipping , which leads to a mix-up training strategy where local and global clippings are applied interchangeably . • Our global clipping is easy-to-code ( see Appendix D ) and generalizable to arbitrary optimizers , network architectures , loss functions , and tasks including federated learning . A quick preview of the comparison among the DP optimizers with the local and the global clipping is as follows : 2 Warmup : Convergence of Non-Private Gradient Descent . We start by reviewing the standard , non-DP Gradient Descent ( GD ) for arbitrary neural network and arbitrary loss , before we dive into the analysis of DP optimizers . In particular , we analyze the training dynamics using the neural tangent kernel ( NTK ) matrix1 . Suppose a neural network f is governed by weights w , with samples xi and labels yi ( i = 1 , ... , n ) . Denote the prediction by fi = f ( xi , w ) , and the per-sample loss by ` i = ` ( f ( xi , w ) , yi ) for some loss function ` . We define the objective function L to be the average of per-sample losses L ( w ) = 1n ∑n i=1 ` ( f ( xi , w ) , yi ) . The discrete gradient descent with a step size η , and the corresponding gradient flow2 are : w ( k + 1 ) = w ( k ) − η ∂L ∂w > , and ẇ ( t ) = − ∂L ∂w > = − 1 n ∑ i ∇w ` i ( t ) . 1We emphasize that our analysis works on any neural networks , not limited to the infinitely wide or over-parameterized ones . Put differently , we don ’ t assume the NTK matrix H to be deterministic nor nearly time-independent , as was the case in ( Arora et al. , 2019a ; Lee et al. , 2019 ; Du et al. , 2018 ; Allen-Zhu et al. , 2019 ; Zou et al. , 2020 ; Fort et al. , 2020 ; Arora et al. , 2019b ) . 2I.e. , the ordinary differential equation ( ODE ) describing the weight updates with infinitely small step size η → 0 in the continuous time . Applying the chain rules , we obtain the following general dynamics of the loss L , L̇ = ∂L ∂w ẇ = − ∂L ∂w ∂L ∂w > = −∂L ∂f ∂f ∂w ∂f ∂w > ∂L ∂f > = −∂L ∂f H ( t ) ∂L ∂f > , ( 2.1 ) where ∂L∂f = 1 n ( ∂ ` 1 ∂f1 , ... , ∂ ` n∂fn ) ∈ R 1×n , and the Gram matrix H ( t ) : = ∂f∂w ∂f ∂w > ∈ Rn×n is known as the NTK matrix , which is positive semi-definite and crucial to analyzing the convergence behavior . To give a concrete example , let ` be the MSE loss ` i ( w ) = ( f ( xi , w ) − yi ) 2 and LMSE = 1 n ∑ i ( fi − yi ) 2 , then L̇MSE = −4 ( f − y ) > H ( t ) ( f − y ) /n2 . Furthermore , if H ( t ) is positive definite , the MSE loss LMSE → 0 exponentially fast ( Du et al. , 2018 ; Allen-Zhu et al. , 2019 ; Zou et al. , 2020 ) , the cross-entropy loss LCE → 0 at rate O ( 1/t ) and any loss convex in the prediction L = ∑ i ` i/n converges to 0 ( Allen-Zhu et al. , 2019 ) . 3 Differentially Private Gradient Methods and Global Clipping . We now introduce the DP optimizers ( Google ; Facebook ) to train the DP neural networks . One popular optimizer is the DP-SGD ( Song et al. , 2013 ; Chaudhuri et al. , 2011 ; Abadi et al. , 2016 ; Bu et al. , 2019 ) in Algorithm 1 and more optimizers such as DP-Adam and DP-FedAvg ( McMahan et al. , 2017 ) for federated learning can be found in Appendix F. In contrast to the standard SGD , the DP-SGD has two unique steps : the per-sample clipping ( to bound the sensitivity of per-sample gradients ) and the random noise addition ( to guarantee the privacy of models ) , both are discussed in details via the Gaussian mechanism in Lemma B.1 . Algorithm 1 DP-SGD ( with local or global flat per-sample clipping ) Parameters : initial weights w0 , learning rate ηt , subsampling probability p , number of iterations T , noise scale σ , gradient norm bound R , maximum norm bound Z ≥ R. for t = 0 , . . . , T − 1 do Subsample a batch It ⊆ { 1 , . . . , n } from training set with probability p for i ∈ It do v ( i ) t ← ∇w ` ( f ( xi , wt ) , yi ) Option 1 : Clocal , i = min { 1 , R/‖v ( i ) t ‖2 } . Local clipping factor ( existing ) Option 2 : Cglobal , i ≡ { R/Z if ‖v ( i ) t ‖2 ≤ Z 0 if ‖v ( i ) t ‖2 > Z . Global clipping factor ( ours ) v̄ ( i ) t ← Ci · v ( i ) t . Clip the gradient V̄t ← ∑ i∈It v̄ ( i ) t . Sum over batch wt+1 ← wt − ηt|It| ( V̄t + σR · N ( 0 , I ) ) . Apply Gaussian mechanism and descend Although the per-sample clipping is widely applied in DP deep learning , its effect on convergence remains a mystery . Empirical observations have found that optimizers with the per-sample clipping ( even when no noise is present ) have much worse convergence and accuracy ( Abadi et al. , 2016 ; Bagdasaryan et al. , 2019 ) . In fact , the current form of clipping is heuristic and lacks theoretical understanding , especially when the noise addition is present . In what follows , we use C to denote Clocal or Cglobal when it is clear from the context . We propose and analyze a new clipping , namely the global clipping ( see Option 2 in Algorithm 1 ) , where the clipping operation takes place on all per-sample gradients that pass the screening procedure . From this viewpoint , the global clipping is a batch clipping instead of an individual clipping ( see Appendix F.6 for comparison with local clipping ) . More precisely , in local clipping , each per-sample gradient ∇w ` i compare its length to R and multiplied with a sample-specific clipping factor 0 < Ci ≤ 1 . In global clipping , only ∇w ` i with norm smaller than Z is used ( otherwise Ci = 0 ) and multiplied with a common clipping factor R/Z , which guarantees the sensitivity to be R as in local clipping . At a high level , the idea of global clipping is to preserve the gradient direction ( i.e . to remove the gradient bias ) while bounding the sensitivity during the clipping , which will guarantee the positive semi-definiteness of the NTK matrix via Theorem 2 .

Summary: The paper analyzes convergence of DP-SGD using the NTK framework. The general idea is that the learning dynamics of SGD converge when the NTK matrix is positive-semi-definite.  They  characterize the effect of clipping and noise and Propose global clipping strategy. The global clipping idea consists of eliminating samples that have gradient norm bigger than some threshold and then applies a common clipping multiplier for each sample gradient. The authors claim that global clipping reduces gradient bias.    Gradient Clipping Summary: DP-SGD consist of the following steps: Over T rounds: 1) the algorithm samples a batch of samples from a dataset I_t. 2) it computes the gradient of each individual samples. 3) Each gradient sample is multiplied by a number such that the final gradient has norm bounded by R. 4) Finally, all clipped gradients are aggregated, noise is added to the average clipped gradient and the result is applied to update the model.  Difference between local and global clipping:  In local clipping, the gradient of each sample is multiplied by a factor specific to that sample such that the gradient  norm  of each sample is bounded by R. In global clipping, there is an additional parameter Z, each sample in a batch with gradient norm smaller than Z are multiplied by a constant factor R/Z and samples with gradient norm bigger than Z are not used. Therefore in global clipping each sample in a batch has gradient norm bounded by R. The intuition behind global clipping is that each gradient in a batch gets multiplied by a constant factor, this means that the direction of the aggregated gradient should be preserve.    Convergence Analysis: The convergence analysis uses the Neural Tangent Kernel (NTK) framework. The premise is that, the convergence of SGD is controlled by the NTK matrix in that the condition for convergence is that the NTK matrix is positive-semi-definite. The paper provides two main theorems: The first one claims that local clipping cannot guarantee convergence because local clipping can break the conditions for convergence under the NTK framework. The second theorem says that when the parameter Z is large then global clipping can guarantee convergence because each gradient is multiplied by a constant factor and does not affect the positive-semi-definitess of the NTK matrix.   Experiments: The paper empirically evaluates global clipping on the MNIST and CIFAR10 dataset. They