Numerical Composition of Differential Privacy

√ k ) . This improves over the best prior method by [ KH21 ] which requires Ω̃ ( k1.5 ) running time . We demonstrate the utility of our algorithm by accurately computing the privacy loss of DP-SGD algorithm of Abadi et al . [ ACG+16 ] and showing that our algorithm speeds up the privacy computations by a few orders of magnitude compared to prior work , while maintaining similar accuracy . 1 Introduction . Differential privacy ( DP ) introduced by [ DMNS06 ] provides a provable and quantifiable guarantee of privacy when the results of an algorithm run on private data are made public . Formally , we can define an ( ε , δ ) -differentially private algorithm as follows . Definition 1.1 ( ( ε , δ ) -DP [ DMNS06 , DKM+06 ] ) . An algorithm M is ( ε , δ ) -DP if for any two neighboring databases D , D′ differing in exactly one user and any subset S of outputs , we have Pr [ M ( D ) ∈ S ] ≤ eε Pr [ M ( D′ ) ∈ S ] + δ . Intuitively , it says that looking at the outcome ofM , we can not tell whether it was run on D or D′ . Hence an adversary can not infer the existence of any particular user in the input database , and therefore can not learn any personal data of any particular user . DP algorithms have an important property called composition . SupposeM1 andM2 are DP algorithms and say M ( D ) = ( M1 ( D ) , M2 ( D ) ) , i.e. , M runs both the algorithms on D and outputs their results . Then M is also a DP algorithm . Proposition 1.2 ( Simple composition [ DKM+06 , DL09 ] ) . If M1 is ( ε1 , δ1 ) -DP and M2 is ( ε2 , δ2 ) DP , then M ( D ) = ( M1 ( D ) , M2 ( D ) ) is ( ε1 + ε2 , δ1 + δ2 ) -DP . This also holds under adaptive composition ( denoted by M = M2 ◦M1 ) , where M2 can look at both the database and the output of M1 ( here M ( D ) = ( M1 ( D ) , M2 ( D , M1 ( D ) ) ) ) . It turns out that both compositions enjoy much better DP guarantees than this simple composition rule . Let M be an ( ε , δ ) -DP algorithm and let M◦k denote the ( adaptive ) composition of M with itself k times . The naive composition rule shows that M◦k is ( kε , kδ ) -DP . This was significantly improved in [ DRV10 ] . Proposition 1.3 ( Advanced composition [ DRV10 , DR+14 ] ) . IfM is ( ε , δ ) -DP , thenM◦k is ( ε′ , kδ+ δ′ ) -DP where ε′ = ε √ 2k log ( 1 δ′ ) + kε ( eε − 1 ) . ∗Author ordering is alphabetical . Code is available at https : //github.com/microsoft/prv_ accountant . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Note that if ε = O ( 1√ k ) and δ = o ( 1 k ) , then M◦k satisfies ( Oδ′ ( 1 ) , δ′ ) -DP . Using simple composi- tion ( Proposition 1.2 ) , we can only claim that M◦k is ( O ( √ k ) , o ( 1 ) ) -DP . Thus advanced composition often results in √ k-factor savings in privacy which is significant in practice . The optimal DP guarantees for k-fold composition of an ( ε , δ ) -DP algorithm were finally obtained by [ KOV15 ] . For composing different algorithms , the situation is more complicated . If M1 , M2 , . . . , Mk are DP algorithms such that Mi is ( εi , δi ) -DP , then it is shown by [ MV16 ] that computing the exact DP guarantees for M = M1 ◦M2 ◦ · · · ◦Mk is # P-complete . They also give an algorithm to approximate the DP guarantees of M to desired accuracy η which runs in Õ ( k3ε̄ ( 1 + ε̄ ) η ) ( 1 ) time where ε̄ = ( ∑k i=1 εi ) /k . 2 If each εi ≈ 1√k ( so that M will satisfy reasonable privacy guarantees by advanced composition ) , then the running time is Õ ( k2.5/η ) . In most situations , DP algorithms come with a collection of ( ε , δ ) -DP guarantees , i.e. , for each value of ε , there exists δ such that the algorithm is ( ε , δ ) -DP . Definition 1.4 ( Privacy curve ) . A DP algorithm M is said to have privacy curve δ : R→ [ 0 , 1 ] , if for every ε ∈ R , M is ( ε , δ ( ε ) ) -DP . For example the privacy curve of a Gaussian mechanism ( with sensitivity 1 and noise scale σ ) is given by δ ( ε ) = Φ ( −εσ + 1/2σ ) −eεΦ ( −εσ − 1/2σ ) where Φ ( · ) is the Gaussian CDF [ BW18 ] . Suppose we want to compose several Gaussian mechanisms , which ( ε , δ ) -DP guarantee should we choose for each mechanism ? Any choice will lead to suboptimal DP guarantees for the final composition . Instead , we need a way to compose the privacy curves directly . This was suggested through the use of privacy region in [ KOV15 ] and explicitly studied in the f -DP framework of [ DRS19 ] . f -DP is a dual way ( and equivalent ) to look at the privacy curve δ ( ε ) . Independently , an algorithm called Privacy Buckets for approximately composing privacy curves using the notion of was initiated in [ MM18 ] . This algorithm depends on the notion of Privacy Loss Random Variable ( PRV ) [ DR16 ] , whose distribution is called Privacy Loss Distribution ( PLD ) . For any DP-algorithm , one can associate a PRV and the privacy curve of that algorithm can be easily obtained from the PRV . The really useful property of PRVs is that under adaptive composition , they just add up ; the PRV Y of the composition M = M1 ◦M2 ◦ · · · ◦Mk is given by Y = ∑k i=1 Yi where Yi is the PRV of Mi.3 Therefore , one can find the distribution of Y by the convolution of the distributions of Y1 , Y2 , . . . , Yk . In an important paper , [ KJH+20 ] proposed that one can speed up the convolutions using Fast Fourier Transform ( FFT ) . Explicit error bounds were obtained for the approximation obtained by their algorithm in [ KJH+20 , KJPH21 , KH21 ] . The running time of this algorithm was analyzed in [ KH21 ] where it was shown that the privacy curve δM ( ε ) of M = M1 ◦M2 ◦ · · · ◦Mk can be computed up to an additive error of δerror in time Õ ( k3ε̄ δerror ) , ( 2 ) if each algorithm Mi is satisfies ( εi , 0 ) -DP and ε̄ = 1k ∑k i=1 εi . Assuming that each εi ≈ 1√ k , we get Õ ( k2.5/δerror ) running time . Note that this is slightly worse than ( 1 ) , where the denominator η is the multiplicative error in δM . When composing the same algorithm with itself for k times , the running time can be improved to Õ ( k2ε̄ δerror ) , which is Õ ( k1.5 δerror ) when ε̄ = 1√ k . Moments Accountant and Renyi DP In an influential paper where they introduce Differentially Private Deep Learning , [ ACG+16 ] proposed a method called the Moments Accountant ( MA ) for giving an upper bound the privacy curve of a composition of DP algorithms . They applied their method to bound the privacy loss of differentially private Stochastic Gradient Descent ( DP-SGD ) algorithm which they introduced . Analyzing the privacy loss of DP-SGD involves composing the 2ε has an additive error of η and δ has a multiplicative error of η . 3 [ KJH+20 ] only state this for non-adaptive composition . In this paper we show how to extend this to adaptive composition as well . privacy curve of each iteration of training with itself k times , where k is the total of number of training iterations . Typical values of k range from 1000 to 300000 ( such as when training large models like GPT3 ) . The Moments Accountant was subsumed into the framework of Renyi Differential Privacy ( RDP ) introduced by [ Mir17 ] . The running time of these accountants are independent of k , but they only give an upper bound and can not approximate the privacy curve to arbitrary accuracy . DP-SGD is one of the most important DP algorithms in practice , because one can use it to train neural networks to achieve good privacy-vs-utility tradeoffs . Therefore obtaining accurate and tight privacy guarantees for DP-SGD is important . For example reducing ε from 2 to 1 , can mean that one can train the network for 4 times more epochs while staying within the same privacy budget . Therefore DP-SGD is one of the main motivations for this work . There are also situations when the PRVs do not have bounded moments and so Moments Accountant or Renyi DP can not be applied for analyzing privacy . An example of such an algorithm is the DP-SGD-JL algorithm of [ BGK+21 ] which uses numerical composition of PRVs to analyze privacy . GDP Accountant [ DRS19 , BDLS19 ] introduced the notion of Gaussian Differential Privacy ( GDP ) and used it to develop an accountant for DP-SGD . The accountant is based on central limit theorem and only gives an approximation to the true privacy curve , where the approximation gets better with k. But as we show in Figure 1 , GDP accountant can significantly underreport the true epsilon value . Several different notions of privacy were introduced for obtaining good upper bounds on the privacy curve of composition of DP algorithms such as Concentrated DP ( CDP ) [ DR16 , BS16 ] , Truncated CDP [ BDRS18 ] etc . None of these methods can approximate the privacy curve of compositions to arbitrary accuracy . The notion of f -DP introduced by [ DRS19 ] , allows for a lossless composition theorem , but computing the privacy curve of composition seems computationally hard and they do not give any algorithms for doing it . 1.1 Our Contributions . The main contribution of this work is a new algorithm with an improved analysis for computing the privacy curve of the composition of a large number of DP algorithms . Theorem 1.5 ( Informal version of Theorem 5.5 ) . SupposeM1 , M2 , . . . , Mk are DP algorithms . Then the privacy curve δM ( ε ) of adaptive composition M = M1 ◦M2 ◦ · · · ◦Mk can be approximated in time O εupper k1.5 log k √ log 1δerror εerror , ( 3 ) where εerror is the additive error in ε , δerror is the additive error in δ and εupper is an upper bound on max { εM ( δerror ) , maxi εMi ( δerror k ) } .4 If each Mi satisfies ( 1√ k , o ( 1 ) k ) -DP , then by advanced composition ( Proposition 1.3 ) , we can set εupper = O ( 1 ) . Therefore the running time of our algorithm in this case is Õ ( k1.5 √ log 1δerror εerror ) . We can save a factor of k , when we compose the same algorithm with itself k times . Theorem 1.6 . Suppose M is a DP algorithm . Then the privacy curve δM◦k ( ε ) of M ( adaptively ) composed with itself k times can be approximated in time O εupper k 12 log k √ log 1δerror εerror , ( 4 ) where εerror is the additive error in ε , δerror is the additive error in δ and εupper is an upper bound on max { εM◦k ( δerror ) , εM ( δerror k ) } . Thus we improve the state-of-the-art by at least a factor of k in running time . We also note that our algorithm improves the memory required by a factor of k. See Figure 1 for a comparison of 4εM ( δ ) is the inverse of δM ( ε ) . our algorithm with that of [ KJPH21 ] . Also note that RDP Accountant ( equivalent to the Moments Accountant ) significantly overestimates the true ε , while the GDP Accountant significantly underestimates the true ε . In contrast , the upper and lower bounds provided by our algorithm lie very close to each other . 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 10 15 20 25 30 35 40 45 50 ε DPSGD steps εup Ours εlow Ours εup [ KJPH21 ] εlow [ KJPH21 ] ( a ) Our algorithm gives much closer upper and lower bounds on the true privacy curve compared to [ KJPH21 ] , under the same mesh size of 4 × 10−5 . Our upper and lower bounds are nearly coinciding . 0 0.5 1 1.5 2 2.5 0 500 1000 1500 2000 ε DPSGD steps Ours Upper Ours Lower GDP RDP ( b ) Our algorithm can improve significantly over the RDP Accountant . We also see that GDP Accountant can significantly underreport the true ε . We have set εerror = 0.1 , δerror = δ/1000 here . Figure 1 : Case study on DP-SGD . Sampling probability p = 10−3 , noise scale σ = 0.8 , δ = 10−7 . Our Techniques Our algorithm ( also the prior work of [ KJH+20 ] ) proceeds by approximating the privacy loss random variables ( PRVs ) by truncating and discretizing them . We then use Fast Fourier Transform ( FFT ) to convolve the distributions efficiently . The main difference is in the approximation procedure and the error analysis . In the approximation procedure , we correct the approximation so that the expected value of the discretization matches with the expected value of the PRV . To analyze the approximation error , we introduce the concept of coupling approximation ( Definition 5.1 ) , which is a variant of Wasserstein ( optimal transport ) distance specifically tailored to this application . We first show that the approximation output by our algorithm to each privacy random variable is a good coupling approximation . We then show that when independent coupling approximations are added , cancellation happens between the errors due to Hoeffding bound , producing a much better coupling approximation than one naively expects from the triangle inequality . This allows us to choose the mesh size in our discretization to be ≈ 1√ k , whereas [ KH21 ] choose a mesh size of ≈ 1k . The other improvement is the truncation procedure . We give a tight tail bound of the PRVs ( Lemma 5.4 ) . This allows us to choose the domain size for in truncation to be ≈ Õ ( 1 ) , whereas [ KH21 ] choose ≈ Õ ( √ k ) . Both ideas together saves a factor of k in the run time and memory . For the analysis , the previous paper analyzes the discretization error by studying the stability of convolution . This leads to complicated calculations with the runtime linear in 1/δerror ( see ( 2 ) ) . Since δerror δ 1/N is required to give meaningful privacy guarantee ( N is the number of users ) , this term 1/δerror is huge . In this paper , we show various facts about how coupling approximation accumulates and use them to give a runtime depending only on √ log ( 1/δerror ) .

Summary: The paper studies numerical composition of DP from a computation complexity perspective. Previous FFT based algorithm is optimized in its selection of discretization grid and truncation threshold. The runtime is significantly improved from $k^{1.5}$ to $k^{0.5}$. It also show that the CLT approach can underreport $\varepsilon$ for very small $\delta$.