Numerical Composition of Differential Privacy

√ k ) . This improves over the best prior method by [ KH21 ] which requires Ω̃ ( k1.5 ) running time . We demonstrate the utility of our algorithm by accurately computing the privacy loss of DP-SGD algorithm of Abadi et al . [ ACG+16 ] and showing that our algorithm speeds up the privacy computations by a few orders of magnitude compared to prior work , while maintaining similar accuracy . 1 Introduction . Differential privacy ( DP ) introduced by [ DMNS06 ] provides a provable and quantifiable guarantee of privacy when the results of an algorithm run on private data are made public . Formally , we can define an ( ε , δ ) -differentially private algorithm as follows . Definition 1.1 ( ( ε , δ ) -DP [ DMNS06 , DKM+06 ] ) . An algorithm M is ( ε , δ ) -DP if for any two neighboring databases D , D′ differing in exactly one user and any subset S of outputs , we have Pr [ M ( D ) ∈ S ] ≤ eε Pr [ M ( D′ ) ∈ S ] + δ . Intuitively , it says that looking at the outcome ofM , we can not tell whether it was run on D or D′ . Hence an adversary can not infer the existence of any particular user in the input database , and therefore can not learn any personal data of any particular user . DP algorithms have an important property called composition . SupposeM1 andM2 are DP algorithms and say M ( D ) = ( M1 ( D ) , M2 ( D ) ) , i.e. , M runs both the algorithms on D and outputs their results . Then M is also a DP algorithm . Proposition 1.2 ( Simple composition [ DKM+06 , DL09 ] ) . If M1 is ( ε1 , δ1 ) -DP and M2 is ( ε2 , δ2 ) DP , then M ( D ) = ( M1 ( D ) , M2 ( D ) ) is ( ε1 + ε2 , δ1 + δ2 ) -DP . This also holds under adaptive composition ( denoted by M = M2 ◦M1 ) , where M2 can look at both the database and the output of M1 ( here M ( D ) = ( M1 ( D ) , M2 ( D , M1 ( D ) ) ) ) . It turns out that both compositions enjoy much better DP guarantees than this simple composition rule . Let M be an ( ε , δ ) -DP algorithm and let M◦k denote the ( adaptive ) composition of M with itself k times . The naive composition rule shows that M◦k is ( kε , kδ ) -DP . This was significantly improved in [ DRV10 ] . Proposition 1.3 ( Advanced composition [ DRV10 , DR+14 ] ) . IfM is ( ε , δ ) -DP , thenM◦k is ( ε′ , kδ+ δ′ ) -DP where ε′ = ε √ 2k log ( 1 δ′ ) + kε ( eε − 1 ) . ∗Author ordering is alphabetical . Code is available at https : //github.com/microsoft/prv_ accountant . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Note that if ε = O ( 1√ k ) and δ = o ( 1 k ) , then M◦k satisfies ( Oδ′ ( 1 ) , δ′ ) -DP . Using simple composi- tion ( Proposition 1.2 ) , we can only claim that M◦k is ( O ( √ k ) , o ( 1 ) ) -DP . Thus advanced composition often results in √ k-factor savings in privacy which is significant in practice . The optimal DP guarantees for k-fold composition of an ( ε , δ ) -DP algorithm were finally obtained by [ KOV15 ] . For composing different algorithms , the situation is more complicated . If M1 , M2 , . . . , Mk are DP algorithms such that Mi is ( εi , δi ) -DP , then it is shown by [ MV16 ] that computing the exact DP guarantees for M = M1 ◦M2 ◦ · · · ◦Mk is # P-complete . They also give an algorithm to approximate the DP guarantees of M to desired accuracy η which runs in Õ ( k3ε̄ ( 1 + ε̄ ) η ) ( 1 ) time where ε̄ = ( ∑k i=1 εi ) /k . 2 If each εi ≈ 1√k ( so that M will satisfy reasonable privacy guarantees by advanced composition ) , then the running time is Õ ( k2.5/η ) . In most situations , DP algorithms come with a collection of ( ε , δ ) -DP guarantees , i.e. , for each value of ε , there exists δ such that the algorithm is ( ε , δ ) -DP . Definition 1.4 ( Privacy curve ) . A DP algorithm M is said to have privacy curve δ : R→ [ 0 , 1 ] , if for every ε ∈ R , M is ( ε , δ ( ε ) ) -DP . For example the privacy curve of a Gaussian mechanism ( with sensitivity 1 and noise scale σ ) is given by δ ( ε ) = Φ ( −εσ + 1/2σ ) −eεΦ ( −εσ − 1/2σ ) where Φ ( · ) is the Gaussian CDF [ BW18 ] . Suppose we want to compose several Gaussian mechanisms , which ( ε , δ ) -DP guarantee should we choose for each mechanism ? Any choice will lead to suboptimal DP guarantees for the final composition . Instead , we need a way to compose the privacy curves directly . This was suggested through the use of privacy region in [ KOV15 ] and explicitly studied in the f -DP framework of [ DRS19 ] . f -DP is a dual way ( and equivalent ) to look at the privacy curve δ ( ε ) . Independently , an algorithm called Privacy Buckets for approximately composing privacy curves using the notion of was initiated in [ MM18 ] . This algorithm depends on the notion of Privacy Loss Random Variable ( PRV ) [ DR16 ] , whose distribution is called Privacy Loss Distribution ( PLD ) . For any DP-algorithm , one can associate a PRV and the privacy curve of that algorithm can be easily obtained from the PRV . The really useful property of PRVs is that under adaptive composition , they just add up ; the PRV Y of the composition M = M1 ◦M2 ◦ · · · ◦Mk is given by Y = ∑k i=1 Yi where Yi is the PRV of Mi.3 Therefore , one can find the distribution of Y by the convolution of the distributions of Y1 , Y2 , . . . , Yk . In an important paper , [ KJH+20 ] proposed that one can speed up the convolutions using Fast Fourier Transform ( FFT ) . Explicit error bounds were obtained for the approximation obtained by their algorithm in [ KJH+20 , KJPH21 , KH21 ] . The running time of this algorithm was analyzed in [ KH21 ] where it was shown that the privacy curve δM ( ε ) of M = M1 ◦M2 ◦ · · · ◦Mk can be computed up to an additive error of δerror in time Õ ( k3ε̄ δerror ) , ( 2 ) if each algorithm Mi is satisfies ( εi , 0 ) -DP and ε̄ = 1k ∑k i=1 εi . Assuming that each εi ≈ 1√ k , we get Õ ( k2.5/δerror ) running time . Note that this is slightly worse than ( 1 ) , where the denominator η is the multiplicative error in δM . When composing the same algorithm with itself for k times , the running time can be improved to Õ ( k2ε̄ δerror ) , which is Õ ( k1.5 δerror ) when ε̄ = 1√ k . Moments Accountant and Renyi DP In an influential paper where they introduce Differentially Private Deep Learning , [ ACG+16 ] proposed a method called the Moments Accountant ( MA ) for giving an upper bound the privacy curve of a composition of DP algorithms . They applied their method to bound the privacy loss of differentially private Stochastic Gradient Descent ( DP-SGD ) algorithm which they introduced . Analyzing the privacy loss of DP-SGD involves composing the 2ε has an additive error of η and δ has a multiplicative error of η . 3 [ KJH+20 ] only state this for non-adaptive composition . In this paper we show how to extend this to adaptive composition as well . privacy curve of each iteration of training with itself k times , where k is the total of number of training iterations . Typical values of k range from 1000 to 300000 ( such as when training large models like GPT3 ) . The Moments Accountant was subsumed into the framework of Renyi Differential Privacy ( RDP ) introduced by [ Mir17 ] . The running time of these accountants are independent of k , but they only give an upper bound and can not approximate the privacy curve to arbitrary accuracy . DP-SGD is one of the most important DP algorithms in practice , because one can use it to train neural networks to achieve good privacy-vs-utility tradeoffs . Therefore obtaining accurate and tight privacy guarantees for DP-SGD is important . For example reducing ε from 2 to 1 , can mean that one can train the network for 4 times more epochs while staying within the same privacy budget . Therefore DP-SGD is one of the main motivations for this work . There are also situations when the PRVs do not have bounded moments and so Moments Accountant or Renyi DP can not be applied for analyzing privacy . An example of such an algorithm is the DP-SGD-JL algorithm of [ BGK+21 ] which uses numerical composition of PRVs to analyze privacy . GDP Accountant [ DRS19 , BDLS19 ] introduced the notion of Gaussian Differential Privacy ( GDP ) and used it to develop an accountant for DP-SGD . The accountant is based on central limit theorem and only gives an approximation to the true privacy curve , where the approximation gets better with k. But as we show in Figure 1 , GDP accountant can significantly underreport the true epsilon value . Several different notions of privacy were introduced for obtaining good upper bounds on the privacy curve of composition of DP algorithms such as Concentrated DP ( CDP ) [ DR16 , BS16 ] , Truncated CDP [ BDRS18 ] etc . None of these methods can approximate the privacy curve of compositions to arbitrary accuracy . The notion of f -DP introduced by [ DRS19 ] , allows for a lossless composition theorem , but computing the privacy curve of composition seems computationally hard and they do not give any algorithms for doing it . 1.1 Our Contributions . The main contribution of this work is a new algorithm with an improved analysis for computing the privacy curve of the composition of a large number of DP algorithms . Theorem 1.5 ( Informal version of Theorem 5.5 ) . SupposeM1 , M2 , . . . , Mk are DP algorithms . Then the privacy curve δM ( ε ) of adaptive composition M = M1 ◦M2 ◦ · · · ◦Mk can be approximated in time O εupper k1.5 log k √ log 1δerror εerror , ( 3 ) where εerror is the additive error in ε , δerror is the additive error in δ and εupper is an upper bound on max { εM ( δerror ) , maxi εMi ( δerror k ) } .4 If each Mi satisfies ( 1√ k , o ( 1 ) k ) -DP , then by advanced composition ( Proposition 1.3 ) , we can set εupper = O ( 1 ) . Therefore the running time of our algorithm in this case is Õ ( k1.5 √ log 1δerror εerror ) . We can save a factor of k , when we compose the same algorithm with itself k times . Theorem 1.6 . Suppose M is a DP algorithm . Then the privacy curve δM◦k ( ε ) of M ( adaptively ) composed with itself k times can be approximated in time O εupper k 12 log k √ log 1δerror εerror , ( 4 ) where εerror is the additive error in ε , δerror is the additive error in δ and εupper is an upper bound on max { εM◦k ( δerror ) , εM ( δerror k ) } . Thus we improve the state-of-the-art by at least a factor of k in running time . We also note that our algorithm improves the memory required by a factor of k. See Figure 1 for a comparison of 4εM ( δ ) is the inverse of δM ( ε ) . our algorithm with that of [ KJPH21 ] . Also note that RDP Accountant ( equivalent to the Moments Accountant ) significantly overestimates the true ε , while the GDP Accountant significantly underestimates the true ε . In contrast , the upper and lower bounds provided by our algorithm lie very close to each other . 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 10 15 20 25 30 35 40 45 50 ε DPSGD steps εup Ours εlow Ours εup [ KJPH21 ] εlow [ KJPH21 ] ( a ) Our algorithm gives much closer upper and lower bounds on the true privacy curve compared to [ KJPH21 ] , under the same mesh size of 4 × 10−5 . Our upper and lower bounds are nearly coinciding . 0 0.5 1 1.5 2 2.5 0 500 1000 1500 2000 ε DPSGD steps Ours Upper Ours Lower GDP RDP ( b ) Our algorithm can improve significantly over the RDP Accountant . We also see that GDP Accountant can significantly underreport the true ε . We have set εerror = 0.1 , δerror = δ/1000 here . Figure 1 : Case study on DP-SGD . Sampling probability p = 10−3 , noise scale σ = 0.8 , δ = 10−7 . Our Techniques Our algorithm ( also the prior work of [ KJH+20 ] ) proceeds by approximating the privacy loss random variables ( PRVs ) by truncating and discretizing them . We then use Fast Fourier Transform ( FFT ) to convolve the distributions efficiently . The main difference is in the approximation procedure and the error analysis . In the approximation procedure , we correct the approximation so that the expected value of the discretization matches with the expected value of the PRV . To analyze the approximation error , we introduce the concept of coupling approximation ( Definition 5.1 ) , which is a variant of Wasserstein ( optimal transport ) distance specifically tailored to this application . We first show that the approximation output by our algorithm to each privacy random variable is a good coupling approximation . We then show that when independent coupling approximations are added , cancellation happens between the errors due to Hoeffding bound , producing a much better coupling approximation than one naively expects from the triangle inequality . This allows us to choose the mesh size in our discretization to be ≈ 1√ k , whereas [ KH21 ] choose a mesh size of ≈ 1k . The other improvement is the truncation procedure . We give a tight tail bound of the PRVs ( Lemma 5.4 ) . This allows us to choose the domain size for in truncation to be ≈ Õ ( 1 ) , whereas [ KH21 ] choose ≈ Õ ( √ k ) . Both ideas together saves a factor of k in the run time and memory . For the analysis , the previous paper analyzes the discretization error by studying the stability of convolution . This leads to complicated calculations with the runtime linear in 1/δerror ( see ( 2 ) ) . Since δerror δ 1/N is required to give meaningful privacy guarantee ( N is the number of users ) , this term 1/δerror is huge . In this paper , we show various facts about how coupling approximation accumulates and use them to give a runtime depending only on √ log ( 1/δerror ) .

Summary: The paper studies an interesting problem --- obtaining arbitrary tight privacy guarantees of composed mechanisms efficiently. On the one hand, the known efficient RDP-based moments accountant (O(1) computational time) cannot achieve an arbitrary accuracy. On the other hand, the Fast Fourier Transform (FFT) based approach provides explicit error bounds, but the computational complexity is O( k^1.5), where k is the number of compositions.  The paper provides an improved analysis for computing the privacy curve of the composition and saves a factor of k in the run time and memory.