Sharper Utility Bounds for Differentially Private Models

) high probability excess population risk bound for differentially private al- gorithms under the assumptions G-Lipschitz , L-smooth , and Polyak-Łojasiewicz condition , based on gradient perturbation method . If we replace the properties G-Lipschitz and L-smooth by α-Hölder smoothness ( which can be used in nonsmooth setting ) , the high probability bound comes to O ( n− 2α 1+2α ) w.r.t n , which can not achieve O ( 1/n ) when α ∈ ( 0 , 1 ] . To solve this problem , we propose a variant of gradient perturbation method , max { 1 , g } -Normalized Gradient Perturbation ( m-NGP ) . We further show that by normalization , the high probability excess population risk bound under assumptions α-Hölder smooth and PolyakŁojasiewicz condition can achieveO ( √p n ) , which is the firstO ( 1/n ) high probability utility bound w.r.t n for differentially private algorithms under non-smooth conditions . Moreover , we evaluate the performance of the new proposed algorithm m-NGP , the experimental results show that m-NGP improves the performance ( measured by accuracy ) of the DP model over real datasets . It demonstrates that m-NGP improves the excess population risk bound and the accuracy of the DP model on real datasets simultaneously . 1 INTRODUCTION . Machine learning has been widely used and found effective in many fields in recent years ( Singha et al. , 2021 ; Swapna & Soman , 2021 ; Ponnusamy et al. , 2021 ) . When training machine learning models , tremendous data was collected , and the data often contains sensitive information of individuals , which may leakage personal privacy ( Shokri et al. , 2017 ; Carlini et al. , 2019 ) . Differential Privacy ( DP ) ( Dwork et al. , 2006 ; Dwork & Lei , 2009 ; Dwork et al. , 2014 ) is a theoretically rigorous tool to prevent sensitive information . It introduces random noise to the machine learning model and blocks adversaries from inferring any single individual included in the dataset by observing the model . The mathematical definition of DP is well accepted and relative technologies are performed by Google ( Erlingsson et al. , 2014 ) , Apple ( McMillan , 2016 ) and Microsoft ( Ding et al. , 2017 ) . As such , DP has attracted attention from the researchers and has been applied to numerous machine learning problems ( Ullman & Sealfon , 2019 ; Xu et al. , 2019 ; Bernstein & Sheldon , 2019 ; Wang & Xu , 2019 ; Heikkilä et al. , 2019 ; Kulkarni et al. , 2021 ; Bun et al. , 2021 ; Nguyen & Vullikanti , 2021 ) . There are mainly three approaches to guarantee differential privacy : output perturbation ( Chaudhuri et al. , 2011 ) , objective perturbation ( Chaudhuri et al. , 2011 ) , and gradient perturbation ( Song et al. , 2013 ) . Considering that gradient descent is a widely used optimization method , the gradient perturbation method can be used for a wide range of applications , and adding random noise to the gradient allows the model to escape local minima ( Raginsky et al. , 2017 ) , we focus on the gradient perturbation method to guarantee DP in this paper . In this paper , we aim to minimize the population risk , and measure the utility of the DP model by the excess population risk . To get the excess population risk , an important step is to analyze the generalization error ( the reason is demonstrated in Section 3 ) . Complexity theory ( Bartlett et al. , 2002 ) and algorithm stability theory ( Bousquet & Elisseeff , 2002 ) are popular tools to analyze the generalization error . On one hand , Chaudhuri et al . ( 2011 ) applied the complexity theory and achieved an O ( max { 1√ n , 2/3 √ p n } ) high probability excess population risk bound under the assumption of strongly convex ; Kifer et al . ( 2012 ) achieved O ( √p n ) expected excess population risk bound via complexity theory . On the other hand , the sharpest known high probability generalization bounds for DP algorithms analyzed via stability theory under different assumptions ( Wu et al. , 2017 ; Bassily et al. , 2019 ; Feldman et al. , 2020 ; Bassily et al. , 2020 ; Wang et al. , 2021 ) are O ( √p n + 1√ n ) or O ( 4√p√ n ) , containing an inevitable O ( 1√ n ) term , which is a bottleneck on the utility analysis . Thus , we are focusing on the following question , which is still an open problem : Can we achieve the high probability excess risk bounds with rate O ( √ p n ) for differentially private models via uniform stability ? This paper answers the question positively under more ( or different ) assumptions and provides the first high probability bound allowing an O ( √p n ) rate of convergence in the setting of DP . By in- troducing Generalized Bernstein condition ( Koltchinskii , 2006 ) , we remove the O ( 1√ n ) term in the generalization error and furthermore improve the high probability excess population risk bound . Comparing with previous high probability bounds , the improvement is approximately up toO ( √ n ) . CONTRIBUTIONS We first prove that by introducing Generalized Bernstein condition ( Koltchinskii , 2006 ) , under the assumptions G-Lipschitz , L-smooth , and Polyak-Łojasiewicz ( PL ) condition , the high probability excess population risk bound can be improved to O ( √p n ) . To the best of our knowledge , this is the first O ( √p n ) high probability excess population risk bound in the field of DP . Then , we relax the assumptions G-Lipschitz and L-smooth , by introducing α-Hölder smooth . Under these assumptions , we prove that the high probability excess population risk bound comes to O ( √p n −2α 1+2α ) . Considering that α ∈ ( 0 , 1 ] , the result can not achieve O ( √p n ) . To overcome the bottleneck , we design a variant of gradient perturbation method , called max { 1 , g } Normalized Gradient Perturbation ( m-NGP ) algorithm . Via this new proposed algorithm , we prove that under the assumptions α-Hölder smooth , PL condition , and generalized Bernstein condition , the high probability excess population risk bound can be improved to O ( √p n ) . To the best of our knowledge , this is the firstO ( √p n ) high probability excess population risk bound for non-smooth loss in the field of DP . Moreover , to evaluate the performance of our proposed max { 1 , g } -Normalized Gradient Perturbation algorithm , we perform experiments on real datasets , the experimental results show that m-NGP method also improves the accuracy of the DP model on real datasets . The rest of the paper is organized as follows . We discuss some related work in Section 2 . Some preliminaries are formally introduced in Section 3 . In Section 4 , we propose sharper utility bounds under different assumptions and design a variant of gradient perturbation method , max { 1 , g } Normalized Gradient Perturbation . The experimental results are shown in Section 5 . Finally , we conclude the paper in Section 6 . 2 RELATED WORK . Dwork et al . ( 2006 ) proposed the mathematical definition of DP for the first time . Then , it was developed to protect the privacy in the field of machine learning ( e.g . Empirical Risk Minimization ( ERM ) ) via output perturbation , objective perturbation , and gradient perturbation methods . For DP-ERM formulations , Chaudhuri et al . ( 2011 ) first proposed output perturbation and objective perturbation methods , and Song et al . ( 2013 ) first proposed the gradient perturbation method . Based on these works , Kifer et al . ( 2012 ) ; Bassily et al . ( 2014 ) ; Abadi et al . ( 2016 ) ; Wang et al . ( 2017 ) ; Zhang et al . ( 2017 ) ; Wu et al . ( 2017 ) ; Bassily et al . ( 2019 ) ; Feldman et al . ( 2020 ) ; Bassily et al . ( 2020 ) further improved the results under different assumptions . Among the works mentioned above , some of them only analyzed the privacy guarantees ( Song et al. , 2013 ; Abadi et al. , 2016 ) , some of them only discussed the excess empirical risk bound ( Wang et al. , 2017 ; Zhang et al. , 2017 ; Wu et al. , 2017 ) . Some works discussed the excess population risk under expectation , from different points of view , such as complexity theory , optimization theory , and stability theory : Kifer et al . ( 2012 ) achieved an O ( √p n ) expected excess population risk bound via complexity theory ; Bassily et al . ( 2014 ) achieved similar expected bound under convexity assumption , via optimization theory ; and Wang et al . ( 2019 ) proposed anO ( p log ( n ) 2 ) expected excess pop- ulation risk bound under non-convex condition , via Langevin Dynamics ( Gelfand & Mitter , 1991 ) and the stability of Gibbs algorithm . Considering that the high probability bound is more concerned by researchers , we focus on the high probability utility bound . Meanwhile , we concentrate on the stability theory in this paper . Among many notions of stability , uniform stability is arguably the most popular one , which yields exponential generalization bounds . Via uniform stability , the high probability excess population risk bounds under different assumptions given by previous works all contain anO ( 1√ n ) term , details can be found in Table 1 . The reason is that when analyzing the generalization error , the technical routes followed works Bousquet & Elisseeff ( 2002 ) ; Hardt et al . ( 2016 ) . In this paper , by introducing Generalized Bernstein condition ( Koltchinskii , 2006 ) , we remove the O ( 1√ n ) term from the generalization error , and further improve the excess population risk bound of DP models . The improved convergence rate is up toO ( √p n ) , which positively answers the question : Can the high probability excess population risk bound achieve O ( 1/n ) w.r.t n. The improvements are shown in Table 1 . Table 1 first shows that by adding more assumptions ( we assume the loss function to be Lipschitz , smooth , and satisfy Polyak-Łojasiewicz ( PL ) condition , while previous results require α-Hölder smoothness and convexity ) , we achieve a better high probability excess population risk bound , O ( √p n ) , which is state-of-the-art to the best of our knowledge . Then , we replace the Lipschitz and smooth property by α-Hölder smoothness and achieveO ( √p n 2α 1+2α ) high probability excess population risk bound , when α ∈ [ 12 , 1 ] , our result is better than previous ones , but it can not achieve the same bound ( O ( 1/n ) w.r.t n ) under the condition that the loss function is Lipschitz , smooth , and satisfies PL condition . To overcome it , we propose an algorithm called m-NGP , and achieve the O ( √p n ) result under the same assumptions : α-Hölder smooth and PL condition . Moreover , although it is hard to directly compare PL condition with convexity , PL condition can be applied to many non-convex conditions ( more information can be found in Section 4.2 ) . So , in this paper , we analyze the utility bound of DP algorithm under cases different from previous scenarios .

Summary: This paper analyzes the utility bounds of the gradient-perturbation based DP algorithm. They first provide DP by previous result (it is not the key point in this paper). Then, by applying the Generalized Bernstein condition, they give $O(p^{0.5}/(n\epsilon))$ high probability excess population risk bound under the properties Lipschitzness, smoothness, and PL condition. Furthermore, under the more general assumption (Holder smoothness), they analyze the utility bound but the result is not so good as before. So they propose an algorithm (called m-NGP) to improve it under the assumption Holder smooth, and it is claimed that the utility bound can be improved to $O(p^{0.5}/(n\epsilon))$. Their results are sharper than previous analyses in different settings: previous results require convex assumption but this paper requires PL condition. Moreover, experiments are performed to evaluate the accuracy of m-NGP.