Sharper Utility Bounds for Differentially Private Models

) high probability excess population risk bound for differentially private al- gorithms under the assumptions G-Lipschitz , L-smooth , and Polyak-Łojasiewicz condition , based on gradient perturbation method . If we replace the properties G-Lipschitz and L-smooth by α-Hölder smoothness ( which can be used in nonsmooth setting ) , the high probability bound comes to O ( n− 2α 1+2α ) w.r.t n , which can not achieve O ( 1/n ) when α ∈ ( 0 , 1 ] . To solve this problem , we propose a variant of gradient perturbation method , max { 1 , g } -Normalized Gradient Perturbation ( m-NGP ) . We further show that by normalization , the high probability excess population risk bound under assumptions α-Hölder smooth and PolyakŁojasiewicz condition can achieveO ( √p n ) , which is the firstO ( 1/n ) high probability utility bound w.r.t n for differentially private algorithms under non-smooth conditions . Moreover , we evaluate the performance of the new proposed algorithm m-NGP , the experimental results show that m-NGP improves the performance ( measured by accuracy ) of the DP model over real datasets . It demonstrates that m-NGP improves the excess population risk bound and the accuracy of the DP model on real datasets simultaneously . 1 INTRODUCTION . Machine learning has been widely used and found effective in many fields in recent years ( Singha et al. , 2021 ; Swapna & Soman , 2021 ; Ponnusamy et al. , 2021 ) . When training machine learning models , tremendous data was collected , and the data often contains sensitive information of individuals , which may leakage personal privacy ( Shokri et al. , 2017 ; Carlini et al. , 2019 ) . Differential Privacy ( DP ) ( Dwork et al. , 2006 ; Dwork & Lei , 2009 ; Dwork et al. , 2014 ) is a theoretically rigorous tool to prevent sensitive information . It introduces random noise to the machine learning model and blocks adversaries from inferring any single individual included in the dataset by observing the model . The mathematical definition of DP is well accepted and relative technologies are performed by Google ( Erlingsson et al. , 2014 ) , Apple ( McMillan , 2016 ) and Microsoft ( Ding et al. , 2017 ) . As such , DP has attracted attention from the researchers and has been applied to numerous machine learning problems ( Ullman & Sealfon , 2019 ; Xu et al. , 2019 ; Bernstein & Sheldon , 2019 ; Wang & Xu , 2019 ; Heikkilä et al. , 2019 ; Kulkarni et al. , 2021 ; Bun et al. , 2021 ; Nguyen & Vullikanti , 2021 ) . There are mainly three approaches to guarantee differential privacy : output perturbation ( Chaudhuri et al. , 2011 ) , objective perturbation ( Chaudhuri et al. , 2011 ) , and gradient perturbation ( Song et al. , 2013 ) . Considering that gradient descent is a widely used optimization method , the gradient perturbation method can be used for a wide range of applications , and adding random noise to the gradient allows the model to escape local minima ( Raginsky et al. , 2017 ) , we focus on the gradient perturbation method to guarantee DP in this paper . In this paper , we aim to minimize the population risk , and measure the utility of the DP model by the excess population risk . To get the excess population risk , an important step is to analyze the generalization error ( the reason is demonstrated in Section 3 ) . Complexity theory ( Bartlett et al. , 2002 ) and algorithm stability theory ( Bousquet & Elisseeff , 2002 ) are popular tools to analyze the generalization error . On one hand , Chaudhuri et al . ( 2011 ) applied the complexity theory and achieved an O ( max { 1√ n , 2/3 √ p n } ) high probability excess population risk bound under the assumption of strongly convex ; Kifer et al . ( 2012 ) achieved O ( √p n ) expected excess population risk bound via complexity theory . On the other hand , the sharpest known high probability generalization bounds for DP algorithms analyzed via stability theory under different assumptions ( Wu et al. , 2017 ; Bassily et al. , 2019 ; Feldman et al. , 2020 ; Bassily et al. , 2020 ; Wang et al. , 2021 ) are O ( √p n + 1√ n ) or O ( 4√p√ n ) , containing an inevitable O ( 1√ n ) term , which is a bottleneck on the utility analysis . Thus , we are focusing on the following question , which is still an open problem : Can we achieve the high probability excess risk bounds with rate O ( √ p n ) for differentially private models via uniform stability ? This paper answers the question positively under more ( or different ) assumptions and provides the first high probability bound allowing an O ( √p n ) rate of convergence in the setting of DP . By in- troducing Generalized Bernstein condition ( Koltchinskii , 2006 ) , we remove the O ( 1√ n ) term in the generalization error and furthermore improve the high probability excess population risk bound . Comparing with previous high probability bounds , the improvement is approximately up toO ( √ n ) . CONTRIBUTIONS We first prove that by introducing Generalized Bernstein condition ( Koltchinskii , 2006 ) , under the assumptions G-Lipschitz , L-smooth , and Polyak-Łojasiewicz ( PL ) condition , the high probability excess population risk bound can be improved to O ( √p n ) . To the best of our knowledge , this is the first O ( √p n ) high probability excess population risk bound in the field of DP . Then , we relax the assumptions G-Lipschitz and L-smooth , by introducing α-Hölder smooth . Under these assumptions , we prove that the high probability excess population risk bound comes to O ( √p n −2α 1+2α ) . Considering that α ∈ ( 0 , 1 ] , the result can not achieve O ( √p n ) . To overcome the bottleneck , we design a variant of gradient perturbation method , called max { 1 , g } Normalized Gradient Perturbation ( m-NGP ) algorithm . Via this new proposed algorithm , we prove that under the assumptions α-Hölder smooth , PL condition , and generalized Bernstein condition , the high probability excess population risk bound can be improved to O ( √p n ) . To the best of our knowledge , this is the firstO ( √p n ) high probability excess population risk bound for non-smooth loss in the field of DP . Moreover , to evaluate the performance of our proposed max { 1 , g } -Normalized Gradient Perturbation algorithm , we perform experiments on real datasets , the experimental results show that m-NGP method also improves the accuracy of the DP model on real datasets . The rest of the paper is organized as follows . We discuss some related work in Section 2 . Some preliminaries are formally introduced in Section 3 . In Section 4 , we propose sharper utility bounds under different assumptions and design a variant of gradient perturbation method , max { 1 , g } Normalized Gradient Perturbation . The experimental results are shown in Section 5 . Finally , we conclude the paper in Section 6 . 2 RELATED WORK . Dwork et al . ( 2006 ) proposed the mathematical definition of DP for the first time . Then , it was developed to protect the privacy in the field of machine learning ( e.g . Empirical Risk Minimization ( ERM ) ) via output perturbation , objective perturbation , and gradient perturbation methods . For DP-ERM formulations , Chaudhuri et al . ( 2011 ) first proposed output perturbation and objective perturbation methods , and Song et al . ( 2013 ) first proposed the gradient perturbation method . Based on these works , Kifer et al . ( 2012 ) ; Bassily et al . ( 2014 ) ; Abadi et al . ( 2016 ) ; Wang et al . ( 2017 ) ; Zhang et al . ( 2017 ) ; Wu et al . ( 2017 ) ; Bassily et al . ( 2019 ) ; Feldman et al . ( 2020 ) ; Bassily et al . ( 2020 ) further improved the results under different assumptions . Among the works mentioned above , some of them only analyzed the privacy guarantees ( Song et al. , 2013 ; Abadi et al. , 2016 ) , some of them only discussed the excess empirical risk bound ( Wang et al. , 2017 ; Zhang et al. , 2017 ; Wu et al. , 2017 ) . Some works discussed the excess population risk under expectation , from different points of view , such as complexity theory , optimization theory , and stability theory : Kifer et al . ( 2012 ) achieved an O ( √p n ) expected excess population risk bound via complexity theory ; Bassily et al . ( 2014 ) achieved similar expected bound under convexity assumption , via optimization theory ; and Wang et al . ( 2019 ) proposed anO ( p log ( n ) 2 ) expected excess pop- ulation risk bound under non-convex condition , via Langevin Dynamics ( Gelfand & Mitter , 1991 ) and the stability of Gibbs algorithm . Considering that the high probability bound is more concerned by researchers , we focus on the high probability utility bound . Meanwhile , we concentrate on the stability theory in this paper . Among many notions of stability , uniform stability is arguably the most popular one , which yields exponential generalization bounds . Via uniform stability , the high probability excess population risk bounds under different assumptions given by previous works all contain anO ( 1√ n ) term , details can be found in Table 1 . The reason is that when analyzing the generalization error , the technical routes followed works Bousquet & Elisseeff ( 2002 ) ; Hardt et al . ( 2016 ) . In this paper , by introducing Generalized Bernstein condition ( Koltchinskii , 2006 ) , we remove the O ( 1√ n ) term from the generalization error , and further improve the excess population risk bound of DP models . The improved convergence rate is up toO ( √p n ) , which positively answers the question : Can the high probability excess population risk bound achieve O ( 1/n ) w.r.t n. The improvements are shown in Table 1 . Table 1 first shows that by adding more assumptions ( we assume the loss function to be Lipschitz , smooth , and satisfy Polyak-Łojasiewicz ( PL ) condition , while previous results require α-Hölder smoothness and convexity ) , we achieve a better high probability excess population risk bound , O ( √p n ) , which is state-of-the-art to the best of our knowledge . Then , we replace the Lipschitz and smooth property by α-Hölder smoothness and achieveO ( √p n 2α 1+2α ) high probability excess population risk bound , when α ∈ [ 12 , 1 ] , our result is better than previous ones , but it can not achieve the same bound ( O ( 1/n ) w.r.t n ) under the condition that the loss function is Lipschitz , smooth , and satisfies PL condition . To overcome it , we propose an algorithm called m-NGP , and achieve the O ( √p n ) result under the same assumptions : α-Hölder smooth and PL condition . Moreover , although it is hard to directly compare PL condition with convexity , PL condition can be applied to many non-convex conditions ( more information can be found in Section 4.2 ) . So , in this paper , we analyze the utility bound of DP algorithm under cases different from previous scenarios .

Summary: The paper achieves high probability excess risk bound with rate O(1/n) w.r.t n for DP models via uniform stability by using Generalized Bernstein condition under G-Lipschitz, L-smooth, and PL condition. Then the authors expand the result to a more general case, only requiring α-Ho ̈lder smoothness, Polyak-Łojasiewicz condition, and generalized Bernstein condition. But the result is worse than before, so in order to get a better result, they propose m-NGP algorithm to achieve O(1/n) high probability bound w.r.t n under α-Ho ̈lder smoothness, Polyak-Łojasiewicz condition, and generalized Bernstein condition. The authors also show the experimental better accuracy results of m-NGP compared to traditional gradient perturbation method on real datasets.