Hyperparameter Tuning with Renyi Differential Privacy

For many differentially private algorithms , such as the prominent noisy stochastic gradient descent ( DP-SGD ) , the analysis needed to bound the privacy leakage of a single training run is well understood . However , few studies have reasoned about the privacy leakage resulting from the multiple training runs needed to fine tune the value of the training algorithm ’ s hyperparameters . In this work , we first illustrate how simply setting hyperparameters based on non-private training runs can leak private information . Motivated by this observation , we then provide privacy guarantees for hyperparameter search procedures within the framework of Renyi Differential Privacy . Our results improve and extend the work of Liu and Talwar ( STOC 2019 ) . Our analysis supports our previous observation that tuning hyperparameters does indeed leak private information , but we prove that , under certain assumptions , this leakage is modest , as long as each candidate training run needed to select hyperparameters is itself differentially private . 1 INTRODUCTION . Machine learning ( ML ) systems memorize training data and regurgitate excerpts from it when probed ( Carlini et al. , 2020 ) . If the training data includes sensitive personal information , then this presents an unacceptable privacy risk ( Shokri et al. , 2017 ) . It may however still be useful to apply machine learning to such data , e.g. , in the case of healthcare ( Kourou et al. , 2015 ; Wiens et al. , 2019 ) . This has led to a significant body of research on the development of privacy-preserving machine learning methods . Differential privacy ( DP ) ( Dwork et al. , 2006b ; a ) provides a robust and quantitative privacy guarantee . It has been widely accepted as the best framework for formally reasoning about the privacy guarantees of a machine learning algorithm . A popular method for ensuring DP is noisy ( stochastic ) gradient descent ( a.k.a . DP-SGD ) ( Song et al. , 2013 ; Bassily et al. , 2014 ; Abadi et al. , 2016 ) . DP-SGD differs from standard ( stochastic ) gradient descent in three ways . First , gradients are computed on a per-example basis rather than directly averaged across a minibatch of training examples . Second , each of these individual gradients is clipped to ensure its 2-norm is bounded . Third , Gaussian noise is added to the gradients as they are averaged and applied to update model parameters . These modifications bound the sensitivity of each update so that the added noise ensures differential privacy . The composition ( Dwork et al. , 2010 ) and privacy amplification by subsampling ( Balle et al. , 2018 ) properties of differential privacy thus imply that the overall training procedure is differentially private . We can compute tight privacy loss bounds for DP-SGD using techniques like the Moments Accountant ( Abadi et al. , 2016 ) or the closely related framework of Rényi DP ( Mironov , 2017 ; Mironov et al. , 2019 ) . Machine learning systems have hyperparameters , such as the learning rate , minibatch size , or choice of a regularizer to prevent overfitting . Details of the model architecture can also be treated as hyperparameters of the optimization problem . Furthermore , learning within the constraints of differential privacy may introduce additional hyperparameters , as illustrated in the DP-SGD optimizer by the 2-norm bound value for clipping , the scale of the Gaussian noise , and the choice of stopping time . Typically the training procedure is repeated many times with different hyperparameter settings in order to select the best setting , an operation known as hyperparameter tuning . This methodology implies that even if each run of the training procedure is privacy-preserving , we need to take into account the fact that the training procedure is repeated ( possibly many times ) when reasoning about the privacy of the overall learning procedure . Can the tuning of hyperparameters reveal private information ? This question has received remarkably little attention and , in practice , it is often ignored entirely . We study this question and provide both positive and negative answers . 1.1 OUR CONTRIBUTIONS . • We show that , under certain circumstances , the setting of hyperparamters can leak private information . Hyperparameters are a narrow channel for private information to leak through , but they can still reveal information about individuals if we are careless . Specifically , if we tune the hyperparameters in an entirely non-private fashion , then individual outliers can noticeably skew the optimal hyperparameter settings . This is sufficient to reveal the presence or absence of these outliers à la membership inference ( Shokri et al. , 2017 ) ; it shows that we must exercise care when setting hyperparameters . • We provide tools for ensuring that the selection of hyperparameters is differentially private . Specifically , if we repeat the training procedure multiple times ( with different hyperparameters ) and each repetition of the training procedure is differentially private on its own , then outputting the best repetition is differentially private . Of course , a basic version of such a result follows from the composition properties of differential privacy ( that is the fact that one can “ sum ” the privacy loss bounds of multiple differentially private analyses performed on the same data to bound the overall privacy loss from analyzing this data ) . However , we provide quantitatively sharper bounds . Specifically , our privacy loss bounds are either independent of the number of repetitions or grow logarithmically in the number of repetitions , whereas composition would give linear bounds . Rather than repeating the training procedure a fixed number of times , our results require repeating the training procedure a random number of times . The privacy guarantees depend on the distribution of the number of runs ; we consider several distributions and provide generic results . We discover a tradeoff between the privacy parameters and how heavy-tailed the distribution of the number of repetitions is . 1.2 BACKGROUND AND RELATED WORK . Differential privacy ( DP ) is a framework to reason about the privacy guarantees of randomized algorithms which analyze data ( Dwork et al. , 2006b ; a ) . An algorithm is said to be DP if its outputs on any pair of datasets that only differ on one individual ’ s data are indistinguishable . A bound on this indistinguishability serves as the quantification for privacy . Formally , a randomized algorithm M : Xn → Y is ( ε , δ ) -DP if for any inputs x , x′ ∈ Xn differing only on the addition , removal , or replacement of one individual ’ s records and for any subset of outputs S ⊆ Y , we have P [ M ( x ) ∈ S ] ≤ eεP [ M ( x′ ) ∈ S ] + δ . ( 1 ) Here , the parameter ε is known as the privacy loss bound – the smaller ε is , the stronger the privacy guarantee provided is , because it is hard for an adversary to distinguish the outputs of the algorithm on two adjacent inputs . The parameter δ is essentially the probability that the guarantee fails to hold . One of the key properties of DP is that it composes : running multiple independent DP algorithms is also DP and composition theorems allow us to bound the privacy parameters of such a sequence of mechanisms in terms of the individual mechanisms ’ privacy parameters ( Dwork & Roth , 2014 ) . There is a vast literature on differential privacy in machine learning . A popular tool is the DP-SGD optimizer ( Abadi et al. , 2016 ) . Because the noise added is Gaussian and DP-SGD applies the same ( differentially private ) training step sequentially , it is easier to reason about its privacy guarantees in the framework of Rényi differential privacy ( Mironov , 2017 ) . Rényi differential privacy ( RDP ) generalizes pure differential privacy ( with δ = 0 ) as follows . An algorithmM is said to be ( λ , ε ) -RDP with λ ≥ 1 and ε ≥ 0 , if for any adjacent inputs x , x′ Dλ ( M ( x ) ‖M ( x′ ) ) : = 1 λ− 1 log E Y←M ( x ) [ ( P [ M ( x ) = Y ] P [ M ( x′ ) = Y ] ) λ−1 ] ≤ ε , ( 2 ) where Dλ ( P‖Q ) is the Rényi divergence of order λ between distributions P andQ . In the framework of RDP , one obtains sharp and simple composition : If each individual mechanism Mi is ( λ , εi ) -RDP , then the composition of running all of the mechanisms on the data satisfies ( λ , ∑ i εi ) -RDP . For instance , the privacy analysis of DP-SGD first analyzes the individual training steps then applies composition . Note that it is common to keep track of multiple orders λ in the analysis . Thus ε should be thought of as a function ε ( λ ) , rather than a single number . In many cases , such as Gaussian noise addition , this is a linear function – i.e. , ε ( λ ) = ρ · λ for some ρ ∈ R – and such a linear bound yields the definition of zero-Concentrated DP with parameter ρ ( ρ-zCDP ) ( Bun & Steinke , 2016 ) . One could naïvely extend this composition-based approach to analyze the privacy of a training algorithm which involves hyperparameter tuning . Indeed , if each training run performed to evaluate one candidate set of hyperparameter values is DP , the overall procedure is also DP by composition over all the hyperparameter values tried . However , this would lead to very loose guarantees of privacy . Chaudhuri & Vinterbo ( 2013 ) were the first to obtain improved DP bounds for hyperparameter tuning , but their results require a stability property of the learning algorithm . The only prior work that has attempted to obtain tighter guarantees for DP hyperparameter tuning in a black-box fashion is the work of Liu & Talwar ( 2019 ) . Their work is the starting point for ours . Liu & Talwar ( 2019 ) show that , if we start with a ( ε , 0 ) -DP algorithm , repeatedly run it a random number of times following a geometric distribution , and finally return the best output produced by these runs , then this system satisfies ( 3ε , 0 ) -differential privacy.1 Liu & Talwar ( 2019 ) also consider algorithms satisfying ( ε , δ ) -DP for δ > 0 . However , their analysis is restricted to the ( ε , δ ) formulation of DP and they do not give any results for Rényi DP . This makes it difficult to apply these results to modern DP machine learning systems , such as models trained with DP-SGD . Our results directly improve on the results of Liu & Talwar ( 2019 ) . We show that replacing the geometric distribution on the number of repetitions in their result with the logarithmic distribution yields ( 2ε , 0 ) -differential privacy as the final result . We also consider other distributions on the number of repetitions , which give a spectrum of results . We simultaneously extend these results to the Rényi DP framework , which yields sharper privacy analyses . Independently Mohapatra et al . ( 2021 ) study adaptive hyperparameter tuning under DP with composition . In contrast , our results are for non-adaptive hyperparameter tuning – a.k.a . “ grid search. ” A closely related line of work is on the problem of private selection . Well-known algorithms for private selection include the exponential mechanism ( McSherry & Talwar , 2007 ) and the sparse vector technique ( Dwork et al. , 2009 ; Zhu & Wang , 2020 ) . However , this line of work assumes that there is a low-sensitivity function determining the quality of each of the options . This is usually not the case for hyperparameters . Our results simply treat the ML algorithm as a black box ; we only assume that its output is private and make no assumptions about how that output was generated . Our results also permit returning the entire trained model along with the selected hyperparameters .

Summary: The paper provides an considerable improvement to the DP analysis of hyperparameter tuning of DP algorithms (such as DP-SGD). The analysis is carried out using Rényi differential privacy (RDP), and the DP bounds are RDP bounds that contain the RDP parameters of the underlying mechanisms (that give the private candidates). Most importantly, the paper considerably improves the state-of-the-art of Liu and Talwar (2019). Also, a nice counterexample using SVMs is constructed, that shows the importance of this problem.