The Role of Adaptive Optimizers for Honest Private Hyperparameter Selection

Hyperparameter optimization is a ubiquitous challenge in machine learning , and1 the performance of a trained model depends crucially upon their effective se-2 lection . While a rich set of tools exist for this purpose , there are currently no3 practical hyperparameter selection methods under the constraint of differential4 privacy ( DP ) . We study honest hyperparameter selection for differentially private5 machine learning , in which the process of hyperparameter tuning is accounted for6 in the overall privacy budget . To this end , we i ) show that standard composition7 tools outperform more advanced techniques in many settings , ii ) empirically and8 theoretically demonstrate an intrinsic connection between the learning rate and9 clipping norm hyperparameters , iii ) show that adaptive optimizers like DPAdam10 enjoy a significant advantage in the process of honest hyperparameter tuning , and11 iv ) draw upon novel limiting behaviour of Adam in the DP setting to design a new12 and more efficient optimizer.13 1 Introduction14 Over the last several decades , the field of machine learning has flourished . However , training machine15 learning models frequently involves personal data , which leaves data contributors susceptible to16 privacy attacks . This isn ’ t purely hypothetical : recent results have shown that models are vulnerable17 to membership inference [ SSSS17 , CLE+19 , NSH19 ] and model inversion attacks [ FJR15 , SRS17 ] .18 The leading approaches for privacy-preserving machine learning are based on differential privacy19 ( DP ) [ DMNS06 ] . Informally , DP rigorously limits and masks the contribution that an individual20 datapoint can have on an algorithm ’ s output . To address the aforementioned issues , DP training21 procedures have been developed [ WM10 , BST14 , SCS13 , ACG+16 ] , which generally resemble non-22 private gradient-based methods , but with the incorporation of gradient clipping and noise injection.23 In both the private and non-private settings , hyperparameter selection is instrumental to achieving24 high accuracy . The most common methods are grid search or random search , both of which incur25 a computational overhead scaling with the number of hyperparameters under consideration . In26 the private setting , this issue is often magnified as most private training procedures introduce new27 hyperparameters . Regardless , and more importantly , hyperparameter tuning on a sensitive dataset28 also costs in terms of privacy , naively incurring a multiplicative cost which scales as the square root29 of the number of candidates ( based on composition properties of differential privacy [ KOV15 ] ) .30 Most prior works on private learning choose not to account for this cost [ ACG+16 , YLP+19 , TB21 ] ,31 focusing instead on demonstrating the accuracy achievable by private learning under idealized condi-32 tions , that is , if the best hyperparameters were somehow known ahead of time . Some works assume33 the presence of supplementary public data resembling the sensitive dataset [ AGD+20 , RTM+20 ] ,34 which may be freely used for hyperparameter tuning . Naturally , such public data may be scarce or35 nonexistent in settings where privacy is a concern , leaving practitioners with little guidance on how36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . to choose hyperparameters in practice . As explored in our paper , poor hyperparameter selection with37 standard private optimizers can have catastrophic effects on model accuracy.38 Hope is afforded by the success of adaptive optimizers in the non-private setting . The canonical39 example is Adam [ KB14 ] , which exploits moments of the gradients to adaptively and dynamically40 determine the learning rate . It works out of the box in many cases , providing accuracy comparable41 with tuned SGD . We navigate the different options available to a practitioner to solve the honest42 private hyperparameter tuning problem and ask , are there optimizers which provide strong privacy,43 require minimal hyperparameter tuning , and perform competitively with tuned counterparts ? 44 Our Contributions45 1 . We investigate techniques for private tuning of hyperparameters . We perform the first empirical46 evaluation of the proposed theoretical method of Liu and Talwar [ LT19 ] , and demonstrate that it can47 be relatively expensive . That is , in certain cases , one can tune over sufficiently many hyperparameters48 using standard composition tools such as moments accountant [ ACG+16 ] .49 2 . We empirically and theoretically demonstrate that two hyperparameters , the learning rate and50 clipping threshold , are intrinsically coupled for non-adaptive optimizers . While other hyperparameters51 and the model architecture are restricted by the scope of the task , privacy and utility targets , and52 computational resources , the learning rate and clipping norm have no a priori bounds . Since the53 resulting hyperparameter grid adds up to the privacy cost while tuning to achieve the model with the54 best utility , we explore leveraging adaptive optimizers to reduce the hyperparameter space.55 3 . We empirically demonstrate the DPAdam optimizer ( with default values for most hyperparameters ) ,56 can match the performance of tuned non-adaptive optimizers on a variety of datasets , thus enabling57 private learning with honest hyperparameter selection . This finding complements a prior claim58 of Papernot et al . [ PCS+20 ] , which suggests that a well-tuned DPSGD can outperform DPAdam.59 However , our findings show that this difference in performance is relatively insignificant . Furthermore,60 in the realistic setting where hyperparameter tuning must be accounted for in the privacy loss , we61 show that DPAdam is much more likely to produce non-catastrophic results.62 4 . We show that the adaptive learning rate of DPAdam converges to a static value . To leverage this,63 we introduce a new private optimizer , DPAdamWOSM that matches the performance of DPAdam64 without computing the second moments.65 1.1 Related Work66 Hyperparameter tuning plays a vital role in machine learning practice . In the non-private setting,67 ML practitioners use grid search , random search , Bayesian optimization techniques [ SSA13 ] or68 AutoML [ HZC21 ] techniques to tune their models . However , there hasn ’ t been much research69 on private hyperparameter tuning procedures due to the significant associated privacy costs . Each70 set of hyperparameter configuration results in a privacy-utility tradeoff . This tradeoff for multiple71 configurations can be captured by Pareto frontiers using multivariate Bayesian optimization over72 parameter dimensions [ AGD+20 ] . However , this method asks the model curator to query the73 dataset multiple times which requires non-private access to the dataset . There have been some74 end-to-end private tuning procedures [ CMS11 , CV13 , KGGW15 ] which work for a selected number75 of hyperparameter sets . These results work either in restricted settings for few combinations of76 candidates or under relaxations of differential privacy . The most relevant work to ours is an approach77 for private selection from private candidates [ LT19 ] . Their work provides two methods , one which78 outputs a candidate with accuracy greater than a given input threshold , and another which randomly79 stops and outputs the best candidate seen so far . The first approach is of limited utility in practice as80 it requires a prior accuracy bound for the dataset . The second variant incurs a considerable overhead81 in the privacy cost . We study this second approach and compare it with naive approaches based on82 Moments Accountant [ ACG+16 ] which would scale as the square root of the number of candidates.83 2 Problem Setup and Overview84 Consider a sensitive dataset D which lies beyond a privacy firewall and has n points of the form85 ( x1 , y1 ) , ( x2 , y2 ) , . . . , ( xn , yn ) where xi ∈ X is the feature vector of the ith point and yi ∈ Y is its86 desired output . Though our experiments are carried out in the supervised setting , all results can be87 translated to unsupervised setting as well . The dataset has been divided into two parts , the training set88 and the validation set . A trusted curator wants to train a machine learning model by making queries89 on the dataset with a total end-to-end training privacy budget of ( εf , δf ) such that the model can90 perform with high accuracy on the validation set . The curator wants to try multiple hyperparameter91 candidates for the model to figure out which candidate gives the maximum accuracy . However , as the92 model is private , each candidate requires multiple queries made on the dataset and all of them need to93 be accounted in the total privacy budget of ( εf , δf ) .94 Note that any validation accuracy must also be measured privately . Since this accuracy is a low-95 sensitivity query with a scalar output , and must only be computed once per choice of hyperparameters,96 the cost of this procedure is generally a lower order term versus the main training procedure . Thus for97 simplicity , we do not noise these validation accuracy queries . As we will see later , some optimizers98 require more candidates to tune and hence would also require more privacy budget than others.99 To tackle private hyperparameter selection , we first compare the available private tuning procedures100 in Section 3 . We show that the privacy cost for training a model depends on the hyperparameter101 grid size and standard composition theorems provide the best guarantees when the grid is small . In102 Section 4 , we investigate different optimizers to see how many candidates are required to output103 a good solution . In Section 4.1 we provide theoretical and empirical evidence to demonstrate an104 intrinsic coupling between two hyperparameters – the learning rate and clipping norm in DPSGD.105 We show that this coupling makes DPSGD sensitive to these parameter choices , which can drastically106 affect the validation accuracy . In Section 4.2 we demonstrate that an adaptive optimizer , DPAdam,107 translates well from the non-private setting and obviates tuning of the learning rate . In Section 5 , we108 empirically compare DPAdam with DPSGD and DPMomentum to show that DPAdam performs at par109 with less hyperparameter tuning . Finally , in Section 6 , we establish that DPAdam converges to a static110 learning rate in restricted settings , and unveil a new optimizer DPAdamWOSM which can leverage111 this converged value without computing the second moments . In the interest of space , we defer112 standard preliminaries such as DP definitions , hyperparameters , and optimizers to the appendix.113 3 The Cost of Privately Tuning DP Optimizers114 Effective hyperparameter tuning is crucial in extracting good utility from an optimizer . Unlike115 the non-private setting , DP optimizers typically i ) have more hyperparameters to tune ; ii ) require116 additional privacy budget for tuning . Existing work on DP optimizers acknowledge this problem ( e.g.,117 [ ACG+16 ] ) , but do not address the privacy cost incurred during hyperparameter tuning [ ACG+16,118 YLP+19 , TB21 ] . There are two main prior general-purpose approaches for private hyperparameter119 selection . The first performs composition via Moments Accountant [ ACG+16 ] , and the second is120 the algorithm of Liu and Talwar ( LT ) [ LT19 ] . The latter is a theoretical result , and to the best of our121 knowledge , has not been previously evaluated in practice . We investigate the privacy cost of these122 two techniques in practice and discuss situations in which each method is preferred.123 3.1 Hyperparameter Selection via [ LT19 ] 124 Liu and Talwar [ LT19 ] propose a random stopping algorithm ( LT ) to output a ‘ good ’ hyperparameter125 candidate from a pool ofK candidates , { x1 , . . . , xK } . They assume sampling access to a randomized126 mechanism Q ( D ) which samples i ∼ [ K ] , and returns the i-th candidate xi , and a score qi for this127 candidate . It is a random stopping algorithm , in which at every iteration , a candidate is picked from128 Q i.i.d . with replacement and a γ-biased coin is flipped to randomly stop the algorithm . When the129 algorithm stops , the candidate with the maximum score seen so far is outputted . In the approximate130 DP version of this algorithm , an extra parameter Υ is set to limit the total of number of iterations.131 The pseudocode of this algorithm is deferred to the appendix.132 Theorem 1 ( [ LT19 ] , Theorem 3.4 ) . Fix any γ ∈ [ 0 , 1 ] , δ2 > 0 and let Υ = 1γ log 1 δ2 . If Q is133 ( ε1 , δ1 ) -DP , then the LT algorithm is ( εf , δf ) -DP for εf = 3ε1 + 3 √ 2δ1 and δf = √ 2δ1Υ + δ2.134 Theorem 1 expresses the privacy cost of the algorithm in terms of the privacy cost of individual135 learners , and parameters of the algorithm itself . The δ2 parameter does not significantly affect the136 final epsilon εf of the algorithm and in practice , one can set it to a very small value ( 10−20 ) . Though137 a small value of δ2 has little effect on δf , it increases the hard stopping time of the algorithm , Υ.138 To understand the LT algorithm , we will compare the privacy costs of training a single hyperparameter139 candidate with a final εf , δf budget via LT and compare it with the privacy cost ε1 , δ1 of the underlying140 individual learner . This setting might seem unnatural for LT as it was designed to select from a pool141 of candidates but we choose this setting to show the minimum privacy cost overhead associated with142 LT and later show how the privacy cost changes for multiple candidates ( varying γ ) . To use LT , one143 needs to figure out the ε1 , δ1 via Theorem 1 using the final εf , δf values and in this case , γ = 1 ( as144 we have just one candidate ) . The individual learner is then trained using ε1 , δ1 budget.145 Due to the delicate balance of δf in Theorem 1 , one can see the δ1 comes out to be much smaller than146 δf . This change in δ1 results in a blowup of ε1 and hence , the final privacy cost of the LT algorithm147 ( 3ε1 + 3 √ 2δ1 ) , is much larger than what it would have been for learning one candidate without LT.148 We call this increase the blowup of privacy . We measure this blowup in Figure 1 ( left ) , for the setting149 of σ = 4 , L = 250 , T = 10 , 000 with varying dataset sizes ( n ) . It can be seen that for n = 5 , 000,150 the blowup is 4.8x whereas for for n = 950 , 000 , the blowup is almost 7.3x ( note the log scale on151 y-axis ) . Qualitatively similar trends persist for other choices of noise multiplier , lot size and iterations.152 We add more experiments to compare LT vs MA with varying candidate sizes in the appendix.153 Furthermore , we show that although LT entails a privacy blowup , decreasing γ , which corresponds154 to training more individual learners with ε1 , δ1 , doesn ’ t result in a significant difference in the final155 epsilon guaranteed by LT . In Figure 1 ( middle ) , we show the final epsilon cost for different dataset156 size and varying values of γ ∈ [ 0.001 , 0.01 , 0.1 , 1 ] . It is interesting to note here that with smaller157 γ values , one can train many candidates ( in expectation , 1γ ) for negligible additional privacy cost.158 The blowup to train 1 candidate ( γ = 1 ) versus 1 , 000 candidates ( γ = 0.001 ) increases from 33159 to 39 for n = 5 , 000 and increases from 0.49 to 0.69 , for n = 950 , 000 . This increase is minimal160 in comparison to advanced composition , which grows proportional to O ( √ k ) . However , another161 resource at play is the total training time , which is proportional to 1/γ ( i.e. , the total number of162 candidates ) . In summary , the LT algorithm is effective if an analyst has the privacy budget to afford163 the initial blowup , as the privacy cost of testing additional hyperparameters is insignificant.164 3.2 Hyperparameter Selection via Moments Accountant165 We learnt from the previous section that , LT permits selection from a large pool of hyperparameters166 ( depending on the γ value ) but incurs a constant privacy blowup . We compare LT with tuning167 using Moments Accountant ( MA ) . We notice using that with the same initial privacy blowup of168 the LT algorithm , MA is able to compose a considerable number of hyperparameter candidates . In169 Figure 1 ( right ) , we show the number of candidates that can be composed using MA for the minimum170 privacy cost for running the LT algorithm ( γ = 1 ) , for the setting of σ = 4 , L = 250 , T = 10 , 000171 and varying dataset size ( n ) on the x-axis . As the T and L is set constant , bigger n values in this172 graph correspond to fewer epochs of training and hence , worse utility . Depending on dataset size,173 MA can compose 14 candidates for n = 5000 and up to 175 candidates when n = 100000 . It is174 perhaps surprising how well a standard composition technique performs versus LT . This information175 can be highly valuable to a practitioner who has limited privacy budget . Qualitatively similar trends176 persist for other choices of batch size , noise multiplier , and iterations.177 From our experiments for both these tuning procedures , we conclude that while tuning with LT entails178 an initial privacy blowup , and the additional privacy cost for trying more candidates ( smaller γ ) is179 minimal . Even though this has an additional computation cost , it can be appealing when an analyst180 wants to try numerous hyperparameters . On the other hand , for the same overall privacy cost , MA181 can be used to compose a significant number of hyperparameter candidates . Additionally , MA allows182 access to all intermediate learners , whereas LT allows access to only the final output parameters . In183 the sequel , this conclusion will be useful in making the naive MA approach a more appealing tool for184 some settings ( e.g. , tighter privacy budgets ) .185 4 Tuning DP Optimizers186 We detail aspects of tuning both non-adaptive and adaptive optimizers . We start with tuning non-187 adaptive optimizers ( Section 4.1 ) . We theoretically and empirically demonstrate a connection between188 the learning rate and clipping threshold . We also establish that non-adaptive optimizers inevitably189 require searching over a large LR-clip grid to extract performant models . Adaptive optimizers forego190 this problem as they do not need to tune the hyperparameter dimension of learning rate . However,191 they introduce other hyperparameters that have known good choices in the non-private setting , and192 we empirically show that they are good candidates in the private setting ( Section 4.2 ) .193 4.1 Tuning DP non-adaptive optimizers194 While many hyperparameters are restricted due to computational and privacy/utility targets , the195 learning rate α and the clipping threshold C have no a priori bounds . In what follows , we show196 an interplay between these parameters by first theoretically analyzing the convergence of DPSGD.197 We then explore an illustrative experiment which demonstrates their entanglement . In the following198 theorem , we derive a bound on the expected excess risk of DPSGD and while doing so , show that the199 optimal learning rate , αopt , is proportional to the inverse of C. The proof appears in Appendix D.200 Theorem 2 . Let f be a convex and β-smooth function , and let x∗ = arg min x∈S f ( x ) . Let x0 be201 an arbitrary point in S , and xt+1 = ΠS ( xt − α ( gt + zt ) ) , where gt = min ( 1 , C‖∇f ( x ) ‖2 ) ∇f ( x ) 202 and zt ∼ N ( 0 , σ2C2 ) is the noise due to privacy . After T iterations , the optimal learning rate is203 αopt = R CT √ 1+σ2 , where E [ f ( 1T ∑T i xt ) − f ( x∗ ) ] ≤ RC √ 1+σ2√ T and R = E [ ‖x0 − x∗‖ ] .204 Though Theorem 2 gives a closed-form expression for the205 optimum learning rate , it is a function of the parameter R,206 which is unknown a priori to the analyst . Given constant207 T and σ , the optimal learning rate αopt is inversely propor-208 tional to the clipping normC . This is crucial information in209 practice because these parameters vary among datasets and210 are unbounded . This unboundedness property thus requires211 us to search over very large ranges of C and α when we212 have no prior knowledge of the dataset . It is natural to ask213 whether one can fix the clipping norm C and search only214 over a wide range for the learning rate α ( or vice versa ) .215 We explore this relationship experimentally , showing that216 fixing one of these two hyperparameters may often but not217 always result in an optimal model.218 In this experiment , we train a linear regression model on a219 10-dimensional synthetic dataset of input-label pairs ( x , y ) 220 sampled from a distribution D as follows : x1 , . . . , xd ∼221 U ( 0 , 1 ) , y = x ·w∗ , w∗ = 10 ·1d . We use the initialization222 w0 = 0 d and train for 100 iterations . In the non-private223 setting , this model converges quickly with any reasonable224 learning rate , but in the private setting , we notice that the225 training loss depends heavily on the choice of α and C. Figure 2 shows a heat map for the log226 training loss when trained on ( α , C ) pairs taken from a large grid consisting of [ 1 , 2 , 4 , 5 , 8 ] at scales227 of [ 10−4 , 10−3 , 10−2 , 10−1 , 100 , 101 ] . The best training is observed when the loss is 0 ( black pixels ) .228 We observe two fundamental phenomena from this figure . First , to achieve the best accuracy , α and229 C need to be tuned on a large grid spanning several orders of magnitude for each of these parameters.230 Second , multiple ( α , C ) pairs achieve the best accuracy and all lie on the same diagonal , validating231 our theory for an inverse relation between learning rate and clipping norm . As mentioned earlier , one232 might hypothesize that by setting the clipping norm C constant and tuning α ( corresponding to a233 vertical line in Figure 2 ) or vice versa , one could eliminate tuning a hyperparameter . However , note234 that not all C and α values correspond to the lowest loss . This phenomenon is evident by noticing235 that not all vertical or horizontal lines on this figure have black pixels . This happens , for example,236 at the extremes ( e.g. , at the top-right corner ) , but also for several intermediate and standard choices237 ( e.g. , C = 0.1 or 0.2 ) . Again , the analyst has no way of knowing this a priori . We conclude that to238 privately tune non-adaptive optimizers , we require a large grid of hyperparameter options.239 4.2 Tuning DP adaptive optimizers240 In the interest of reducing this space of tuning we turn to adaptive optimizers , where we can at least241 reduce one dimension of this search space . These approaches automatically adapt over the learning242 rate α , requiring us to tune only over the clipping norm C. But recall our key question : can we train243 models that perform competitively with the fine-tuned counterparts from DPSGD ? 244 Adam [ KB14 ] , the canonical adaptive optimizer introduces two new hyperparameters , which are the245 first and second moment exponential decay parameters ( β1 and β2 ) . In the non-private setting , these246 parameters are relatively insensitive , and default values of α = 0.001 , β1 = 0.9 , and β2 = 0.999 are247 recommended based on empirical findings , requiring no additional tuning for this hyperparameter248 triple . Hence before we compare DPAdam and DPSGD , we first find and establish such recommended249 values for this hyperparameter triple in the DP setting next , and then show that DPAdam with a small250 hyperparameter space performs competitively with DPSGD in Section 5.251 To establish default choices of α , β1 , and β2 for DPAdam , we evaluate this private optimizer over four252 diverse datasets ( details in Appendix B , Table 2 ) and two learning models including logistic regression253 and a neural network with one 100 neurons hidden layer ( TLNN ) . These selected datasets include254 both low-dimensional data ( where the number of samples greatly outnumbers the dimensionality ) and255 high-dimensional data ( where the number of samples and dimensionality are at same scale ) . Since256 we still have a large hyperparameter space to tune over , for the rest of this work , we fix a constant lot257 size ( L = 250 ) , and consider tuning over three different noise levels , σ ∈ [ 2 , 4 , 8 ] , so that we can258 study the effects of tuning the other hyperparameters more thoroughly . All experiments are repeated259 three times and averaged before reporting . Additionally , in this particular experiment since we focus260 on α , β1 , and β2 , we also fix the clipping threshold C = 0.5 , and T = 2500 iterations of training.261 For each dataset and model , we run DPAdam three times with hyperparameters ( α , β1 , β2 ) from the262 grids , α ∈ [ 0.001 , 0.05 , 0.01 , 0.2 , 0.5 ] , β1 , β2 ∈ [ 0.8 , 0.85 , 0.9 , 0.95 , 0.99.0.999 ] .263 We show that the default hyperparameter choice ( α , β1 , β2 ) of Adam in the non-private setting also264 works well for DPAdam . Figure 3 shows the boxplots of testing accuracies of DPAdam over the265 different hyperparameter choices . When α is 0.001 ( same as in the non-private setting ) , all the266 datasets and models have final testing accuracies ( marked in black ) close to the best possible ( and267 in most cases it is in fact the best ) accuracy . Furthermore , we also highlight the accuracy of the268 suggested default choice ( α = 0.001 , β1 = 0.9 , β2 = 0.999 ) using gold dots . Hence , for the ease of269 using DPAdam , we suggest the non-private default values for these parameters in the private setting270 as well and hence in all our subsequent experiments.271 5 Advantages of tuning using DPAdam272 In the non-private setting , adaptive optimizers like Adam enjoy a smaller hyperparameter tuning273 space than SGD . We ask two questions in this section . First , can DPAdam ( with little tuning ) achieve274 accuracy comparable to a well-tuned DPSGD ? Second , what is the privacy-accuracy tradeoff one275 incurs when using either of the two methods we detail in Section 3 for hyperparameter selection.276 To answer both questions , we compare DPAdam and DPSGD over the same set of datasets and277 models from the previous section . We report the accuracy of DPSGD with a range of learning rates278 and clipping values shown in Table 3 ( Appendix C ) , and the testing accuracy of DPAdam with default279 parameter choice from Section 4.2 ( α = 0.001 , β1 = 0.9 , β2 = 0.999 ) and a range of clipping values280 C in Table 3 . In total , DPSGD has 40 candidates to tune over , and DPAdam has 4 . This is because281 we have shown in Section 4.1 that DPSGD needs a wide grid to obtain the best accuracy when data282 distributions are unknown . Additionally , we also consider the DPMomentum optimizer . Similar to283 how we searched for default tuning choices for DPAdam in Section 4.2 , we investigate if there exists284 a qualitatively good choice for the momentum hyperparameter , and unfortunately our results show285 that there is no such choice . We detail this process in the supplement.286 In order to show the comparison from both sides of the privacy-accuracy tradeoff , we compare the287 three optimizers through i ) the privacy cost when extracting the best accuracy from these optimizers,288 and ii ) the accuracy one would obtain from them under the tight privacy constraints.289 5.1 Prioritizing Accuracy290 For brevity , we show experiments for σ = 4 in Figure 4 , results for other values of σ are displayed in291 the supplement . For each dataset and model , we train three times for each hyperparameter candidate292 and report the max every 100 iterations , corresponding to the dark lines for each optimizer . We note293 that their maxima are extremely similar . However , Table 1 shows the final privacy costs incurred294 by each of these max accuracy lines , and reflects our claims from Section 3.1 that using fewer295 hyperparameter candidates and composing privacy via MA gives a much tighter privacy guarantee.296 5.2 Prioritizing Privacy297 Additionally in Figure 4 , DPSGD and DPMomentum have pastel dotted lines corresponding to their298 mean accuracy attained using the MA composition that provides the tightest privacy guarantees for299 DPAdam . These pastel lines are the mean accuracies ( with 95 % CI ) from 100 repetitions of this300 experiment . Since DPAdam has only 4 hyperparameter candidates , for this experiment , we sample 4301 of the candidates at random for DPSGD and DPMomentum so that they all incur the same privacy302 cost . Since the candidate pool is significantly larger for DPSGD and DPMomentum , we additionally303 scrutinize the parameter grid for them and prune the learning rates that perform poorly . Our pruning304 process ( detailed in the supplement ) is quite generous , and favours minimizing the hyperparameter305 space of DPSGD and DPMomentum as much possible . 1 Despite the pruning advantage we see that306 these optimizers perform subpar than DPAdam when constrained with tight privacy requirements.307 6 DPAdam without second moment : DPAdamWOSM308 As illustrated in the previous section , adaptivity can indeed be a boon , enabling DPAdam to match309 the performance of tuned DPSGD while consuming roughly a third of the privacy budget as seen310 in Table 1 . However , in addition to a decaying average of the past gradient updates , DPAdam311 also requires maintaining a decaying average of their second moments . In this section , we design312 DPAdamWOSM , a new DP optimizer that operates only using a decaying average of past gradients,313 as well as eliminates the need to tune the learning rate parameter . We achieve this by analyzing the314 convergence behavior of the second-moment decaying average in DPAdam in regimes where the scale315 of noise added is much higher than the scale of the clipped gradients . Setting the effective step size316 ( ESS ) of DPAdam to the converged constant , and removing all computations related to the second-317 moment updates , results in DPAdamWOSM . We empirically demonstrate that DPAdamWOSM318 matches the utility of DPAdam , while requiring less computation than DPAdam.319 Observe that removing the second-moment updates from DPAdam reduces it to DPMomentum with320 one additional feature : bias-correction to the first-moment decaying average , which DPAdam does321 to account for its initialization at the origin . While the resultant optimizer still requires tuning the322 learning rate ( in addition to other hyperparameters like the clipping threshold ) , DPAdamWOSM can323 be viewed as self-tuning the learning rate by fixing it to the converged effective step size in DPAdam.324 6.1 Effective step size ( ESS ) in DPAdam325 DPAdam produces results with a smaller variance than DPSGD due to its adaptive learning rate.326 To understand this phenomenon better , we look closely into the update step of DPAdam [ KB14 ] .327 DPAdam being an adaptive optimizer picks per-parameter ESS as α√ v̂t+ξ , which is the base learning328 rate α scaled by the second moment of the individual parameter gradients . We notice that when329 g → 0 , the ESS for DPAdam converges for the first moment gradient , which innately accounts for330 the clip bound one is training with . This may happen at later iterations , when the model is close to its331 minima and the gradients get close to zero.332 Theorem 3 . The effective step size ( ESS ) for DPAdam with g → 0 converges to ESS∗ = α ( σC/L ) +ξ .333 Proof . Recall that the average noisy gradient over a lot is g̃ = g +N ( 0 , σ2C2 ) /L . We now look at334 the effect of this noisy gradient on the effective step size ( ESS ) of DPAdam . As g → 0 , the second335 moment of DPAdam converges to σ 2C2 L2 . This gives us the converging value for ESS:336 ESS∗ = α√ v̂t + ξ = α√ σ2C2 L2 + ξ = α ( σC/L ) + ξ 1Note , pruning itself is of course unfair ; the intent was to design a DP optimizer that can be used on any data distributions that we have no prior knowledge of . To do so with DPSGD one would have to consider a significantly wide range of ( α , C ) pairs to cover ‘ good ’ candidates as we illustrated in Section 4.1 Theorem 3 gives a closed form expression that ESS converges to . We can use this value in place337 of α√ v̂t+ξ in the update step from the inception of the learning process . Since the second-moment338 updates ( e.g. , v̂t ) are not used anymore , removing them results in our new optimizer DPAdamWOSM.339 We provide a pseudo-code for DPAdamWOSM in the appendix.340 6.2 Comparing Adaptive Optimizers341 We evaluate DPAdamWOSM by running it alongside DPAdam and ADADP with the same hyperpa-342 rameter grid in the appendix . For brevity , we show experiments on σ = 4 and others appear in the343 supplement . In Figure 5 , we show the maximum and median accuracy curves for all the optimizers.344 We display the median accuracy curves ( shown in dotted ) , as an indicator of the quality of the entire345 pool of hyperparameter candidates for a given optimizer ; which in this case is strictly over the choices346 of clip . The max lines for ADADP lies beneath DPAdam and DPAdamWOSM for all dataset except347 Adult . Also , the max accuracy line for DPAdamWOSM runs alongside DPAdam which means that348 it can perform as good as DPAdam throughout training . The median line for DPAdamWOSM also349 performs alongside DPAdam and in some cases is able to beat it ( e.g , the median for DPAdamWOSM350 for MNIST-LR and MNIST-TLNN lies above the median line of DPAdam ) . This occurrence is seen351 because DPAdamWOSM uses the converged ESS from the first iteration of training.352 7 Conclusion353 In this paper , we performed a thorough investigation of honest hyperparameter selection for DP354 Optimizers . We compared two existing private methods , LT and MA to search for hyperparameter355 candidates and showed that , the former incurs a significant privacy cost but can compose a great many356 candidates , while the latter is helpful when the number of candidates is small . Next , we explored357 connections between the clipping norm and the step size hyperparameter to show an inverse relation-358 ship between them . Additionally , we compared non-adaptive and adaptive optimizers , demonstrating359 that the latter typically achieves more consistent performance over a variety of hyperparameter360 settings . This can be vital for applications where public data is scarce , resulting in difficulties when361 tuning hyperparameters . Finally , we brought to light that DPAdam converges to a static learning rate362 when the noise starts dominating the gradients . This insight allowed us to derive a novel optimizer363 DPAdamWOSM , a variant of DPAdam which avoids the second-moment computation and enjoys364 better accuracy especially at earlier iterations . Future work remains to investigate further implications365 of these results to provide tuning-free end-to-end private ML optimizers.366 References367 [ ACG+16 ] Martin Abadi , Andy Chu , Ian Goodfellow , H. Brendan McMahan , Ilya Mironov , Kunal368 Talwar , and Li Zhang . Deep learning with differential privacy . In Proceedings of the369 2016 ACM SIGSAC Conference on Computer and Communications Security ( CCS ’ 16 ) ,370 2016.371 [ AGD+20 ] Brendan Avent , Javier González , Tom Diethe , Andrei Paleyes , and Borja Balle . Auto-372 matic discovery of privacy–utility pareto fronts . Proceedings on Privacy Enhancing373 Technologies , 2020 ( 4 ) :5–23 , 2020.374 [ BBG18 ] Borja Balle , Gilles Barthe , and Marco Gaboardi . Privacy amplification by subsampling:375 Tight analyses via couplings and divergences . In Advances in Neural Information376 Processing Systems 31 , NeurIPS ’ 18 , pages 6277–6287 . Curran Associates , Inc. , 2018.377 [ BST14 ] Raef Bassily , Adam D. Smith , and Abhradeep Thakurta . Private empirical risk min-378 imization : Efficient algorithms and tight error bounds . In FOCS , pages 464–473,379 2014.380 [ CLE+19 ] Nicholas Carlini , Chang Liu , Úlfar Erlingsson , Jernej Kos , and Dawn Song . The secret381 sharer : Evaluating and testing unintended memorization in neural networks . In 28th382 USENIX Security Symposium ( USENIX Security 19 ) , pages 267–284 , Santa Clara , CA,383 August 2019 . USENIX Association.384 [ CMS11 ] Kamalika Chaudhuri , Claire Monteleoni , and Anand D Sarwate . Differentially private385 empirical risk minimization . Journal of Machine Learning Research , 12 ( 3 ) , 2011.386 [ CV13 ] Kamalika Chaudhuri and Staal A Vinterbo . A stability-based validation procedure for387 differentially private machine learning . Advances in Neural Information Processing388 Systems , 26:2652–2660 , 2013.389 [ DHS11 ] John Duchi , Elad Hazan , and Yoram Singer . Adaptive subgradient methods for online390 learning and stochastic optimization . Journal of machine learning research , 12 ( 7 ) ,391 2011.392 [ DKM+06 ] Cynthia Dwork , Krishnaram Kenthapadi , Frank McSherry , Ilya Mironov , and Moni393 Naor . Our data , ourselves : Privacy via distributed noise generation . In EUROCRYPT,394 volume 4004 , pages 486–503 . Springer , 2006.395 [ DMNS06 ] Cynthia Dwork , Frank McSherry , Kobbi Nissim , and Adam Smith . Calibrating noise to396 sensitivity in private data analysis . In Proceedings of the 3rd Conference on Theory of397 Cryptography , TCC ’ 06 , pages 265–284 , 2006.398 [ FJR15 ] Matt Fredrikson , Somesh Jha , and Thomas Ristenpart . Model inversion attacks that399 exploit confidence information and basic countermeasures . In Proceedings of the 22nd400 ACM SIGSAC Conference on Computer and Communications Security ( CCS ’ 15 ) , 2015.401 [ GAYB17 ] R. Gylberth , R. Adnan , S. Yazid , and T. Basaruddin . Differentially private optimization402 algorithms for deep neural networks . In 2017 International Conference on Advanced403 Computer Science and Information Systems ( ICACSIS ) , pages 387–394 , 2017.404 [ HSS12 ] Geoffrey Hinton , Nitish Srivastava , and Kevin Swersky . Neural networks for machine405 learning lecture 6a overview of mini-batch gradient descent . Cited on , 14 ( 8 ) , 2012.406 [ HZC21 ] Xin He , Kaiyong Zhao , and Xiaowen Chu . Automl : A survey of the state-of-the-art.407 Knowledge-Based Systems , 212:106622 , 2021.408 [ KB14 ] Diederik P Kingma and Jimmy Ba . Adam : A method for stochastic optimization . arXiv409 preprint arXiv:1412.6980 , 2014.410 [ KGGW15 ] Matt Kusner , Jacob Gardner , Roman Garnett , and Kilian Weinberger . Differentially411 private bayesian optimization . In International conference on machine learning , pages412 918–927 . PMLR , 2015.413 [ KH20 ] Antti Koskela and Antti Honkela . Learning rate adaptation for differentially private414 learning . In International Conference on Artificial Intelligence and Statistics , pages415 2465–2475 . PMLR , 2020.416 [ KOV15 ] Peter Kairouz , Sewoong Oh , and Pramod Viswanath . The composition theorem for417 differential privacy . In International conference on machine learning , pages 1376–1385.418 PMLR , 2015.419 [ LT19 ] Jingcheng Liu and Kunal Talwar . Private selection from private candidates . In Pro-420 ceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , pages421 298–309 , 2019.422 [ MAE+18 ] H Brendan McMahan , Galen Andrew , Ulfar Erlingsson , Steve Chien , Ilya Mironov,423 Nicolas Papernot , and Peter Kairouz . A general approach to adding differential privacy424 to iterative training procedures . arXiv preprint arXiv:1812.06210 , 2018.425 [ NSH19 ] M. Nasr , R. Shokri , and A. Houmansadr . Comprehensive privacy analysis of deep learn-426 ing : Passive and active white-box inference attacks against centralized and federated427 learning . In 2019 IEEE Symposium on Security and Privacy ( SP ’ 19 ) , 2019.428 [ PCS+20 ] Nicolas Papernot , Steve Chien , Shuang Song , Abhradeep Thakurta , and Ulfar Erlingsson.429 Making the shoe fit : Architectures , initializations , and tuning for learning with privacy,430 2020.431 [ Qia99 ] Ning Qian . On the momentum term in gradient descent learning algorithms . Neural432 Networks , 1999.433 [ RHW86 ] David E Rumelhart , Geoffrey E Hinton , and Ronald J Williams . Learning representations434 by back-propagating errors . nature , 323 ( 6088 ) :533–536 , 1986.435 [ RTM+20 ] Swaroop Ramaswamy , Om Thakkar , Rajiv Mathews , Galen Andrew , H Brendan McMa-436 han , and Françoise Beaufays . Training production language models without memorizing437 user data . arXiv preprint arXiv:2009.10031 , 2020.438 [ SCS13 ] Shuang Song , Kamalika Chaudhuri , and Anand D. Sarwate . Stochastic gradient descent439 with differentially private updates . In GlobalSIP , pages 245–248 , 2013.440 [ SRS17 ] Congzheng Song , Thomas Ristenpart , and Vitaly Shmatikov . Machine learning models441 that remember too much . In Proceedings of the 2017 ACM SIGSAC Conference on442 Computer and Communications Security ( CCS ’ 17 ) , 2017.443 [ SSA13 ] Kevin Swersky , Jasper Snoek , and Ryan P Adams . Multi-task bayesian optimization.444 Advances in Neural Information Processing Systems , 26:2004–2012 , 2013.445 [ SSSS17 ] R. Shokri , M. Stronati , C. Song , and V. Shmatikov . Membership inference attacks446 against machine learning models . In 2017 IEEE Symposium on Security and Privacy447 ( SP ) , 2017.448 [ TB21 ] Florian Tramer and Dan Boneh . Differentially private learning needs better features ( or449 much more data ) . In International Conference on Learning Representations , 2021.450 [ WM10 ] Oliver Williams and Frank McSherry . Probabilistic inference and differential privacy.451 In NIPS , pages 2451–2459 , 2010.452 [ YLP+19 ] L. Yu , L. Liu , C. Pu , M. E. Gursoy , and S. Truex . Differentially Private Model Publishing453 for Deep Learning . In 2019 IEEE Symposium on Security and Privacy ( SP ’ 19 ) , 2019.454 [ ZFM+20 ] Pan Zhou , Jiashi Feng , Chao Ma , Caiming Xiong , Steven HOI , et al . Towards theo-455 retically understanding why sgd generalizes better than adam in deep learning . arXiv456 preprint arXiv:2010.05627 , 2020.457 Checklist458 1 . For all authors ... 459 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s460 contributions and scope ? [ Yes ] 461 ( b ) Did you describe the limitations of your work ? [ Yes ] 462 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] 463 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to464 them ? [ Yes ] 465 2 . If you are including theoretical results ... 466 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] 467 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] 468 3 . If you ran experiments ... 469 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-470 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] 471 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they472 were chosen ) ? [ Yes ] 473 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-474 ments multiple times ) ? [ Yes ] 475 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type476 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] 477 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 478 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 479 ( b ) Did you mention the license of the assets ? [ Yes ] 480 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 481 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re482 using/curating ? [ Yes ] 483 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable484 information or offensive content ? [ N/A ] 485 5 . If you used crowdsourcing or conducted research with human subjects ... 486 ( a ) Did you include the full text of instructions given to participants and screenshots , if487 applicable ? [ N/A ] 488 ( b ) Did you describe any potential participant risks , with links to Institutional Review489 Board ( IRB ) approvals , if applicable ? [ N/A ] 490 ( c ) Did you include the estimated hourly wage paid to participants and the total amount491 spent on participant compensation ? [ N/A ] 492

Summary: This paper looks to take a deeper dive into DP hyperparameter tuning, an often overlooked aspect of DP learning. It makes several practical claims about this problem, including comparisons of two well-known DP selection strategies and arguing for the promise of adaptive optimizers. For the latter, they provide evidence for why learning rate and clipping are difficult to tune for DP-SGD and how DPAdam can help mitigate this issue. Finally they also propose a new optimizer which matches DPAdam on performance but reduces the amount of necessary computations.