The Role of Adaptive Optimizers for Honest Private Hyperparameter Selection

Hyperparameter optimization is a ubiquitous challenge in machine learning , and1 the performance of a trained model depends crucially upon their effective se-2 lection . While a rich set of tools exist for this purpose , there are currently no3 practical hyperparameter selection methods under the constraint of differential4 privacy ( DP ) . We study honest hyperparameter selection for differentially private5 machine learning , in which the process of hyperparameter tuning is accounted for6 in the overall privacy budget . To this end , we i ) show that standard composition7 tools outperform more advanced techniques in many settings , ii ) empirically and8 theoretically demonstrate an intrinsic connection between the learning rate and9 clipping norm hyperparameters , iii ) show that adaptive optimizers like DPAdam10 enjoy a significant advantage in the process of honest hyperparameter tuning , and11 iv ) draw upon novel limiting behaviour of Adam in the DP setting to design a new12 and more efficient optimizer.13 1 Introduction14 Over the last several decades , the field of machine learning has flourished . However , training machine15 learning models frequently involves personal data , which leaves data contributors susceptible to16 privacy attacks . This isn ’ t purely hypothetical : recent results have shown that models are vulnerable17 to membership inference [ SSSS17 , CLE+19 , NSH19 ] and model inversion attacks [ FJR15 , SRS17 ] .18 The leading approaches for privacy-preserving machine learning are based on differential privacy19 ( DP ) [ DMNS06 ] . Informally , DP rigorously limits and masks the contribution that an individual20 datapoint can have on an algorithm ’ s output . To address the aforementioned issues , DP training21 procedures have been developed [ WM10 , BST14 , SCS13 , ACG+16 ] , which generally resemble non-22 private gradient-based methods , but with the incorporation of gradient clipping and noise injection.23 In both the private and non-private settings , hyperparameter selection is instrumental to achieving24 high accuracy . The most common methods are grid search or random search , both of which incur25 a computational overhead scaling with the number of hyperparameters under consideration . In26 the private setting , this issue is often magnified as most private training procedures introduce new27 hyperparameters . Regardless , and more importantly , hyperparameter tuning on a sensitive dataset28 also costs in terms of privacy , naively incurring a multiplicative cost which scales as the square root29 of the number of candidates ( based on composition properties of differential privacy [ KOV15 ] ) .30 Most prior works on private learning choose not to account for this cost [ ACG+16 , YLP+19 , TB21 ] ,31 focusing instead on demonstrating the accuracy achievable by private learning under idealized condi-32 tions , that is , if the best hyperparameters were somehow known ahead of time . Some works assume33 the presence of supplementary public data resembling the sensitive dataset [ AGD+20 , RTM+20 ] ,34 which may be freely used for hyperparameter tuning . Naturally , such public data may be scarce or35 nonexistent in settings where privacy is a concern , leaving practitioners with little guidance on how36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . to choose hyperparameters in practice . As explored in our paper , poor hyperparameter selection with37 standard private optimizers can have catastrophic effects on model accuracy.38 Hope is afforded by the success of adaptive optimizers in the non-private setting . The canonical39 example is Adam [ KB14 ] , which exploits moments of the gradients to adaptively and dynamically40 determine the learning rate . It works out of the box in many cases , providing accuracy comparable41 with tuned SGD . We navigate the different options available to a practitioner to solve the honest42 private hyperparameter tuning problem and ask , are there optimizers which provide strong privacy,43 require minimal hyperparameter tuning , and perform competitively with tuned counterparts ? 44 Our Contributions45 1 . We investigate techniques for private tuning of hyperparameters . We perform the first empirical46 evaluation of the proposed theoretical method of Liu and Talwar [ LT19 ] , and demonstrate that it can47 be relatively expensive . That is , in certain cases , one can tune over sufficiently many hyperparameters48 using standard composition tools such as moments accountant [ ACG+16 ] .49 2 . We empirically and theoretically demonstrate that two hyperparameters , the learning rate and50 clipping threshold , are intrinsically coupled for non-adaptive optimizers . While other hyperparameters51 and the model architecture are restricted by the scope of the task , privacy and utility targets , and52 computational resources , the learning rate and clipping norm have no a priori bounds . Since the53 resulting hyperparameter grid adds up to the privacy cost while tuning to achieve the model with the54 best utility , we explore leveraging adaptive optimizers to reduce the hyperparameter space.55 3 . We empirically demonstrate the DPAdam optimizer ( with default values for most hyperparameters ) ,56 can match the performance of tuned non-adaptive optimizers on a variety of datasets , thus enabling57 private learning with honest hyperparameter selection . This finding complements a prior claim58 of Papernot et al . [ PCS+20 ] , which suggests that a well-tuned DPSGD can outperform DPAdam.59 However , our findings show that this difference in performance is relatively insignificant . Furthermore,60 in the realistic setting where hyperparameter tuning must be accounted for in the privacy loss , we61 show that DPAdam is much more likely to produce non-catastrophic results.62 4 . We show that the adaptive learning rate of DPAdam converges to a static value . To leverage this,63 we introduce a new private optimizer , DPAdamWOSM that matches the performance of DPAdam64 without computing the second moments.65 1.1 Related Work66 Hyperparameter tuning plays a vital role in machine learning practice . In the non-private setting,67 ML practitioners use grid search , random search , Bayesian optimization techniques [ SSA13 ] or68 AutoML [ HZC21 ] techniques to tune their models . However , there hasn ’ t been much research69 on private hyperparameter tuning procedures due to the significant associated privacy costs . Each70 set of hyperparameter configuration results in a privacy-utility tradeoff . This tradeoff for multiple71 configurations can be captured by Pareto frontiers using multivariate Bayesian optimization over72 parameter dimensions [ AGD+20 ] . However , this method asks the model curator to query the73 dataset multiple times which requires non-private access to the dataset . There have been some74 end-to-end private tuning procedures [ CMS11 , CV13 , KGGW15 ] which work for a selected number75 of hyperparameter sets . These results work either in restricted settings for few combinations of76 candidates or under relaxations of differential privacy . The most relevant work to ours is an approach77 for private selection from private candidates [ LT19 ] . Their work provides two methods , one which78 outputs a candidate with accuracy greater than a given input threshold , and another which randomly79 stops and outputs the best candidate seen so far . The first approach is of limited utility in practice as80 it requires a prior accuracy bound for the dataset . The second variant incurs a considerable overhead81 in the privacy cost . We study this second approach and compare it with naive approaches based on82 Moments Accountant [ ACG+16 ] which would scale as the square root of the number of candidates.83 2 Problem Setup and Overview84 Consider a sensitive dataset D which lies beyond a privacy firewall and has n points of the form85 ( x1 , y1 ) , ( x2 , y2 ) , . . . , ( xn , yn ) where xi ∈ X is the feature vector of the ith point and yi ∈ Y is its86 desired output . Though our experiments are carried out in the supervised setting , all results can be87 translated to unsupervised setting as well . The dataset has been divided into two parts , the training set88 and the validation set . A trusted curator wants to train a machine learning model by making queries89 on the dataset with a total end-to-end training privacy budget of ( εf , δf ) such that the model can90 perform with high accuracy on the validation set . The curator wants to try multiple hyperparameter91 candidates for the model to figure out which candidate gives the maximum accuracy . However , as the92 model is private , each candidate requires multiple queries made on the dataset and all of them need to93 be accounted in the total privacy budget of ( εf , δf ) .94 Note that any validation accuracy must also be measured privately . Since this accuracy is a low-95 sensitivity query with a scalar output , and must only be computed once per choice of hyperparameters,96 the cost of this procedure is generally a lower order term versus the main training procedure . Thus for97 simplicity , we do not noise these validation accuracy queries . As we will see later , some optimizers98 require more candidates to tune and hence would also require more privacy budget than others.99 To tackle private hyperparameter selection , we first compare the available private tuning procedures100 in Section 3 . We show that the privacy cost for training a model depends on the hyperparameter101 grid size and standard composition theorems provide the best guarantees when the grid is small . In102 Section 4 , we investigate different optimizers to see how many candidates are required to output103 a good solution . In Section 4.1 we provide theoretical and empirical evidence to demonstrate an104 intrinsic coupling between two hyperparameters – the learning rate and clipping norm in DPSGD.105 We show that this coupling makes DPSGD sensitive to these parameter choices , which can drastically106 affect the validation accuracy . In Section 4.2 we demonstrate that an adaptive optimizer , DPAdam,107 translates well from the non-private setting and obviates tuning of the learning rate . In Section 5 , we108 empirically compare DPAdam with DPSGD and DPMomentum to show that DPAdam performs at par109 with less hyperparameter tuning . Finally , in Section 6 , we establish that DPAdam converges to a static110 learning rate in restricted settings , and unveil a new optimizer DPAdamWOSM which can leverage111 this converged value without computing the second moments . In the interest of space , we defer112 standard preliminaries such as DP definitions , hyperparameters , and optimizers to the appendix.113 3 The Cost of Privately Tuning DP Optimizers114 Effective hyperparameter tuning is crucial in extracting good utility from an optimizer . Unlike115 the non-private setting , DP optimizers typically i ) have more hyperparameters to tune ; ii ) require116 additional privacy budget for tuning . Existing work on DP optimizers acknowledge this problem ( e.g.,117 [ ACG+16 ] ) , but do not address the privacy cost incurred during hyperparameter tuning [ ACG+16,118 YLP+19 , TB21 ] . There are two main prior general-purpose approaches for private hyperparameter119 selection . The first performs composition via Moments Accountant [ ACG+16 ] , and the second is120 the algorithm of Liu and Talwar ( LT ) [ LT19 ] . The latter is a theoretical result , and to the best of our121 knowledge , has not been previously evaluated in practice . We investigate the privacy cost of these122 two techniques in practice and discuss situations in which each method is preferred.123 3.1 Hyperparameter Selection via [ LT19 ] 124 Liu and Talwar [ LT19 ] propose a random stopping algorithm ( LT ) to output a ‘ good ’ hyperparameter125 candidate from a pool ofK candidates , { x1 , . . . , xK } . They assume sampling access to a randomized126 mechanism Q ( D ) which samples i ∼ [ K ] , and returns the i-th candidate xi , and a score qi for this127 candidate . It is a random stopping algorithm , in which at every iteration , a candidate is picked from128 Q i.i.d . with replacement and a γ-biased coin is flipped to randomly stop the algorithm . When the129 algorithm stops , the candidate with the maximum score seen so far is outputted . In the approximate130 DP version of this algorithm , an extra parameter Υ is set to limit the total of number of iterations.131 The pseudocode of this algorithm is deferred to the appendix.132 Theorem 1 ( [ LT19 ] , Theorem 3.4 ) . Fix any γ ∈ [ 0 , 1 ] , δ2 > 0 and let Υ = 1γ log 1 δ2 . If Q is133 ( ε1 , δ1 ) -DP , then the LT algorithm is ( εf , δf ) -DP for εf = 3ε1 + 3 √ 2δ1 and δf = √ 2δ1Υ + δ2.134 Theorem 1 expresses the privacy cost of the algorithm in terms of the privacy cost of individual135 learners , and parameters of the algorithm itself . The δ2 parameter does not significantly affect the136 final epsilon εf of the algorithm and in practice , one can set it to a very small value ( 10−20 ) . Though137 a small value of δ2 has little effect on δf , it increases the hard stopping time of the algorithm , Υ.138 To understand the LT algorithm , we will compare the privacy costs of training a single hyperparameter139 candidate with a final εf , δf budget via LT and compare it with the privacy cost ε1 , δ1 of the underlying140 individual learner . This setting might seem unnatural for LT as it was designed to select from a pool141 of candidates but we choose this setting to show the minimum privacy cost overhead associated with142 LT and later show how the privacy cost changes for multiple candidates ( varying γ ) . To use LT , one143 needs to figure out the ε1 , δ1 via Theorem 1 using the final εf , δf values and in this case , γ = 1 ( as144 we have just one candidate ) . The individual learner is then trained using ε1 , δ1 budget.145 Due to the delicate balance of δf in Theorem 1 , one can see the δ1 comes out to be much smaller than146 δf . This change in δ1 results in a blowup of ε1 and hence , the final privacy cost of the LT algorithm147 ( 3ε1 + 3 √ 2δ1 ) , is much larger than what it would have been for learning one candidate without LT.148 We call this increase the blowup of privacy . We measure this blowup in Figure 1 ( left ) , for the setting149 of σ = 4 , L = 250 , T = 10 , 000 with varying dataset sizes ( n ) . It can be seen that for n = 5 , 000,150 the blowup is 4.8x whereas for for n = 950 , 000 , the blowup is almost 7.3x ( note the log scale on151 y-axis ) . Qualitatively similar trends persist for other choices of noise multiplier , lot size and iterations.152 We add more experiments to compare LT vs MA with varying candidate sizes in the appendix.153 Furthermore , we show that although LT entails a privacy blowup , decreasing γ , which corresponds154 to training more individual learners with ε1 , δ1 , doesn ’ t result in a significant difference in the final155 epsilon guaranteed by LT . In Figure 1 ( middle ) , we show the final epsilon cost for different dataset156 size and varying values of γ ∈ [ 0.001 , 0.01 , 0.1 , 1 ] . It is interesting to note here that with smaller157 γ values , one can train many candidates ( in expectation , 1γ ) for negligible additional privacy cost.158 The blowup to train 1 candidate ( γ = 1 ) versus 1 , 000 candidates ( γ = 0.001 ) increases from 33159 to 39 for n = 5 , 000 and increases from 0.49 to 0.69 , for n = 950 , 000 . This increase is minimal160 in comparison to advanced composition , which grows proportional to O ( √ k ) . However , another161 resource at play is the total training time , which is proportional to 1/γ ( i.e. , the total number of162 candidates ) . In summary , the LT algorithm is effective if an analyst has the privacy budget to afford163 the initial blowup , as the privacy cost of testing additional hyperparameters is insignificant.164 3.2 Hyperparameter Selection via Moments Accountant165 We learnt from the previous section that , LT permits selection from a large pool of hyperparameters166 ( depending on the γ value ) but incurs a constant privacy blowup . We compare LT with tuning167 using Moments Accountant ( MA ) . We notice using that with the same initial privacy blowup of168 the LT algorithm , MA is able to compose a considerable number of hyperparameter candidates . In169 Figure 1 ( right ) , we show the number of candidates that can be composed using MA for the minimum170 privacy cost for running the LT algorithm ( γ = 1 ) , for the setting of σ = 4 , L = 250 , T = 10 , 000171 and varying dataset size ( n ) on the x-axis . As the T and L is set constant , bigger n values in this172 graph correspond to fewer epochs of training and hence , worse utility . Depending on dataset size,173 MA can compose 14 candidates for n = 5000 and up to 175 candidates when n = 100000 . It is174 perhaps surprising how well a standard composition technique performs versus LT . This information175 can be highly valuable to a practitioner who has limited privacy budget . Qualitatively similar trends176 persist for other choices of batch size , noise multiplier , and iterations.177 From our experiments for both these tuning procedures , we conclude that while tuning with LT entails178 an initial privacy blowup , and the additional privacy cost for trying more candidates ( smaller γ ) is179 minimal . Even though this has an additional computation cost , it can be appealing when an analyst180 wants to try numerous hyperparameters . On the other hand , for the same overall privacy cost , MA181 can be used to compose a significant number of hyperparameter candidates . Additionally , MA allows182 access to all intermediate learners , whereas LT allows access to only the final output parameters . In183 the sequel , this conclusion will be useful in making the naive MA approach a more appealing tool for184 some settings ( e.g. , tighter privacy budgets ) .185 4 Tuning DP Optimizers186 We detail aspects of tuning both non-adaptive and adaptive optimizers . We start with tuning non-187 adaptive optimizers ( Section 4.1 ) . We theoretically and empirically demonstrate a connection between188 the learning rate and clipping threshold . We also establish that non-adaptive optimizers inevitably189 require searching over a large LR-clip grid to extract performant models . Adaptive optimizers forego190 this problem as they do not need to tune the hyperparameter dimension of learning rate . However,191 they introduce other hyperparameters that have known good choices in the non-private setting , and192 we empirically show that they are good candidates in the private setting ( Section 4.2 ) .193 4.1 Tuning DP non-adaptive optimizers194 While many hyperparameters are restricted due to computational and privacy/utility targets , the195 learning rate α and the clipping threshold C have no a priori bounds . In what follows , we show196 an interplay between these parameters by first theoretically analyzing the convergence of DPSGD.197 We then explore an illustrative experiment which demonstrates their entanglement . In the following198 theorem , we derive a bound on the expected excess risk of DPSGD and while doing so , show that the199 optimal learning rate , αopt , is proportional to the inverse of C. The proof appears in Appendix D.200 Theorem 2 . Let f be a convex and β-smooth function , and let x∗ = arg min x∈S f ( x ) . Let x0 be201 an arbitrary point in S , and xt+1 = ΠS ( xt − α ( gt + zt ) ) , where gt = min ( 1 , C‖∇f ( x ) ‖2 ) ∇f ( x ) 202 and zt ∼ N ( 0 , σ2C2 ) is the noise due to privacy . After T iterations , the optimal learning rate is203 αopt = R CT √ 1+σ2 , where E [ f ( 1T ∑T i xt ) − f ( x∗ ) ] ≤ RC √ 1+σ2√ T and R = E [ ‖x0 − x∗‖ ] .204 Though Theorem 2 gives a closed-form expression for the205 optimum learning rate , it is a function of the parameter R,206 which is unknown a priori to the analyst . Given constant207 T and σ , the optimal learning rate αopt is inversely propor-208 tional to the clipping normC . This is crucial information in209 practice because these parameters vary among datasets and210 are unbounded . This unboundedness property thus requires211 us to search over very large ranges of C and α when we212 have no prior knowledge of the dataset . It is natural to ask213 whether one can fix the clipping norm C and search only214 over a wide range for the learning rate α ( or vice versa ) .215 We explore this relationship experimentally , showing that216 fixing one of these two hyperparameters may often but not217 always result in an optimal model.218 In this experiment , we train a linear regression model on a219 10-dimensional synthetic dataset of input-label pairs ( x , y ) 220 sampled from a distribution D as follows : x1 , . . . , xd ∼221 U ( 0 , 1 ) , y = x ·w∗ , w∗ = 10 ·1d . We use the initialization222 w0 = 0 d and train for 100 iterations . In the non-private223 setting , this model converges quickly with any reasonable224 learning rate , but in the private setting , we notice that the225 training loss depends heavily on the choice of α and C. Figure 2 shows a heat map for the log226 training loss when trained on ( α , C ) pairs taken from a large grid consisting of [ 1 , 2 , 4 , 5 , 8 ] at scales227 of [ 10−4 , 10−3 , 10−2 , 10−1 , 100 , 101 ] . The best training is observed when the loss is 0 ( black pixels ) .228 We observe two fundamental phenomena from this figure . First , to achieve the best accuracy , α and229 C need to be tuned on a large grid spanning several orders of magnitude for each of these parameters.230 Second , multiple ( α , C ) pairs achieve the best accuracy and all lie on the same diagonal , validating231 our theory for an inverse relation between learning rate and clipping norm . As mentioned earlier , one232 might hypothesize that by setting the clipping norm C constant and tuning α ( corresponding to a233 vertical line in Figure 2 ) or vice versa , one could eliminate tuning a hyperparameter . However , note234 that not all C and α values correspond to the lowest loss . This phenomenon is evident by noticing235 that not all vertical or horizontal lines on this figure have black pixels . This happens , for example,236 at the extremes ( e.g. , at the top-right corner ) , but also for several intermediate and standard choices237 ( e.g. , C = 0.1 or 0.2 ) . Again , the analyst has no way of knowing this a priori . We conclude that to238 privately tune non-adaptive optimizers , we require a large grid of hyperparameter options.239 4.2 Tuning DP adaptive optimizers240 In the interest of reducing this space of tuning we turn to adaptive optimizers , where we can at least241 reduce one dimension of this search space . These approaches automatically adapt over the learning242 rate α , requiring us to tune only over the clipping norm C. But recall our key question : can we train243 models that perform competitively with the fine-tuned counterparts from DPSGD ? 244 Adam [ KB14 ] , the canonical adaptive optimizer introduces two new hyperparameters , which are the245 first and second moment exponential decay parameters ( β1 and β2 ) . In the non-private setting , these246 parameters are relatively insensitive , and default values of α = 0.001 , β1 = 0.9 , and β2 = 0.999 are247 recommended based on empirical findings , requiring no additional tuning for this hyperparameter248 triple . Hence before we compare DPAdam and DPSGD , we first find and establish such recommended249 values for this hyperparameter triple in the DP setting next , and then show that DPAdam with a small250 hyperparameter space performs competitively with DPSGD in Section 5.251 To establish default choices of α , β1 , and β2 for DPAdam , we evaluate this private optimizer over four252 diverse datasets ( details in Appendix B , Table 2 ) and two learning models including logistic regression253 and a neural network with one 100 neurons hidden layer ( TLNN ) . These selected datasets include254 both low-dimensional data ( where the number of samples greatly outnumbers the dimensionality ) and255 high-dimensional data ( where the number of samples and dimensionality are at same scale ) . Since256 we still have a large hyperparameter space to tune over , for the rest of this work , we fix a constant lot257 size ( L = 250 ) , and consider tuning over three different noise levels , σ ∈ [ 2 , 4 , 8 ] , so that we can258 study the effects of tuning the other hyperparameters more thoroughly . All experiments are repeated259 three times and averaged before reporting . Additionally , in this particular experiment since we focus260 on α , β1 , and β2 , we also fix the clipping threshold C = 0.5 , and T = 2500 iterations of training.261 For each dataset and model , we run DPAdam three times with hyperparameters ( α , β1 , β2 ) from the262 grids , α ∈ [ 0.001 , 0.05 , 0.01 , 0.2 , 0.5 ] , β1 , β2 ∈ [ 0.8 , 0.85 , 0.9 , 0.95 , 0.99.0.999 ] .263 We show that the default hyperparameter choice ( α , β1 , β2 ) of Adam in the non-private setting also264 works well for DPAdam . Figure 3 shows the boxplots of testing accuracies of DPAdam over the265 different hyperparameter choices . When α is 0.001 ( same as in the non-private setting ) , all the266 datasets and models have final testing accuracies ( marked in black ) close to the best possible ( and267 in most cases it is in fact the best ) accuracy . Furthermore , we also highlight the accuracy of the268 suggested default choice ( α = 0.001 , β1 = 0.9 , β2 = 0.999 ) using gold dots . Hence , for the ease of269 using DPAdam , we suggest the non-private default values for these parameters in the private setting270 as well and hence in all our subsequent experiments.271 5 Advantages of tuning using DPAdam272 In the non-private setting , adaptive optimizers like Adam enjoy a smaller hyperparameter tuning273 space than SGD . We ask two questions in this section . First , can DPAdam ( with little tuning ) achieve274 accuracy comparable to a well-tuned DPSGD ? Second , what is the privacy-accuracy tradeoff one275 incurs when using either of the two methods we detail in Section 3 for hyperparameter selection.276 To answer both questions , we compare DPAdam and DPSGD over the same set of datasets and277 models from the previous section . We report the accuracy of DPSGD with a range of learning rates278 and clipping values shown in Table 3 ( Appendix C ) , and the testing accuracy of DPAdam with default279 parameter choice from Section 4.2 ( α = 0.001 , β1 = 0.9 , β2 = 0.999 ) and a range of clipping values280 C in Table 3 . In total , DPSGD has 40 candidates to tune over , and DPAdam has 4 . This is because281 we have shown in Section 4.1 that DPSGD needs a wide grid to obtain the best accuracy when data282 distributions are unknown . Additionally , we also consider the DPMomentum optimizer . Similar to283 how we searched for default tuning choices for DPAdam in Section 4.2 , we investigate if there exists284 a qualitatively good choice for the momentum hyperparameter , and unfortunately our results show285 that there is no such choice . We detail this process in the supplement.286 In order to show the comparison from both sides of the privacy-accuracy tradeoff , we compare the287 three optimizers through i ) the privacy cost when extracting the best accuracy from these optimizers,288 and ii ) the accuracy one would obtain from them under the tight privacy constraints.289 5.1 Prioritizing Accuracy290 For brevity , we show experiments for σ = 4 in Figure 4 , results for other values of σ are displayed in291 the supplement . For each dataset and model , we train three times for each hyperparameter candidate292 and report the max every 100 iterations , corresponding to the dark lines for each optimizer . We note293 that their maxima are extremely similar . However , Table 1 shows the final privacy costs incurred294 by each of these max accuracy lines , and reflects our claims from Section 3.1 that using fewer295 hyperparameter candidates and composing privacy via MA gives a much tighter privacy guarantee.296 5.2 Prioritizing Privacy297 Additionally in Figure 4 , DPSGD and DPMomentum have pastel dotted lines corresponding to their298 mean accuracy attained using the MA composition that provides the tightest privacy guarantees for299 DPAdam . These pastel lines are the mean accuracies ( with 95 % CI ) from 100 repetitions of this300 experiment . Since DPAdam has only 4 hyperparameter candidates , for this experiment , we sample 4301 of the candidates at random for DPSGD and DPMomentum so that they all incur the same privacy302 cost . Since the candidate pool is significantly larger for DPSGD and DPMomentum , we additionally303 scrutinize the parameter grid for them and prune the learning rates that perform poorly . Our pruning304 process ( detailed in the supplement ) is quite generous , and favours minimizing the hyperparameter305 space of DPSGD and DPMomentum as much possible . 1 Despite the pruning advantage we see that306 these optimizers perform subpar than DPAdam when constrained with tight privacy requirements.307 6 DPAdam without second moment : DPAdamWOSM308 As illustrated in the previous section , adaptivity can indeed be a boon , enabling DPAdam to match309 the performance of tuned DPSGD while consuming roughly a third of the privacy budget as seen310 in Table 1 . However , in addition to a decaying average of the past gradient updates , DPAdam311 also requires maintaining a decaying average of their second moments . In this section , we design312 DPAdamWOSM , a new DP optimizer that operates only using a decaying average of past gradients,313 as well as eliminates the need to tune the learning rate parameter . We achieve this by analyzing the314 convergence behavior of the second-moment decaying average in DPAdam in regimes where the scale315 of noise added is much higher than the scale of the clipped gradients . Setting the effective step size316 ( ESS ) of DPAdam to the converged constant , and removing all computations related to the second-317 moment updates , results in DPAdamWOSM . We empirically demonstrate that DPAdamWOSM318 matches the utility of DPAdam , while requiring less computation than DPAdam.319 Observe that removing the second-moment updates from DPAdam reduces it to DPMomentum with320 one additional feature : bias-correction to the first-moment decaying average , which DPAdam does321 to account for its initialization at the origin . While the resultant optimizer still requires tuning the322 learning rate ( in addition to other hyperparameters like the clipping threshold ) , DPAdamWOSM can323 be viewed as self-tuning the learning rate by fixing it to the converged effective step size in DPAdam.324 6.1 Effective step size ( ESS ) in DPAdam325 DPAdam produces results with a smaller variance than DPSGD due to its adaptive learning rate.326 To understand this phenomenon better , we look closely into the update step of DPAdam [ KB14 ] .327 DPAdam being an adaptive optimizer picks per-parameter ESS as α√ v̂t+ξ , which is the base learning328 rate α scaled by the second moment of the individual parameter gradients . We notice that when329 g → 0 , the ESS for DPAdam converges for the first moment gradient , which innately accounts for330 the clip bound one is training with . This may happen at later iterations , when the model is close to its331 minima and the gradients get close to zero.332 Theorem 3 . The effective step size ( ESS ) for DPAdam with g → 0 converges to ESS∗ = α ( σC/L ) +ξ .333 Proof . Recall that the average noisy gradient over a lot is g̃ = g +N ( 0 , σ2C2 ) /L . We now look at334 the effect of this noisy gradient on the effective step size ( ESS ) of DPAdam . As g → 0 , the second335 moment of DPAdam converges to σ 2C2 L2 . This gives us the converging value for ESS:336 ESS∗ = α√ v̂t + ξ = α√ σ2C2 L2 + ξ = α ( σC/L ) + ξ 1Note , pruning itself is of course unfair ; the intent was to design a DP optimizer that can be used on any data distributions that we have no prior knowledge of . To do so with DPSGD one would have to consider a significantly wide range of ( α , C ) pairs to cover ‘ good ’ candidates as we illustrated in Section 4.1 Theorem 3 gives a closed form expression that ESS converges to . We can use this value in place337 of α√ v̂t+ξ in the update step from the inception of the learning process . Since the second-moment338 updates ( e.g. , v̂t ) are not used anymore , removing them results in our new optimizer DPAdamWOSM.339 We provide a pseudo-code for DPAdamWOSM in the appendix.340 6.2 Comparing Adaptive Optimizers341 We evaluate DPAdamWOSM by running it alongside DPAdam and ADADP with the same hyperpa-342 rameter grid in the appendix . For brevity , we show experiments on σ = 4 and others appear in the343 supplement . In Figure 5 , we show the maximum and median accuracy curves for all the optimizers.344 We display the median accuracy curves ( shown in dotted ) , as an indicator of the quality of the entire345 pool of hyperparameter candidates for a given optimizer ; which in this case is strictly over the choices346 of clip . The max lines for ADADP lies beneath DPAdam and DPAdamWOSM for all dataset except347 Adult . Also , the max accuracy line for DPAdamWOSM runs alongside DPAdam which means that348 it can perform as good as DPAdam throughout training . The median line for DPAdamWOSM also349 performs alongside DPAdam and in some cases is able to beat it ( e.g , the median for DPAdamWOSM350 for MNIST-LR and MNIST-TLNN lies above the median line of DPAdam ) . This occurrence is seen351 because DPAdamWOSM uses the converged ESS from the first iteration of training.352 7 Conclusion353 In this paper , we performed a thorough investigation of honest hyperparameter selection for DP354 Optimizers . We compared two existing private methods , LT and MA to search for hyperparameter355 candidates and showed that , the former incurs a significant privacy cost but can compose a great many356 candidates , while the latter is helpful when the number of candidates is small . Next , we explored357 connections between the clipping norm and the step size hyperparameter to show an inverse relation-358 ship between them . Additionally , we compared non-adaptive and adaptive optimizers , demonstrating359 that the latter typically achieves more consistent performance over a variety of hyperparameter360 settings . This can be vital for applications where public data is scarce , resulting in difficulties when361 tuning hyperparameters . Finally , we brought to light that DPAdam converges to a static learning rate362 when the noise starts dominating the gradients . This insight allowed us to derive a novel optimizer363 DPAdamWOSM , a variant of DPAdam which avoids the second-moment computation and enjoys364 better accuracy especially at earlier iterations . Future work remains to investigate further implications365 of these results to provide tuning-free end-to-end private ML optimizers.366 References367 [ ACG+16 ] Martin Abadi , Andy Chu , Ian Goodfellow , H. Brendan McMahan , Ilya Mironov , Kunal368 Talwar , and Li Zhang . Deep learning with differential privacy . In Proceedings of the369 2016 ACM SIGSAC Conference on Computer and Communications Security ( CCS ’ 16 ) ,370 2016.371 [ AGD+20 ] Brendan Avent , Javier González , Tom Diethe , Andrei Paleyes , and Borja Balle . Auto-372 matic discovery of privacy–utility pareto fronts . Proceedings on Privacy Enhancing373 Technologies , 2020 ( 4 ) :5–23 , 2020.374 [ BBG18 ] Borja Balle , Gilles Barthe , and Marco Gaboardi . Privacy amplification by subsampling:375 Tight analyses via couplings and divergences . In Advances in Neural Information376 Processing Systems 31 , NeurIPS ’ 18 , pages 6277–6287 . Curran Associates , Inc. , 2018.377 [ BST14 ] Raef Bassily , Adam D. Smith , and Abhradeep Thakurta . Private empirical risk min-378 imization : Efficient algorithms and tight error bounds . In FOCS , pages 464–473,379 2014.380 [ CLE+19 ] Nicholas Carlini , Chang Liu , Úlfar Erlingsson , Jernej Kos , and Dawn Song . The secret381 sharer : Evaluating and testing unintended memorization in neural networks . In 28th382 USENIX Security Symposium ( USENIX Security 19 ) , pages 267–284 , Santa Clara , CA,383 August 2019 . USENIX Association.384 [ CMS11 ] Kamalika Chaudhuri , Claire Monteleoni , and Anand D Sarwate . Differentially private385 empirical risk minimization . Journal of Machine Learning Research , 12 ( 3 ) , 2011.386 [ CV13 ] Kamalika Chaudhuri and Staal A Vinterbo . A stability-based validation procedure for387 differentially private machine learning . Advances in Neural Information Processing388 Systems , 26:2652–2660 , 2013.389 [ DHS11 ] John Duchi , Elad Hazan , and Yoram Singer . Adaptive subgradient methods for online390 learning and stochastic optimization . Journal of machine learning research , 12 ( 7 ) ,391 2011.392 [ DKM+06 ] Cynthia Dwork , Krishnaram Kenthapadi , Frank McSherry , Ilya Mironov , and Moni393 Naor . Our data , ourselves : Privacy via distributed noise generation . In EUROCRYPT,394 volume 4004 , pages 486–503 . Springer , 2006.395 [ DMNS06 ] Cynthia Dwork , Frank McSherry , Kobbi Nissim , and Adam Smith . Calibrating noise to396 sensitivity in private data analysis . In Proceedings of the 3rd Conference on Theory of397 Cryptography , TCC ’ 06 , pages 265–284 , 2006.398 [ FJR15 ] Matt Fredrikson , Somesh Jha , and Thomas Ristenpart . Model inversion attacks that399 exploit confidence information and basic countermeasures . In Proceedings of the 22nd400 ACM SIGSAC Conference on Computer and Communications Security ( CCS ’ 15 ) , 2015.401 [ GAYB17 ] R. Gylberth , R. Adnan , S. Yazid , and T. Basaruddin . Differentially private optimization402 algorithms for deep neural networks . In 2017 International Conference on Advanced403 Computer Science and Information Systems ( ICACSIS ) , pages 387–394 , 2017.404 [ HSS12 ] Geoffrey Hinton , Nitish Srivastava , and Kevin Swersky . Neural networks for machine405 learning lecture 6a overview of mini-batch gradient descent . Cited on , 14 ( 8 ) , 2012.406 [ HZC21 ] Xin He , Kaiyong Zhao , and Xiaowen Chu . Automl : A survey of the state-of-the-art.407 Knowledge-Based Systems , 212:106622 , 2021.408 [ KB14 ] Diederik P Kingma and Jimmy Ba . Adam : A method for stochastic optimization . arXiv409 preprint arXiv:1412.6980 , 2014.410 [ KGGW15 ] Matt Kusner , Jacob Gardner , Roman Garnett , and Kilian Weinberger . Differentially411 private bayesian optimization . In International conference on machine learning , pages412 918–927 . PMLR , 2015.413 [ KH20 ] Antti Koskela and Antti Honkela . Learning rate adaptation for differentially private414 learning . In International Conference on Artificial Intelligence and Statistics , pages415 2465–2475 . PMLR , 2020.416 [ KOV15 ] Peter Kairouz , Sewoong Oh , and Pramod Viswanath . The composition theorem for417 differential privacy . In International conference on machine learning , pages 1376–1385.418 PMLR , 2015.419 [ LT19 ] Jingcheng Liu and Kunal Talwar . Private selection from private candidates . In Pro-420 ceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , pages421 298–309 , 2019.422 [ MAE+18 ] H Brendan McMahan , Galen Andrew , Ulfar Erlingsson , Steve Chien , Ilya Mironov,423 Nicolas Papernot , and Peter Kairouz . A general approach to adding differential privacy424 to iterative training procedures . arXiv preprint arXiv:1812.06210 , 2018.425 [ NSH19 ] M. Nasr , R. Shokri , and A. Houmansadr . Comprehensive privacy analysis of deep learn-426 ing : Passive and active white-box inference attacks against centralized and federated427 learning . In 2019 IEEE Symposium on Security and Privacy ( SP ’ 19 ) , 2019.428 [ PCS+20 ] Nicolas Papernot , Steve Chien , Shuang Song , Abhradeep Thakurta , and Ulfar Erlingsson.429 Making the shoe fit : Architectures , initializations , and tuning for learning with privacy,430 2020.431 [ Qia99 ] Ning Qian . On the momentum term in gradient descent learning algorithms . Neural432 Networks , 1999.433 [ RHW86 ] David E Rumelhart , Geoffrey E Hinton , and Ronald J Williams . Learning representations434 by back-propagating errors . nature , 323 ( 6088 ) :533–536 , 1986.435 [ RTM+20 ] Swaroop Ramaswamy , Om Thakkar , Rajiv Mathews , Galen Andrew , H Brendan McMa-436 han , and Françoise Beaufays . Training production language models without memorizing437 user data . arXiv preprint arXiv:2009.10031 , 2020.438 [ SCS13 ] Shuang Song , Kamalika Chaudhuri , and Anand D. Sarwate . Stochastic gradient descent439 with differentially private updates . In GlobalSIP , pages 245–248 , 2013.440 [ SRS17 ] Congzheng Song , Thomas Ristenpart , and Vitaly Shmatikov . Machine learning models441 that remember too much . In Proceedings of the 2017 ACM SIGSAC Conference on442 Computer and Communications Security ( CCS ’ 17 ) , 2017.443 [ SSA13 ] Kevin Swersky , Jasper Snoek , and Ryan P Adams . Multi-task bayesian optimization.444 Advances in Neural Information Processing Systems , 26:2004–2012 , 2013.445 [ SSSS17 ] R. Shokri , M. Stronati , C. Song , and V. Shmatikov . Membership inference attacks446 against machine learning models . In 2017 IEEE Symposium on Security and Privacy447 ( SP ) , 2017.448 [ TB21 ] Florian Tramer and Dan Boneh . Differentially private learning needs better features ( or449 much more data ) . In International Conference on Learning Representations , 2021.450 [ WM10 ] Oliver Williams and Frank McSherry . Probabilistic inference and differential privacy.451 In NIPS , pages 2451–2459 , 2010.452 [ YLP+19 ] L. Yu , L. Liu , C. Pu , M. E. Gursoy , and S. Truex . Differentially Private Model Publishing453 for Deep Learning . In 2019 IEEE Symposium on Security and Privacy ( SP ’ 19 ) , 2019.454 [ ZFM+20 ] Pan Zhou , Jiashi Feng , Chao Ma , Caiming Xiong , Steven HOI , et al . Towards theo-455 retically understanding why sgd generalizes better than adam in deep learning . arXiv456 preprint arXiv:2010.05627 , 2020.457 Checklist458 1 . For all authors ... 459 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s460 contributions and scope ? [ Yes ] 461 ( b ) Did you describe the limitations of your work ? [ Yes ] 462 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] 463 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to464 them ? [ Yes ] 465 2 . If you are including theoretical results ... 466 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] 467 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] 468 3 . If you ran experiments ... 469 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-470 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] 471 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they472 were chosen ) ? [ Yes ] 473 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-474 ments multiple times ) ? [ Yes ] 475 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type476 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] 477 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 478 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 479 ( b ) Did you mention the license of the assets ? [ Yes ] 480 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 481 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re482 using/curating ? [ Yes ] 483 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable484 information or offensive content ? [ N/A ] 485 5 . If you used crowdsourcing or conducted research with human subjects ... 486 ( a ) Did you include the full text of instructions given to participants and screenshots , if487 applicable ? [ N/A ] 488 ( b ) Did you describe any potential participant risks , with links to Institutional Review489 Board ( IRB ) approvals , if applicable ? [ N/A ] 490 ( c ) Did you include the estimated hourly wage paid to participants and the total amount491 spent on participant compensation ? [ N/A ] 492

Summary: This paper presents a combination of analytical and (mostly) empirical results evaluating the task of "honest hyperparameter selection" under differential privacy; that is, accounting for the full set of tuning experiments conducted when computing the resulting privacy guarantee. The authors provide some empirical comparison between two existing composition methods for hyperparameter selections -- Moments Accountant (MA) and Liu & Talwar (LT) and give a relation between privacy of individual learners and the final learner in Theorem 1; however, their focus is mostly on showing the competitiveness of DPAdam relative to DPSGD. They conclude with a proposed adaptation to DPAdam which discards the second moments term since they observe this to converge to a constant value in practice.