Title: Private Hyperparameter Tuning with Ex-Post Guarantee
Venue: NeurIPS 2025
Accepted: yes
Keywords: differential privacy

hyperparameter tuning

ex-post differential privacy

Renyi differential privacy
Abstract: The conventional approach in differential privacy (DP) literature formulates the privacy-utility tradeoff with a "privacy-first" perspective: 
for a predetermined level of privacy, a certain utility is achievable. 
However, practitioners often operate under a "utility-first" paradigm, prioritizing a desired level of utility and then determining 
the corresponding privacy cost.
        
Wu et al. [2019] initiated a formal study of this ``utility-first'' perspective by introducing ex-post DP. 
They demonstrated that by adding correlated Laplace noise and progressively reducing it on demand, a sequence of increasingly
accurate estimates of a private parameter can be generated, with the privacy cost attributed only to the least noisy iterate released. 
This led to a Laplace mechanism variant that achieves a specified utility with minimal privacy loss.
However, their work, and similar findings by Whitehouse et al. [2023], are primarily limited to simple mechanisms based
on Laplace or Gaussian noise. 
        
In this paper, we significantly generalize these results.  In particular, we extend the findings of Wu et al. [2019] and Liu and Talwar [2019] 
to support any sequence of private estimators, incurring at most a doubling of the original privacy budget. 
Furthermore, we demonstrate that hyperparameter tuning for these estimators, including the selection of an optimal privacy budget, 
can be performed without additional privacy cost. 
Finally, we extend our results to  ex-post R\'{e}nyi DP, further broadening the applicability of utility-first privacy mechanisms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel framework for hyperparameter tuning in differential privacy (DP) with a utility-first perspective, building on the ex-post DP concept introduced by Wu et al. (2019). It proposes a general algorithm that supports any sequence of private estimators, achieving ex-post DP and Rényi DP guarantees with at most a doubling of the privacy budget. The algorithm extends to complex mechanisms like DP-SGD, unlike prior work limited to Laplace or Gaussian noise. Additionally, the paper introduces a privacy filter for fully adaptive composition in ex-post Rényi DP and provides empirical evidence of superior performance in machine learning tasks, such as linear regression on Twitter data and classification on MNIST.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The algorithm significantly generalizes prior work by Wu et al. (2019) and Liu and Talwar (2019), extending ex-post DP to arbitrary private estimators, including DP-SGD. \n- The proof of Theorem 8, which establishes ex-post pure-DP guarantees, is notably simple and leverages the Sparse Vector Technique. Its elementary nature enhances its accessibility and strengthens the theoretical contribution.\n- The experimental results in Section 5.2 demonstrate substantial improvements in machine learning tasks. \n\nWeaknesses\n- There is an inconsistency between Definition 2 (ex-post pure-DP) and Definition 5 (ex-post approximate-DP), as Definition 5 includes a summation over the output set, absent in Definition 2. \n- The parametrization of the geometric distribution in Algorithm 1, using the probability of failure instead of the conventional probability of success, is unconventional and not highlighted. Without reading the “Probability Notation” section, readers might implement the algorithm incorrectly.\n- Theorem 8 introduces a uniform additive term across all indices, which partially undermines the output-specific flexibility of ex-post DP, aligning it closer to ex-ante DP. This trade-off is not sufficiently discussed.\n- A typo in the proof of Theorem 8 (final line) omits the $\\varepsilon$-term and incorrectly uses $\\mathcal{A} $ instead of $\\mathcal{A}' $, which could compromise the proof’s integrity.\n- Section 4 is poorly introduced and feels disconnected from the rest of the paper, lacking a clear transition or motivation.\n- The general writing could be improved, with occasional unclear phrasing and lack of coherence in some sections, particularly Section 4.\n\nQUESTIONS:\n1. Can the authors clarify or correct the inconsistency between Definition 2 and Definition 5 regarding the summation over the output set? \n2. Could the authors add a note near Algorithm 1 or in its description to highlight the unconventional use of the probability of failurefor the geometric distribution? This would prevent implementation errors and could improve the Clarity score to 3.\n3. Can the authors discuss the implications of the uniform $\\varepsilon' $ term in Theorem 8, which reduces the output-specific flexibility of ex-post DP? \n4. In Theorem 10, are the parameters $l_1, \\dots, l_d $ arbitrary, or do they require optimization? If optimization is not needed, would swapping the quantifier order improve clarity? \n5. Can the authors provide a better introduction or motivation for Section 4 to connect it to the paper’s main narrative?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWu et al. 2019 proposed the framework of Ex-post DP, which aims to search for the smallest privacy budget that passes  a specified  utility.  Despite practical relevance, literature on Ex-post is limited and existing methods do not apply to DP-SGD.\n \nAuthor’s contribution starts by extending ex-post DP to ex-post RDP. Their main contribution is a new algorithm and a bound (inspired from the proof of SVT) for DP selection problem under ex-post DP/RDP.  The DP selection problem has applications in hyperparameter tuning/model selection process.\n \nA major limitation of seminal results from Liu and Talwar 2019 and Papernot and Steinke 2022 for (ex-ante) DP-aware hyperparameter tuning (model selection) is that they assume the same privacy cost for all base mechanisms. This means one can tune the batch size, DP noise scales or number of epochs for a fixed epsilon, but not the base epsilon itself.\n \nTake-away from their bound: Among several candidate ex-ante DP mechanisms with (possibly) heterogeneous ex-ante $\\epsilon$’s, if the ith mechanism produces the best result, then the total ex-post budget of the entire exploration is only $2\\epsilon_i$. The factor of 2 seems unavoidable.\n\nThey also propose a privacy filter to adaptively compose several ex-post RDP mechanisms to obtain the final ex-ante guarantee.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1)\tYou only incur the privacy cost of  the best mechanism which may have much smaller budget than the suboptimal mechanisms with higher budgets. This effect is quite surprising.\n2)\tThe bound places no restrictions on the mechanisms used, and also applies to DP-SGD which is the main the main beneficiary of hyperparameter tuning methods.  \n\nWeakness:\n1)\tThe runtime of this method has higher variance than Papernot & Steinke for the same expectation. But authors already mention this. \n2)\tThe final ex-post epsilon can vary a lot, e.g., when the accuracy gap between the best and second-best mechanism is tiny but their base epsilons differ a lot.    \n3)\tThe bound introduces several new hyperparameters ($\\ell_i’s $). These can be set equal, and upper bounded, but it’s not clear how to choose these in general.\n\nQUESTIONS:\n1)\tAlgorithm 1 does not include the step for checking whether output passes the utility threshold for each $M_i$. Can you please clarify? \n\n2)\tThe line 138 seems to suggest that 'M_1,...,M_d' can be executed multiple times. However, there is no outer loop in Algorithm 1. So, it seems the number of distinct mechanisms need to be much lower than d to have each mechanism run multiple times. If my understanding is correct, please mention this explicitly.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

comments: SUMMARY:
This work introduces a hyperparameter tuning algorithm that provides ex-post DP guarantees, allowing a utility-first approach to DP selection. They propose a privacy filter that allows using their ex-post Rényi DP algorithm interactively ensuring a ex-ante guarantee. The authors demonstrate that they can achieve stricter privacy guarantees than prior work for ML applications.

STRENGTHS AND WEAKNESSES:
This paper is well-motivated and addresses a relevant problem within the field. The authors provide a thorough theoretical treatment of their novel algorithm (hyperparameter tuning via random dropping) proving that it guarantees pure DP and Rényi DP respectively. They also evaluate their algorithm empirically, confirming its practical benefits (and limitations) on two types of problems (analytical and ML).

QUESTIONS:
Some clarification is needed for Theorem 11—for example, the notation $IT^0$ and $IT^1$ is used without definition.

Minor suggestions and comments:
- shortly mention the definition of privacy filter 
- briefly discuss what having a larger variance implies (Figure 1)
- Definition 6: is there a reason why you use $\epsilon$ instead of $\tilde{\epsilon}$ here?
- Line 93: missing brackets (Theorem 10) 
- for better reproducibility, mention the CNN model architecture (in 5.2 second experiment)

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
The paper presents a general algorithm for hyperparameter tuning under ex-post differential privacy, compatible with both pure DP and Rényi DP. It privately selects among multiple candidate mechanisms, incurring privacy cost only for the selected one. The method supports complex workflows like DP-SGD and introduces an ex-post RDP filter to enable interactive composition. Empirical results demonstrate improved privacy-utility tradeoffs compared to prior approaches.

STRENGTHS AND WEAKNESSES:
**Strength**:

This work presents a novel result in private hyperparameter tuning, with potential for practical applications. The algorithm’s design is elegant, and the accompanying proof is communicated with clarity. The author also derives a corresponding privacy filter, which appears to be of additional interest.


**Weakness**:

Please refer to the question section

QUESTIONS:
1. Would the expression above line 178 be more accurately written as $\tilde{\varepsilon}(o, i)$?

2. Can the authors provide more intuition on why the final ex-post DP guarantee depends only on the selected mechanism?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03