Title: Purifying Approximate Differential Privacy with Randomized Post-processing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Differential Privacy

approximate DP to Pure DP conversion

Empirical Risk Minimization
Abstract: We propose a framework to convert $(\varepsilon, \delta)$-approximate Differential Privacy (DP) mechanisms into $(\varepsilon', 0)$-pure DP mechanisms under certain conditions, a process we call ``purification.'' This algorithmic technique leverages randomized post-processing with calibrated noise to eliminate the $\delta$ parameter while achieving near-optimal privacy-utility tradeoff for pure DP. It enables a new design strategy for pure DP algorithms: first run an approximate DP algorithm with certain conditions, and then purify. This approach allows one to leverage techniques such as strong composition and propose-test-release that require $\delta>0$ in designing pure-DP methods with $\delta=0$. We apply this framework in various settings, including Differentially Private Empirical Risk Minimization (DP-ERM), stability-based release, and query release tasks. To the best of our knowledge, this is the first work with a statistically and computationally efficient reduction from approximate DP to pure DP. Finally, we illustrate the use of this reduction for proving lower bounds under approximate DP constraints with explicit dependence in $\delta$, avoiding the sophisticated fingerprinting code construction.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDifferential privacy (DP) can be viewed as a bound on the difference between mechanism output distributions induced by neighboring datasets. This difference is measured in terms of the log probability ratio (privacy loss) between the two distributions. The two main variants of DP are pure-DP (the privacy loss is bounded by $\\epsilon$ a.s.), and approximate-DP (roughly speaking, the privacy loss exceeds $\\epsilon$ w.p. $\\approx\\le\\delta$).\n\nIt is considered folklore that approx-DP mechanisms $\\mathcal{M}$ with finite range can be turned into pure-DP by a simple random post-process, which w.p. $\\omega$ samples uniformly at random some output and w.p. $1-\\omega$ returns the output of $\\mathcal{M}$. Scaling $\\omega$ one can balance the degradation in privacy ($\\epsilon$) and utility. Intuitively speaking, this process trades the low probability event of a large privacy loss for the low probability event of large utility loss.\n\nThis works moves beyond finite range, and prove that if in the $1-\\omega$ case the post processing adds a small amount of Laplace noise, this result extends to any range contained in a $q$-norm ball in $\\mathbb{R}^{d}$. This is achieved by converting TV distance to a variant of Wasserstein distance, thus using the known result which converts $(\\epsilon, \\delta)$-approx-DP to \"$\\delta$-TV distance away from $\\epsilon$-pure-DP\".\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. While approx-DP is typically treated as sufficiently private for most use cases, pure-DP is a much cleaner and natural definition, so the ability to convert nearly any approx-DP mechanism to pure-DP is an important contribution.\n2. From the theoretical perspective, this work is even more valuable, since lower bounds for pure-DP are generally tighter than those known for approx-DP. Using the proposed conversion, these lower bounds can be extended in some cases.\n3. The authors survey a large range of classical approx-DP mechanisms, and analyze how purifying them affects their utility.\n\n**Weaknesses**\n\nI have two concerns regarding this work, which - if satisfactory addressed can affect my score.\n\n*Main concern*\n\nThe authors do not consider a natural baseline, which - at first glance - seems to provide similar utility guarantees in slightly more general setting, without the need for complex proofs. It might very well be the case that this baseline has some limitations or is inferior in some other way, but without a direct comparison it is hard for me to assess the contribution of this work.\n\nThe proposed alternative is simple. Cover the range by a $\\Delta$-net, round the output of the approx-DP mechanism to the nearest grid point, then purify using the finite range method. The utility error will be the sum of the low probability event of random output and the rounding error. Using this technique, one can purify a $(\\epsilon, \\delta)$-DP mechanism into a $2 \\epsilon$-DP mechanism by bounding the number of grid points by $e^{\\epsilon} (e^{\\epsilon}-1)\\frac{\\omega}{\\delta}$. The number of $\\Delta$-net points required to cover a $q$-norm $d$ ball is $\\approx \\left(\\frac{R}{\\Delta}\\right)^{d}\\cdot\\frac{1}{\\Gamma(1+d/q)}$ (its volume divided by $\\Delta^{d}$). Combining the two we get $\\Delta \\approx R \\left(\\Gamma(1+d/q) e^{\\epsilon} (e^{\\epsilon}-1)\\frac{\\omega}{\\delta} \\right)^{1/d}$ which can be further simplified using Stirling's approximation.\n\nThis expression shares many similarities with the one proposed in this work (Algorithm 1), in particular the relation of $\\delta$ and $d$, while differing in others (which might be lower order effects). Besides it simplicity, this method does not depend on the inner radius $r$, which makes this result more general. In fact, it does not depend on the diameter either, only on the range's volume. \n\n*Minor gaps*\n\nThe list of utility comparisons for the various purifying techniques mostly lacks the results for the SOTA pure-DP mechanisms. The method proposed in this work is of significant importance, even if the approx-DP -> purify route is sub-optimal relative to the directly pure algorithm in some cases, but this comparison is crucial for a honest analysis of this method.\n\nQUESTIONS:\nPlease address my two concerns.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel \"purification\" framework built on randomized post-processing techniques. This framework transforms mechanisms with $(\\epsilon, \\delta)$-approximate differential privacy into $(\\epsilon + \\epsilon', 0)$-pure differential privacy mechanisms, albeit with additional utility costs. The authors demonstrate the framework's versatility by applying it to differentially private empirical risk minimization (DP-ERM), stability-based data release mechanisms, and query processing tasks. \n\nAnother theoretical contribution of this framework is its use of purification to derive lower bounds for approximate differential privacy that explicitly depend on $\\delta$. This provides a novel proof methodology that serves as an alternative to traditional fingerprinting approaches.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written and presents an intellectually compelling approach to bridging approximate and pure differential privacy paradigms. The quest for a unified theoretical framework that can transform approximate DP solutions into pure DP guarantees represents an ambitious and potentially valuable research direction.\n\nHowever, my main concern is about the practical relevance of this work. General theoretical frameworks of this nature often suffer from being too abstract to yield meaningful practical benefits. To better evaluate the contribution, I would appreciate a comparative analysis showing how existing pure DP solutions from the literature (as referenced in Table 1) perform relative to their purified approximate DP counterparts. A clearer articulation of when purification offers advantages versus when direct pure DP methods are preferable would substantially strengthen the paper.\n\nQUESTIONS:\n1. **Mechanism Design**: Doesn't designing approximate DP mechanisms typically present greater challenges than pure DP mechanisms? Given that smaller $\\delta$ values bring us closer to pure DP guarantees, does the purification algorithm consistently select $\\log(1/\\delta) \\approx d$ as its operating parameter?\n\n2. **Rate Discrepancies**: In Table 1, why do Theorems 4 and 7 yield different convergence rates with respect to $n\\epsilon$ ? What accounts for this theoretical difference?\n\n3. **Norms**: The domain assumptions appear to be tailored for $\\ell_p$ norms. What represents the primary obstacle in extending this framework to more general norm spaces?\n\n4. **Notation**: In the introduction, you use the notation $\\mathcal{O}_p$. Should we interpret this as $p = q$ throughout the analysis?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers an efficient generic method to take an arbitrary approximate DP algorithm $M \\colon X \\rightarrow Y$ (i.e., $(\\varepsilon,\\delta)$-DP with $\\delta>0$) with bounded $Y \\subseteq R^d$ (Namely, that $Y$ is contained by an $\\ell_q$ ball of radius $R$ and contains an $\\ell_q$ ball of radius $r$), and transform it into a pure $\\varepsilon’$-DP algorithm, where the resulting $\\ell_q$ error is a function of all parameters (and more additional parameters like the internal mixture parameter $\\omega \\in [0,1]$). For finite output spaces, this was already known (mentioned in Appendix C): Given a dataset $D$ and an approximate DP algorithm $M \\colon X \\rightarrow Y$, output $M(D)$ with probability $1-\\omega$ (for some parameter $\\omega$), and otherwise output a uniform element in the range $Y$. But when $Y$ is infinite, the authors show that this method simply does not work. The first step of their solution is essentially the same, but this alone is not enough in the infinite $Y$ case and requires an additional step: They show that under some conditions on the parameters, for any neighboring datasets $D,D’$, an $\\omega$-mixure of $M(D)$ with $Unif(Y)$ (denote by $\\mu$) and an $\\omega$-mixure of $M(D’)$ with $Unif(Y)$ (denote by $\\mu’$) has a small enough TV distance that enables coupling in a way the ensures that if we draw $(x,x’) \\sim (\\mu,\\mu’)$ then almost surely the $\\ell_q$ distance between $x$ and $x’$ is bounded (this is called the $\\infty$-Wasserstain distance). Given that, they simply add an additional step of Laplace noise per coordinate to guarantee that the final outputs are purely indistinguishable. \n\nThey illustrate the applicability of their method by presenting new pure-DP algorithms for Stochastic Gradient Descent, Frank-Wolfe Algorithm, Propose-Test-Release, Linear Regression, and Query Release. In all (or most of) these examples, they match state-of-the-art accuracy results, but sometimes with improved running time, like in Stochastic Gradient Descent and Query Release. \n\nIn addition to all of that, they show that their method is even useful for proving lower bounds for approximate DP algorithms, because for some problems, it is easier to provide pure DP lower bound, so if an accuracy guarantee for approximate DP is translated (using their method) to an pure DP accuracy that violates the lower bound, then we conclude that the approximate DP accuracy guanatee is impossible. They apply this method to the problem of estimating the average of points in the $d$-dimensional cube, and prove, using their method, a lower bound that is almost as the fingerprinting lower bound (but without the need to use the fingerprint machinery).\n\nSTRENGTHS AND WEAKNESSES:\nThe statements and proofs are not easy to read, and I believe it is possible to simplify them and make them more accessible. But besides that, this paper is one of the most interesting and significant papers that I have reviewed for NeurIPS, and therefore, I support acceptance. \n\nIn general, I’m not a fan of pure-DP. I think it is too stringent, and I don’t think that allowing a small $\\delta$ loss should be considered as a real privacy concern. But I do appreciate the improved pure DP upper bounds, especially since they have been achieved using a simple and generic method. Furthermore, even for someone like me, this paper offers a new method to prove approximate DP lower bounds, and I really like that.\n\nQUESTIONS:\nI’d appreciate it if you could provide a high-level intuition for the proof of Lemma 6 (converting TV distance to $W_{\\infty}$) under simplified assumptions like d=1 or 2 (and maybe using more relaxations). I just want to get the feeling of how the coupling between $\\mu$ and $\\nu$ looks.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper studies a question that sounds, when you hear it the first time, shocking. Is it possible to convert an algorithm with weak privacy guarantees to an algorithm with stronger privacy guarantees using some black-box approach. 
The answer is surprisingly yes when the weak privacy guarantee means the mechanism is approx DP and the stronger prviacy guarantee means pure DP (with comparable epsilon).

The paper describes the transformation and uses it to give the best known mechanism for ERM and some other problems.

STRENGTHS AND WEAKNESSES:
Approx DP, while loved by specialists, has multiple issues in practice: e.g., it is not clear whether any delta is acceptable for real-life solutions operating with sensitive data. 

Hence, the ability to construct pure DP mechanisms with performance similar to approx-DP counterparts is opening a lot of new directions. In addition, this method could simplify proofs of lower bounds for approx-DP algorithms since proving pure-DP bounds is usually not too hard.

QUESTIONS:
1. I am not sure I am able to understand Figure 1: what is the dimension used? 
2. I think there is a confusion and sometimes the term support is used instead of range. 
3. In Remark 3, the transformation is applied to DP-SGD, but typically DP-SGD has unbounded range...
4. line 190: I don't think all of these papers need to be cited.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-01