Title: Private Statistical Estimation via Truncation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Theory

Differential Privacy

Truncated Statistics

Computationally Efficient Estimation
Abstract: We introduce a novel framework for differentially private (DP) statistical estimation via data truncation, addressing a key challenge in DP estimation when the data support is unbounded. Traditional approaches rely on problem-specific sensitivity analysis, limiting their applicability. By leveraging techniques from truncated statistics, we develop computationally efficient DP estimators for exponential family distributions, including Gaussian mean and covariance estimation, achieving near-optimal sample complexity. Previous works on exponential families only consider bounded or one-dimensional families. Our approach mitigates sensitivity through truncation while carefully correcting for the introduced bias using maximum likelihood estimation and DP stochastic gradient descent. Along the way, we establish improved uniform convergence guarantees for the log-likelihood function of exponential families, which may be of independent interest. Our results provide a general blueprint for DP algorithm design via truncated statistics.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a differentially private statistical estimation framework by using techniques from truncated statistics. They then apply their framework to exponential family distributions. A distribution $q_\\theta$ (on $R^d$) from the exponential families takes the form $q_\\theta(x) \\propto h(x) \\exp(\\theta^\\top T(x))$ where $\\theta \\in R^m$ is the unknown parameter, $h : R^d \\to R_+$ is the base measure, $T : R^d \\to R^m$ is a sufficient statistic. \nThis paper obtains an algorithm with sample complexity that depends linearly on $m$ (dimension of sufficient statistics), polynomially on $1/\\alpha\\varepsilon$, $1/\\varepsilon$, $1/\\alpha^2$, exponentially on the condition number. Their algorithm runs in time poly(n, m, d) (n is the number of samples and d is the dimension of each sample). They instantiate their algorithms for Gaussian mean estimation (with known covariance matrices) and Gaussian covariance estimation. For both problems, their algorithm gets the right sample complexity dependence on the dimension.\nTheir algorithmic framework first chooses a \"survival set\" and truncates data to that set, and then minimizes the empirical truncated NLL function using DP-SGD.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** \n- This paper made a nice observation that results/techniques from truncated statistics could be made use of for DP estimation. I think this connection is very natural and might have great potential. \n- This paper is generally well written.\n\n**Weakness:**\n- The results for Gaussian estimation are actually suboptimal and the claim of (near-)optimality for Gaussian estimation in the paper is overstated. For Gaussian mean estimation (Theorem 4.1), e.g., let's consider the regime when beta is a sufficiently small constant. Then the optimal sample complexity is $d/\\alpha^2 + d/\\alpha\\varepsilon + \\log(1/\\delta)/\\varepsilon$, while the sample complexity in Theorem 4.1 is $d/\\alpha^2 + d\\log(1/\\delta)/\\alpha\\varepsilon$. For Gaussian covariance estimation (Theorem 4.2), their sample complexity depends on the condition number. Although the dependence is only logarithmic, the optimal sample complexity should not depend on the condition number at all.\n- It's not very clear how good the sample complexity stated in Theorem 3.1 is, without enough context about what can be achieved non-privately. I think there should be more detailed discussion. E.g. where does the exponential dependence on the condition number arise?\n\nQUESTIONS:\n- How does the sample complexity in Theorem 3.1 compare with the non-private one (i.e. what can be achieved without privacy)?\n- The framework shares some similarity with the clip-and-noise method. Can you elaborate on their differences?\n- Can the 1-d algorithm from [MSU20] be extended to high-dimensional settings?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers DP statistical estimation, i.e. estimating the parameter of a distribution given samples from that distribution with differential privacy. Normally, differential privacy requires a bound on the sensitivity of some function (e.g. the sample mean) of the samples, which is at odds with sampling from unbounded distributions. The authors consider methods from the non-private statistics literature for working with samples restricted to a \"truncation set\". They give an algorithm which obtains the optimal sample complexity dependence on the dimension $m$, privacy parameter $\\epsilon$, and error $\\alpha$ of $O(m(\\frac{1}{\\epsilon} + \\frac{1}{\\alpha^2} + \\frac{1}{\\alpha \\epsilon})$ in polynomial time.\n\nTheir algorithm has a number of components and associated proofs. First, they give a procedure for padding/truncating a dataset so all samples appear in the truncation set and the dataset has fixed size, and with high probability the final dataset matches n samples from the distribution restricted to the truncation set, which allows them to work with the replace adjacency. Next, they assume a warm-start which they show can be obtained by a recursive mean estimation algorithm. Given the warm-start they run DP-SGD using the negative log likelihood ratio of the samples as a loss. They show this is strongly convex under an assumption about the condition number of the covariance of the sufficient statistics, hence DP-SGD has a known bounded error for this loss. They also show how to obtain an unbiased estimator of gradients of this loss with bounded gradient norms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* First paper to get the right dependence on the privacy parameter $\\epsilon$ and error parameter $\\alpha$ for general exponential families without requiring boundedness or dimension 1. \n* The paper retrieves the best rates for Gaussian mean and covariance estimation using a more generally applicable algorithm.\n* The technical aspects of the paper are reasonably novel; e.g. optimization of the negative LLR is pretty different from the private selection-based approaches used for the optimal results on Gaussians.\n* Presentation in the paper is good; the key steps are frequently outlined throughout the paper, the assumptions are clearly stated, the technical barriers and how they are dealt with are summarized nicely.\n\nWeaknesses:\n* Some of the results from the DP optimization literature which are used as a black box are outdated and the paper might be improved \"for free\" by using more recent results; see Questions for more detail.\n\nQUESTIONS:\n* Theorem 3.4 is somewhat outdated. For example, the authors could maybe use results from Section 5 of https://arxiv.org/pdf/2005.04763, which removes an unnecessary $\\log^2 (n/\\delta)$ factor from BST14, and also only requires a single pass which would alleviate the problem discussed in Section 3.3 \"Uniform Convergence of Empirical Likelihood\". Have the authors considered this / is it straightforward to integrate these to simplify the analysis and slightly improve the bounds?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new framework for differentially private (DP) statistical estimation using truncated statistics, focusing on unbounded exponential family distributions. It proposes an efficient DP algorithm for estimating parameters, like Gaussian mean and covariance, with optimal sample complexity. The method truncates data to control sensitivity, corrects bias using DP-SGD and maximum likelihood estimation, and adapts existing algorithms for truncated exponential families.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper’s is well written and clear, making it easy to follow. \n- The framework for combining truncation with DP is innovative, and recovers optimal sample complexities for Gaussian estimation. \n- The generality of the article make it applicable to many real-world scenarios\n\nWeaknesses\n- It can be hard to tell what’s new versus what’s built on existing results. \n- Moving basic exponential family facts to the appendix and diving deeper into technical contributions would help clarify the paper’s novelty. \n- Also, the paper doesn’t present a single numerical experiment, which weakens its practical validation and leaves questions about real-world performance.\n\nQUESTIONS:\n- At line 173, the article supposes that the distributions are absolutely continuous. Is this with respect to Lebesgue measure? If so, what is the technical step that is blocking with more general measures? Conceptually, shouldn’t it work as long as the empirical log-likelihood is almost surely differentiable (in order for first-order methods to make sense) and that the other assumptions are satisfied?\n- I did not see (but I may have missed it) an assumption on the boundedness of the set in which we are searching the parameter. However, without such an assumption, how do you guarantee that the loss is Lipschitz, which is necessary for the privacy of DP-SGD? Indeed, as the losses are supposed strongly convex, the gradients are coercive, and ensuring their boundedness typically requires a priori information on the location of the optimal parameter.\n- Linked to the last question, can the authors detail how the sample complexity depends on the size of the a-priori set in which we expect to find the optimal parameter?\n- In terms of pure technical contributions, is the main technical advancement Lemma 2.1?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for differentially private (DP) statistical estimation via data truncation, addressing the fundamental challenge of sensitivity control when the data support is unbounded. The authors propose a truncation-enhanced DP-SGD algorithm that enables efficient parameter estimation for exponential-family distributions while achieving near-optimal sample complexity. The theoretical contributions are significant.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1) Novel and interesting approach. Using truncated data to bound sensitivity in DP estimation is both theoretically sound and practically valuable.\n2) Technical innovations. Compared with [3], the complexity of estimating Gaussian statistics is reduced.\n3) Theoretical proofs. The authors make a substantial effort to establish rigorous theoretical results.\n\nWeaknesses\n1) Although the paper provides strong theoretical results for Gaussian data, it is unclear how to apply the method to real-world datasets. For example, how to choose the survival set in practice is not well explained; how to warm-start and preconditioning; and what failure probability beta is reasonable.\n2) The algorithm is more complex than classical DP methods, so it would be helpful to include visualizations to better illustrate the practical benefits over classical DP. It remains unclear whether the increased computational complexity is acceptable for large-scale datasets.\n3) Many key results rely on a set of strong and potentially difficult-to-verify assumptions. It remains unclear whether these assumptions are realistically satisfiable in practical scenarios. If these assumptions are violated, how the performance or guarantees of the proposed algorithm would degrade?\n4) Although many theoretical results are provided, no numerical experiments or empirical validation are included to demonstrate practical performance, convergence behavior, and related aspects.\n\nQUESTIONS:\n1. In practice, how should the survival set, i.e. the ball radius, be chosen?\n2. Could you elaborate on warm-start and preconditioning?\n3. What failure probability beta is reasonable for real-world data sets?\n\nLIMITATIONS:\nThe same as above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-06"}