Title: Continual Release Moment Estimation with Differential Privacy
Venue: NeurIPS 2025
Accepted: yes
Keywords: Differential Privacy

Continual Release

Moment Estimation
Abstract: We propose *Joint Moment Estimation* (JME), a method for continually and privately estimating both the first and second moments of a data stream with reduced noise compared to naive approaches. JME supports the *matrix mechanism* and exploits a joint sensitivity analysis to identify a privacy regime in which the second-moment estimation incurs no additional privacy cost, thereby improving accuracy while maintaining privacy. We demonstrate JME’s effectiveness in two applications: estimating the running mean and covariance matrix for Gaussian density estimation and model training with DP-Adam.


Peer reviews:
comments: SUMMARY:
The authors propose a way to release both the first and second moments in a differentially private manner, concurrently. This is also specifically for data that is being continually released, that is streaming data. A standard approach to do such a thing is allocation some budget for each of the moments. As opposed to a standard approach the proposed work reformulates the problem as a matrix and do a joint privatization using the matrix mechanism.

STRENGTHS AND WEAKNESSES:
A large endeavor in DP is being able to save budget wherever possible. Being able to release a second summary statistic at no additional cost is a great strength. As a weakness, I don't truly see the "continual release" being used thoroughly. That is, it seems to me that the theorems do not take into account what happens at time t.

QUESTIONS:
In section 5 it is mentioned that the covariance matrices are symmetrized and projected onto the cone. Can the authors elaborate on this. How did they do the projection? If there was some clipping of eigen values, how was the clipping constant chosen?

I believe Figure 3 has an error as it states n=100 but the steps is 200.

Minor typo on page 5 toprivatize -> to privatize
Minor consistency issue, the appendix is referred to as both the "Appendix" and "appendix".

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Joint Moment Estimation, a method for continual and differentially private estimation of both the first and second moments of streaming data. Unlike naive approaches that split the privacy budget between estimates, JME uses a joint sensitivity analysis to estimate the second moment without extra privacy cost—achieving what the authors call “privacy for free.” It is built upon the matrix mechanism and is adaptable to various settings (e.g., exponential weights or sliding windows). The authors prove JME's privacy and utility guarantees and show its practical benefits through applications to private Gaussian density estimation and differentially private Adam optimization, demonstrating better performance under high-privacy constraints compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n- The paper is well-structured, with clear algorithmic descriptions.\n\n- The paper compares JME against multiple relevant baselines (IME, CS, PP).\n\n- For the weakness, please refer to the questions.\n\nQUESTIONS:\n- Figure 1: Pareto frontier behavior of \\$\\lambda\\$-JME. Do different values of \\$\\lambda\\$ in \\$\\lambda\\$-JME result in different Pareto frontiers? (Do all \\$\\lambda\\$-JME variants share the same frontier, or does the error trade-off change with \\$\\lambda\\$?) If so, how should we interpret the dominance order of \\$\\lambda\\_1\\$-JME versus \\$\\lambda\\_2\\$-JME when \\$\\lambda\\_1 < \\lambda\\_2\\$?\n\n- Could the authors discuss the computational efficiency of the proposed method, and compare it with the baselines?\n\n- Interpretation of “privacy for free”: In line 90, the paper states that “the total sensitivity of estimating both moments...” Can this be more precisely interpreted as “the sensitivity of a linear combination of the first and second moment estimates (parameterized by \\$\\lambda\\$)”? If so, clarifying this relationship could help make the core intuition of \"privacy for free\" more transparent.\n\n- The notation \\$\\Vert \\cdot \\Vert\\_{1\\to 2}\\$ appears before it is explained (e.g., in Algorithm 1), but its definition is given later in line 122. This makes the algorithm hard to parse on first read. It would improve readability to define or briefly explain this operator norm earlier.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method called Joint Moment Estimation (JME), designed for continual, differentially private estimation of both first and second moments of a data stream. JME is tailored for continual release settings—where estimates are updated as new data arrive—and is based on the matrix mechanism for differential privacy.\nThe key innovation lies in a joint sensitivity analysis, which reveals a privacy regime where the second moment can be estimated without additional privacy cost relative to the first moment. This property is described as “privacy for free” for the second moment. The paper provides both theoretical guarantees and empirical results. JME achieves better accuracy than baseline methods, especially in high-privacy or small-batch settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThis paper makes strong theoretical and practical contributions by providing rigorous guarantees on privacy, utility, and unbiasedness for the proposed JME algorithm. It demonstrates that second-moment estimates can be achieved with \"privacy for free\" through careful joint sensitivity analysis. Moment estimation being a core component in many machine learning applications further highlights JME’s real-world relevance. Empirically and theoretically, JME is shown to Pareto-dominate existing baselines such as IME, CS, and PP across utility-privacy trade-offs. The algorithm's design—leveraging matrix mechanisms and Khatri-Rao products—is both mathematically elegant and clearly presented.\n\nWeaknesses:\nThe main limitations of the paper lie in its reliance on bounded input assumptions and noise-shaping matrices, which may be difficult to select or tune in practice. Additionally, the method has only been evaluated in relatively controlled or benchmark settings, lacking demonstrations in more complex, real-world machine learning pipelines. While the paper briefly mentions debiased post-processing as a baseline, it does not deeply explore its practical performance, and the theoretical complexity of the proposed approach may pose a barrier to broader adoption.\n\nQUESTIONS:\n1. Could JME be extended to support higher-order moments (e.g., skewness, kurtosis) in a similarly privacy-efficient way?\n2. How sensitive is JME’s performance to the choice of noise-shaping matrices , ? Are there principled ways to select them for arbitrary workloads?\n3. Can JME be integrated with federated learning or decentralized streaming?\n\nLIMITATIONS:\nThe authors acknowledge that the method assumes bounded norm constraints on the data and relies on the matrix mechanism framework, which may not be straightforward for all users. Additionally, while the paper claims \"privacy for free\", this is bounded to the setting of joint sensitivity, and does not generalize to all DP compositions. Also, the performance is not extensively tested in large-scale or production scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new method named Joint Moment Estimation (JME) for continual differentially private estimation of the first and second moments of a stream of data in arbitrary number of dimensions. The method uses Gaussian noise. First and second moments are written both as matrix multiplications. Then their joint sensitivity is found, and Gaussian noise is added to both moments scaled according to the joint sensitivity. Key to reducing the noise level is to user different linear transformations and scaling factors on first and second moments to control the noise levels, hence making the joint sensitivity small. Consequentially, the second moment does not necessarily increase the noise level for the first moment. Results show improvements compared to more traditional approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.  If I understand this correctly, the results in this paper indicates that by using approximate matrix multiplications and scaling on the Gaussian noise, we can add the same noise level to the first moment. The theoretical result is interesting and promising, since all privacy-budget splitting mechanisms should fail to achieve this. \n2. Even when the scaling factor $\\lambda$ does not give free second moment estimates, JME still pareto-dominates the IME algorithm.\n3. The paper studies the continuous computation of the moments, which adds to its importance and makes it applicable for a wider variety of practical problems.\n4. The algorithm is easily implementable.\n\nWeakness:\n1. The improvement over baselines in the experiments seem to be more moderate than the theory suggests. It seems that we would need $\\varepsilon$ to be really small for the new algorithm to beat the baseline in DP-Adam. I wonder why that is.\n2. Getting the second moment \"for free\" does not necessarily mean that its noise level is small, and some equations cannot be solved in closed form so it's hard to further characterize the improvement over more traditional approaches.\n\nQUESTIONS:\n1. This is a minor issue, but I believe the notation $||\\cdot ||_{1 \\rightarrow 2}$ has already been used before it becomes formally defined.\n2. In the joint sensitivity (and hence noise level computation) the matrices $C_1$ and $C_2$ only comes out in the $||\\cdot||_{1 \\rightarrow 2}$ norm. Is that the only thing abut them that affects the noise level? Will it ever be more beneficial to use $C_1$ and $C_2$ other than the identity matrix $I$?\n3. I find it a bit counter-intuitive that $c_d$ should remain a constant for every $d \\geq 2$, but I didn't have to time to go through the math calculations in the appendix. Do the authors have a more high-level explanation of that?\n4. The results of first moment not needing a bigger noise scale is surprising to me. I didn't check the math step by step so my confidence in this conclusion is not as high as it should be. What's the key difference between the joint sensitivity definition and the vector sensitivity in the CS method that lead to this result? For, when you concatenate the moments into a $d(d+1)$-dimension vector you can also apply the matrix $C_1$ and $C_2$, and $\\lambda$ coefficient first, then take the norm and add noise according to it. Or are they roughly equivalent? Do you have any high-level intuition for that?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}