Title: Second-Order Convergence in Private Stochastic Non-Convex Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: differential privacy

non-convex optimization

saddle points
Abstract: We investigate the problem of finding second-order stationary points (SOSP) in differentially private (DP) stochastic non-convex optimization. Existing methods suffer from two key limitations: \textbf{(i)} inaccurate convergence error rate due to overlooking gradient variance in the saddle point escape analysis, and \textbf{(ii)} dependence on auxiliary private model selection procedures for identifying DP-SOSP, which can significantly impair utility, particularly in distributed settings. To address these issues, we propose a generic perturbed stochastic gradient descent (PSGD) framework built upon Gaussian noise injection and general gradient oracles. A core innovation of our framework is using model drift distance to determine whether PSGD escapes saddle points, ensuring convergence to approximate local minima without relying on second-order information or additional DP-SOSP identification. By leveraging the adaptive DP-SPIDER estimator as a specific gradient oracle, we develop a new DP algorithm that rectifies the convergence error rates reported in prior work. We further extend this algorithm to distributed learning with heterogeneous data, providing the first formal guarantees for finding DP-SOSP in such settings. Our analysis also highlights the detrimental impacts of private selection procedures in distributed learning under high-dimensional models, underscoring the practical benefits of our design. Numerical experiments on real-world datasets validate the efficacy of our approach.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generic perturbed stochastic gradient descent (PSGD) framework based on Gaussian noise injection and general gradient oracles to address the problem of second-order stationary points (SOSP) in differential privacy (DP) stochastic non-convex optimization, addressing two limitations of existing methods: （1）Inaccuracy of convergence error rates due to neglect of gradient variance; （2）Utility degradation caused by reliance on auxiliary private model selection procedures. The paper also discusses the limitation of private model selection in high-dimensional distributed settings, and provides numerical experiments to support its approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper demonstrates specific flaws in their saddle point escape analysis in prior works. Based on these findings, the paper motivates and delivers an algorithmic redesign that resolves these inherent limitations.\n2. The paper introduces of Gauss-PSGD , which uses model drift distance as the escape trigger and allows the algorithm to avoid the need for high-overhead, additional DP model selection.\n3. The paper is technically sound and clear in its theorems, algorithmic procedures, and justification for each design and proof step.\n4. Another significant contribution is the extension to federated/distributed learning with arbitrary data heterogeneity.\n\n\n\n**Weakness**\n\n1. Parameter setting explanations (e.g., for step sizes, batch sizes, drift thresholds in Section 3.3 and 4.2) are quite technical and packed with dense notation. More intuitive explanations are required to help understand trade-offs.\n2. Although the results advance DP-SOSP theory, the discussion in the main paper is focused on theoretical mechanisms and error rates, with little commentary on how these results could impact industrial-scale machine learning tasks where privacy and distributed non-convex optimization interact.\n3. There is only a brief mention of numerical experiments in the abstract and summary, and detailed results are placed in the appendix. It is better to remove them to the main submission, especially the comparison with [29].\n4. Several important pieces, such as the detailed discussion of prior works, and experimental evaluations, are delegated to the appendix. A more self-contained and empirically substantiated presentation would considerably strengthen the case for publication.\n\nQUESTIONS:\n1. How are the hyperparameters, including the escape threshold, model drift threshold, maximum drift threshold, maximum escape steps, and maximum repeat number of escape, determined? .\n2. Beyond what is covered in the appendix, what important usage scenarios or risks where model drift-based DP-SOSP is especially helpful or potentially problematic?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This manuscript aims to find second-order stationary points in differentially private non-convex optimization. They propose a generic non-convex stochastic optimization framework PSGD, and provide an improved error rate bound. Besides, they extend their theoretical analysis to distributed learning and obtain the first error bound for DP-SOSP under arbitrarily heterogeneous data.

STRENGTHS AND WEAKNESSES:
**Strengths**
- This manuscript is well-organized, clearly presenting their theoretical analysis.

- They obtain the recified error rate compared to prior work for finding SOSP and obtain the first error bound for DP-SOSP under arbitrarily heterogeneous data.

**Weaknesses**

- I notice that authors have conducted experiments for varying privacy budget and the number of clients in the Appendix. I suggest authors briefly mention their empirical results, which validate the effectiveness of their method, in the main text for better readability.

- I acknowledge the statistical contribution of this manuscript. It would be better for machine learning community to understand if more explanations or insights are included. For example, why studying DP-SOSP is important; what insights we can get from all those theorems.

QUESTIONS:
- This manuscript mainly compares their result with "Private (stochastic) non-convex optimization revisited: Second-order stationary points and excess risks" regarding the population risk. What about the empirical risk, compared with SOTA and Lower bound?

LIMITATIONS:
Yes.(Section I in Appendix)
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the second order convergence guarantees of a differentially private stochastic non-convex optimization problem. Authors assume a sufficiently smooth loss function (Lipschitz, smooth and Lipschitz Hessian), and propose a generic optimization algorithm Gauss-PSGD that obtains the gradients from arbitrary gradient oracles. The algorithm tries to detect if the optimization has ran into a saddle point, and tries to escape it by continuing the stochastic optimization and checking if the parameter value has moved enough from the supposed saddle point. Authors prove several theoretical properties for the Gauss-PSGD: high-probability bound of escaping the saddle point, high-probability convergence guarantee into a 2nd order optimal point, and using specific DP gradient oracle show an improved error rate for DP non-convex optimization. Authors also extend the setting for a distributed learning, where the data can be heterogeneous among the clients. In this setting authors can recover very similar error rate as with the general DP non-convex optimization.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The paper presents a novel 2nd order error analysis for DP non-convex optimization problem. The proposed error rates improve (although somewhat moderately) over the existing bounds, allowing to better understand the limits of DP non-convex optimization.  (**Originality + Significance**) \n2. The extension to distributed setting is interesting, especially given that the prior work only has convergence results for the homogeneous data. (**Significance**) \n3. The proposed optimization framework Gauss-PSGD gives a simple unified framework for the non-convex optimization. For DP (and distributed DP) authors propose a specific gradient oracle Ada-DP-SPIDER, that tries to reduce the variance in the optimization. Authors demonstrate through some simple experiments that the proposed method out-performs an existing DP non-convex optimization method targeted for 2nd order guarantees. (**Significance + Quality**) \n4. Paper is rather busy, but the key points are explained and argued extremely well. (**Clarity**)\n\n## Weaknesses\n1. While the paper is mainly theoretical, it does propose an algorithm. So far, the experiments have been limited to somewhat simple models, so some more demanding tasks would strengthen the claims. Furthermore, having other baselines would allow putting the results in a better perspective. E.g. how would plain DP-SGD perform in the chosen tasks. (**Significance**)\n2. I think having the conclusions (amongst many other important sections) in the Appendix is a bit weird. I know that you operate under a strict page limit, but compressing the main paper does not seem impossible, especially as the conclusions and limitations allow contain some important discussion about the proposed method. (**Clarity**)\n\nQUESTIONS:\n1. Is there a reason to have the constants $c_1$ and $c_2$ in Thm 2.? I guess the constants arise from the moments-accountant-like analysis, and hence for a fixed $\\epsilon$ and $\\delta$ mainly affect the noise level, which (I guess) is not explicitly present in Thm 2. Maybe there is some reason to have them there, but to me it seemed just confusing to have the explicitly stated when the expression does not depend on them.\n2. The constants $c_1$ and $c_2$ in Thm. 2 remain a bit unclear to me. First, as they are not explicitly part of the bound, is is necessary to have them? Second, are these constants effectively the same as in moments accountant privacy analysis by Abadi et al. 2016?\n3. It seems that the Gauss-PSGD achieves better test accuracy with $\\epsilon=0.5$ than $\\epsilon=2$ (Fig. 2, m=10 result). Do you have any idea why this is?\n4. Does the single hidden layer NN you have selected for the experiments actually satisfy some of the regularity conditions (Assumptions 1 and 2)? If the objective is not Lipschitz, I guess you have used the biased clipped gradient oracle? And if so, can you report the clipping threshold for clarity sake.\n5. typo: \"requirment\"\n\nLIMITATIONS:\nYes, although the discussion is pushed mainly to the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new approach to find second-order stationary points (SOSP) in differentially private (DP) stochastic non-convex optimization. \nWith their analysis, the authors correct an existing convergence rate result by means of a more careful consideration of gradient variances when escaping saddle points.\nThe approach does not rely on separate private model selection and therefore can be applied to distributed settings.\nThe proposed Gauss-PSGD algorithm uses Gaussian perturbations and model drift to efficiently escape saddle points, and achieves improved convergence rates compared to existing results.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n\nThe proposed Gauss-PSGD algorithm improves (and corrects) upon existing work with respect to convergence error rates.\nIt also avoids the need for separate private model selection procedures to identify SOSPs, which is especially problematic in distributed settings due to high communication and privacy costs.\nAs a methodological contribution, the authors use model drift (instead of function value) to determine whether a saddle point has been escaped.\nThe extension to a distributed (heterogeneous) setting has been previously unaddressed by related work.\n\n\n## Weaknesses\n\n**Empirical evaluation.**\nThe empirical evaluation does not, in my opinion, satisfactorily support the practical utility of the method presented.\nWhile it is clear from your problem statement that your contributions are of theoretical rather than empirical nature, it would be important to see how your approach compares to other strong DP baselines.\nIn the classifications tasks you perform on MNIST and CIFAR-10 it is not clear how your approach compares to other state-of-the-art DP classification techniques.\nI do nevertheless acknowledge the importance of comparing your approach to the baseline method from Liu et al. [1], which more directly compares to your problem setting.\n\n**Theoretical assumptions and ease of implementation.**\nAs far as I can judge, your results seem theoretically sound.\nThey require, however, a number of assumptions which may easily not hold in real-world tasks.\nI think it would be important to comment to which degree the Lipschitzness and smoothness assumptions, as well as the necessity for hyperparameter tuning, affect real-world applicability.\nBig-O notation often omits, as commonly done, polylog factors.\nThis however makes it difficult to keep track of the actual impact of the different contributions in the bounds, which both hinders practical implementations and makes it difficult to understand how much the hidden factors weigh on the bound.\nA more structured presentation, in the appendix, of the bounds with some additional comments on the practical implications would be a useful addition.\n\n\n[1] Daogao Liu, Arun Ganesh, Sewoong Oh, and Abhradeep Guha Thakurta. Private (stochastic) non-convex optimization revisited: Second-order stationary points and excess risks. Advances in Neural Information Processing Systems, 36, 2024.\n\n**Minor comments.**\n\n* The paper lacks visual aids like diagrams, which could help explain saddle point escape or the distributed setup more intuitively.\n* The \"failure probability\" $\\omega$ appears for the first time in algorithm 1 without introduction.\nOther symbols like $\\eta$ (in the same section) also require introduction.\n* \"Saddle point escape/escaping saddle points\", as an expression, could use for some more formal definition.\n\nQUESTIONS:\n* Can the model drift criterion misidentify local minima as saddle points?\n* Are your results useful/applicable to some degree to, e.g.,  Rényi DP?\n* What happens if the gradient noise is not sub-Gaussian?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}