Title: Zeroth-Order Optimization Finds Flat Minima
Venue: NeurIPS 2025
Accepted: yes
Keywords: Zeroth-order optimization

Convex optimization

Implicit regularization

Flat minima
Abstract: Zeroth-order methods are extensively used in machine learning applications where gradients are infeasible or expensive to compute, such as black-box attacks, reinforcement learning, and language model fine-tuning. Existing optimization theory focuses on convergence to an arbitrary stationary point, but less is known on the implicit regularization that provides a fine-grained characterization on which particular solutions are finally reached. We show that zeroth-order optimization with the standard two-point estimator favors solutions with small trace of Hessian, which is widely used in previous work to distinguish between sharp and flat minima. We further provide convergence rates of zeroth-order optimization to approximate flat minima for convex and sufficiently smooth functions, where flat minima are defined as the minimizers that achieve the smallest trace of Hessian among all optimal solutions. Experiments on binary classification tasks with convex losses and language model fine-tuning support our theoretical findings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the implicit regularization properties of zeroth-order (ZO) optimization methods, specifically focusing on the behavior of the standard two-point gradient estimator. The central claim of the paper is that ZO optimization with the two-point estimator implicitly favors and converges to \"flat minima\". The authors provide the theoretical analysis to support their method. Also, authors provide empirical validation across several settings to support their theoretical findings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.  Rigorous Theoretical Analysis for Convex Case.\n2. The paper is generally well-written and logically structured.\n3. The experimental results provide support for the theoretical claims, especially in convex settings.\n\nWeaknesses:\n1. Previous work on SAM-based models has primarily focused on analyzing their methods in non-convex cases. If the entire region under consideration were convex and smooth, it is likely that many existing methods would demonstrate good properties. Therefore, evaluating the theoretical findings of this paper becomes challenging. I believe the lack of analysis for non-convex cases and the absence of a theoretical comparison with other methods weaken the paper's contribution.\n\n2. It is surprising that a two-point gradient estimator can achieve good performance in learning tasks. In my opinion, this type of method lacks second-order information and should theoretically struggle or fail in ill-conditioned optimization problems. The authors should provide numerical results on well-known optimization problems, such as the Lévy, Ackley, and Rastrigin functions, to support their method's convergence claims.\n\n3. The paper lacks a sufficient comparison with baseline methods. I believe there are more than ten relevant zeroth-order (and potentially simple first-order) methods, but the authors choose not to compare their approach against any of them.\n\n4. In Figure 3, it appears that the methods have not fully converged.\n\nQUESTIONS:\nSee the weaknesses part.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that a smoothed function based on a two-point zeroth-order gradient estimator incorporates the trace of the Hessian matrix implicitly as a regularization term. Zeroth-order minimization thus implicitly favors small eigenvalues of the Hessian. A small trace of the Hessian matrix has been shown to correlate with flat minima. The authors give a proof of convergence of zeroth-order optimization to flat minima, and show experimentally that the trace of the Hessian is smaller for zeroth-order than for first-order optimization methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Nice use of Taylor's theorem to show the connection of two-point gradient approximation and Hessian matrix (eq.2)\n- Rigorous mathematical proof of convergence\n\nWeaknesses:\n- Very strong assumptions of function to be optimized being twice continuously differentiable and convex\n- Experimental results consistent with the theory are only found for convex losses (SVM, logistic regression), but not for non-convex losses (LLMs)\n- No attempt to link flat minima to improved generalization\n\nQUESTIONS:\n1. Zeroth-order optimization is particularly interesting if it enables optimization of non-differentiable functions by Gaussian smoothing. However, your work depends on twice continuously differentiable objective functions (eq. 2). Do you have an idea if your work could be extended to non-smooth C^{0,0} functions?\n2. Why is it that experimental results consistent with the theory are only found for convex losses (SVM, logistic regression), but not for non-convex losses (LLMs)? Is the convexity assumption perhaps more that a theoretical tool to prove convergence?\n3. Isn't the main claim of convergence to flat minima the fact that such methods show improved generalization? Is this the case in your experiments?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-24"}

comments: SUMMARY:
This paper studies the convergence of zeroth-order algorithms toward flat minima. Specifically, the authors analyze the convergence rate of a zeroth-order algorithm employing the standard two-point estimator, to solutions with minimal Hessian trace within the optimal solution set for smooth convex functions.
 Experiments on SVM, logistic regression, and language model fine-tuning validate the theoretical findings.

STRENGTHS AND WEAKNESSES:
Strengths

1. This paper is well written and easy to follow. The derivation of implicit regularization and flat minima (Sections 2 and 3) are clear and easy to understand.

2. The derivation of implicit regularization on Hessian trace in zeroth-order algorithms is both novel and theoretically interesting.

3. The convergence analysis in Section 3.1 is clearly presented, offering valuable theoretical insights.


Weaknesses

1.  This paper primarily focuses on the convex setting, where the Hessian matrix is positive semi-definite, making the trace of the Hessian an appropriate measure of sharpness. However, it remains unclear whether the conclusions in this paper hold in non-convex scenarios, although the derivation of implicit regularization on the Hessian trace does not require convexity assumptions.

QUESTIONS:
1. Can the results in this paper be extended to non-convex settings? Since the derivation of implicit regularization (Eq.(3)) does not require convexity assumptions.

2. In Figure 3, it seems that GD is more effective than ZO in reducing the Hessian trace. Could the authors provide further explanation for this phenomenon?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates an implicit bias in two-point zeroth-order optimization of $C^3$ functions towards flatter minima, as measured by the Hessian trace, and provides convergence analysis. This is done via a smoothed surrogate, which is shown to implicitly encode a regularizer term containing the Hessian trace. The smoothing parameter is framed as a trade-off between this regularization and the optimization error induced by a bias term. Experiments on binary classification problems in the overparameterized setting, and fine-tuning language models are shown.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* This paper has very clear writing and proofs are relatively easy to follow. The paper also clearly distinguishes its analysis from that provided by Nesterov.\n* The paper addresses an important problem of explicitly connecting zeroth-order optimization to the research on flat minima.\n\n## Weaknesses\n\n* In Appendix D.2, the authors state that they chose the learning rate and smoothing parameter via grid search. What metric was used to perform this? In [1], the authors, show the batch GD (which is used in this paper) can also converge to flatter minima (compared to gradient flow), and establish a linear convergence rate under the edge of stability regime. Therefore, this paper's evidence may be stronger if the metric used to compare also takes into account the sharpness (via the Hessian norm, for example) of the solutions, rather than just a validation loss (which I assume was used here).\n* On L297 (page 9), can you expand on the $2/\\delta^2$ constant in your estimation of sharpness? I could not find that leading constant in either Neyshabur et al. (ref. 75) or Zhu et al. (ref. 103). In the former, the closest equation is (5), which uses $\\mathbb{E}\\_{u \\sim \\mathcal{N}(0, \\sigma I)} [f(x + u)] - f(x)$, and Zhu et al. use (from their Supplementary), $\\mathbb{E}_{u \\sim \\mathcal{N}(0, \\delta^2 I)} [f(x + u)] - f(x)$.\n* There are relatively few experiments, and these do not account for the stochastic nature of the methods (caused, e.g., by random initialization). That said, I do not think empirical analysis is the point of this paper, and that the experiments are moreso for a cursory confirmation of the established theory.\n\n[1] Beneventano, P., & Woodworth, B. (2025). Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks. arXiv preprint arXiv:2501.09137.\n\nQUESTIONS:\n1. Could you clarify the $2/\\delta^2$ constant in the sharpness estimation equation, via a simple derivation, for instance?\n2. Do the results of your experiments change when the grid search is used to prioritize both validation loss and Hessian norm? If not, can you comment on why your experiments show such a strong bias (as a trade-off for convergence rate) for low Hessian trace compared to batch GD?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical analysis of zeroth-order (ZO) optimization, revealing its implicit regularization effect towards finding flat minima. \n\nThe authors provide a theoretical framework to explain this phenomenon\nThe core idea—that the smoothing inherent in ZO methods acts as a regularizer minimizing the trace of the Hessian—is both insightful and elegant. \n\nThe theoretical contributions are well-supported by proofs.\nWhile the experiments provide corroborating evidence, there are some questions regarding the practical implications in highly non-convex settings. \nOverall, this is a paper that makes contribution to our understanding of zeroth-order methods\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nNovelty and Conceptual Contribution: The primary strength of this work lies in its novel perspective. While the analysis of ZO methods via smoothed functions is standard, this paper is the first to re-interpret the Hessian trace term not as a bias to be suppressed, but as a desirable regularization term to be embraced. This leap, connecting a classic optimization technique to the modern pursuit of flat minima for better generalization, is a fundamental contribution.\n\nTheoretical Rigor: The paper's claims are backed by solid theoretical work. The authors formally define flat minima in a global context, construct a novel proof strategy around a regularized objective function F(x), and provide a full convergence complexity analysis (T = O(d⁴/ε²)) for finding approximate flat minima. The technical depth required for the proofs, such as handling 8th-order moments of Gaussian variables, demonstrates the non-triviality of the theoretical analysis.\n\nPoints for improvement\n\nWhile the theoretical contributions are clear, I have some questions regarding the experimental section, particularly concerning the practical implications of the findings in the non-convex LLM fine-tuning setting.\n\nOn the Practical Significance in the Non-Convex Setting: The authors are commended for extending their investigation to the challenging setting of fine-tuning a 355M parameter RoBERTa model. However, the practical significance of the findings is not entirely clear from the results presented. As shown in Figure 2 and 3, standard gradient descent (GD) often achieves a lower training loss, a smaller Hessian trace (i.e., finds an even \"flatter\" solution), and higher test accuracy.\n\nQUESTIONS:\nThis raises the question of the practical utility of using ZO for this task. \n\nIf ZO does not offer a clear advantage in either final performance or the \"flatness\" of the solution compared to a well-tuned first-order method, what is the key takeaway for a practitioner? \nThe paper convincingly shows that ZO also seeks flat minima, but it does not make a strong case for why one would prefer it over GD in this realistic, non-convex scenario.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}