Title: Accelerating Model-Free Optimization via Averaging of Cost Samples
Venue: NeurIPS 2025
Accepted: yes
Keywords: Optimization

Model-Free Optimization

Zeroth-Order Optimization

Derivative-Free Optimization
Abstract: Model-free optimization methods typically rely on cost samples gathered by perturbing the current solution estimate along a finite and fixed set of directions. However, at each iteration, only the current cost samples are used, while potentially informative, previously collected samples are discarded. In this work, we challenge this conventional approach by introducing a simple yet effective memory mechanism that maintains an auxiliary vector of iteratively updated cost samples. By leveraging this stored information, our method estimates descent directions through an averaging of all perturbing directions weighted by the auxiliary vector components. This results in faster convergence without increasing the number of function queries. By interpreting the resulting algorithm as a time-varying dynamical system, we are able to establish its convergence properties in the strongly convex case. In particular, by using tools from system theory based on timescale separation, we are able to guarantee a linear convergence rate toward an arbitrarily small neighborhood of the optimal solution. Numerical simulations on regression problems demonstrate that the proposed approach significantly outperforms existing model-free optimization methods.


Peer reviews:
comments: SUMMARY:
This paper proposes a simple memory mechanism for single-sample zeroth-order optimization. Instead of discarding past cost evaluations, the algorithm maintains a buffer that stores the most recent cost of each predefined perturbation direction. Under strong convexity, the method converges at a linear rate to a neighborhood of the optimal solution.

STRENGTHS AND WEAKNESSES:
### Strengths
1. Appending a simple memory mechanism to a classical single-point ZO method is both novel and easy to implement. 

2. Convergence is rigorously established, with explicit rates provided.


### Weaknesses
1. Because the analysis relies on strong convexity and noiseless evaluations, the current theory may not cover many general convex problems. 

2. Experiments focus on a 50-dimensional synthetic logistic-regression task and compare against only two baseline methods. Additional studies on larger dimensions, real-world black-box tasks, or stronger baselines would better demonstrate practical advantages.

3. The paper reports that parameters (e.g., step size gamma) were tuned manually, which may introduce bias into the comparisons.

QUESTIONS:
1. Can the convergence analysis be generalized to weaker conditions, for example, smooth (but not strongly) convex objectives or stochastic noise in function values? 

2. Would the authors consider adding stronger baselines and real-world benchmarks? In addition, given that the synthetic experiment uses a modest dimension (n=50), might larger-scale experiments be included to illustrate scalability?

3. Have you explored a principled or adaptive strategy for selecting the parameters (e.g., gamma and epsilon)?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a memory-based paradigm for model-free (derivative-free) optimization. The key innovation is an auxiliary memory mechanism that stores function evaluations for each perturbation direction and reuses them across iterations. The algorithm maintains memory variables that are updated when new function evaluations become available and uses weighted averages of all stored samples to estimate descent directions. The authors also provide theoretical proof of this method, then use numerical experiments on logistic regression demonstrate its performance.\n\nSTRENGTHS AND WEAKNESSES:\nThe memory mechanism is intuitive and makes sense. This paper also use the two-timescale system framework to provide rigorous theoretical analysis.\n\nThe idea of using historical information is well-established in optimization (momentum, Adam, etc.), though the application to zeroth-order methods is novel.\n\nThis method requires Lipschitz continuous gradients and strong convexity, limiting applicability to broader function classes.\nAnd the numerical experiment only tested on logistic regression, lacking evaluation on other problem types also didn't compare with SOTA derivative-free optimization algorithms.\n\nQUESTIONS:\n1. The experiments are limited to 53-dimensional logistic regression. Could the authors provide results on different dimensional case?\n\n2. In chapter 5, the author mentioned γ is manually selected, and also different for two method. It would be better to show how different γ affect the result and compare the results with same γ. Could the authors provide sensitivity analysis of performance to parameter choices\n\n3. Since the method needs to store a vector, can you discuss how memory efficiency and computational cost in your method?\n\nLIMITATIONS:\nNo. I think that the sensitivity to parameter choices is limited in discussion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a meta-algorithm for zeroth-order optimization designed to accelerate convergence without increasing the number of function queries per iteration. The key idea is to challenge the standard practice of discarding cost function samples from previous iterations. The proposed method maintains an auxiliary vector of stored cost samples, one for each of a fixed set of $D$ perturbation directions. When a cost is evaluated for a particular direction, its corresponding entry in the memory vector is updated; otherwise, the old value is retained. The descent direction at each step is then calculated by averaging all $D$ perturbation directions, weighted by their corresponding cost samples from the memory vector.\n\nThe authors provide a rigorous theoretical analysis for the strongly convex case by interpreting the algorithm as a time-varying two-time-scale dynamical system. They prove that the method achieves a linear convergence rate to an arbitrarily small neighborhood of the optimal solution.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The idea of maintaining and reusing the most recent cost sample for each perturbation direction is simple, intuitive, and, to the best of my knowledge, novel in this context.\n\n2. The convergence analysis is a major strength. Modeling the algorithm as a two-time-scale system is an appropriate choice. The resulting guarantee of linear convergence to a tunable neighborhood (Theorem 1) is a strong theoretical result for this class of algorithms. The inclusion of the detailed derivation in the appendix (especially Theorem 2) is a valuable contribution in its own right.\n\n3. The algorithm's main promise is faster convergence without increasing function queries. The method appears easy to implement and adds only a small memory overhead ($D$ scalars).\n\nWeaknesses:\n\n1. The core idea is not too different from coordinate descent, where descent is performed in one direction at a time. In fact, a very recent work by Karandikar et al. (Revisiting Stochastic Approximation and Stochastic Gradient Descent) has strong parallels with the proposed work in a more general setting, where they also derive sufficient conditions for the convergence of zero-order SGD, wherein the stochastic gradient is computed using $2D$ function evaluations, but no gradient computations. As such the core idea does not appear to be novel.\n\n2. Assumption 3 introduces $T_{\\max}$, the maximum time between sampling any given direction. The convergence rate in Theorem 1 depends on $1/T_{\\max}$. This suggests that using more directions $D$ (which would likely increase $T_{\\max}$ under cyclic sampling) could lead to a slower convergence rate. This creates an interesting trade-off: more directions might give a better gradient estimate, but the staleness of information could slow convergence. A more explicit discussion of this trade-off would improve the paper. Also, convergence of algorithms under assumptions of a constant such as $T_{\\max}$ is widely dealt in the distributed optimization literature wherein the network is assumed to be connected over any $T_{\\max}$ time (though it may be disconnected at a given instant).\n\n3. The introduction positions the work against methods that discard samples. However, it would be valuable to contrast this \"memory of costs\" approach with other memory-based ZO methods, such as those using momentum or filter-based techniques (like the cited Chen et al. [2022]). The proposed memory is on the raw function values $l(x_t + \\epsilon d^j)$, not on the aggregated gradient estimate or the iterates themselves. Highlighting this distinction more clearly would further sharpen the paper's contribution.\n\nA minor typo: (8) should have $z_t^j$.\n\nQUESTIONS:\nThank you for this interesting and well-written paper. I have a few questions to better understand the nuances of your work:\n\n1. On the $T_{\\max}$ Trade-off: The convergence rate in Theorem 1 includes the term $\\min{\\gamma \\mu, \\kappa}/T_{\\max}$. This seems to imply that a larger $T_{\\max}$ (resulting from using more directions $D$ with single-sample-per-iteration cyclic updates) would slow down the linear convergence factor. This appears to be in tension with the need for a sufficiently rich set of directions to accurately approximate the gradient. Could you comment on this trade-off? Is there an optimal number of directions $D$ that balances the quality of the gradient estimate against the staleness of the cost samples?\n\n2. Stochastic Sampling Analysis: You mention that Assumption 3 could be extended to a stochastic setting. Could you elaborate on the main technical challenges in doing so? In your two-time-scale analysis, the fast-layer dynamics for $z$ converge due to the guaranteed periodic updates. If the selector $s_\\tau^j$ becomes a random variable, how would you ensure the fast subsystem $z_t$ remains sufficiently close to the equilibrium $L_\\epsilon(x_t)$ for the slow dynamics to be well-approximated by the reduced system?\n\n3. Comparison to Momentum: How do you see your memory mechanism relating to momentum-based methods in ZO optimization? Momentum methods also leverage past information by maintaining a velocity vector, which is an exponential moving average of past gradient estimates. Your method, in contrast, stores raw cost samples. What are the conceptual advantages of storing costs over storing aggregated gradient estimates?\n\nLIMITATIONS:\nI can think of a few limitations:\n\n1. Reliance on Strong Convexity and Deterministic Sampling: The theoretical analysis (Theorem 1) relies on two strong assumptions. First, the objective function $l$ is assumed to be $\\mu$-strongly convex (Assumption 1). This is crucial for establishing the stability of the reduced system (Lemma 2) and ensuring that the gradient descent dynamics converge to a unique optimum. Extending the analysis to the general convex or non-convex case is a non-trivial but important next step. For non-convex functions, it is unclear if the method would be guaranteed to converge to a stationary point or how the memory mechanism would interact with local minima and saddle points. Of course, lack of strong convexity will probably not guarantee linear convergence, which is expected.\n\nSecond, the proof of linear convergence hinges on the deterministic, essentially-cyclic sampling condition (Assumption 3), which guarantees that every direction is sampled within a finite window $T_{\\max}$. This is essential for the stability of the boundary-layer system (Lemma 1), as it ensures that no component of the memory vector $z_t$ can remain stale indefinitely. Many popular ZO methods employ random sampling of perturbation directions. Adapting the analysis to such a stochastic setting would require different analytical tools, likely establishing convergence in expectation or with high probability.\n\n2. Memory and Dimensionality Trade-off: The core of the method is a memory vector $z$ of size $D$, where $D$ is the number of perturbation directions. This introduces a memory cost of $O(D)$ that is not present in standard single-point methods. For many gradient estimation techniques, $D$ scales with the problem dimension $n$ (e.g., $D=2n$ for a two-sided gradient estimator). While this is acceptable for moderately-sized problems, the memory overhead could become a practical limitation in very high-dimensional settings where $n$ is in the millions. This presents a clear trade-off: the algorithm trades $O(D)$ memory for faster convergence.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a zeroth-order gradient method for unconstrained convex optimization when the gradient of the objective (or cost) is unavailable, but can be estimated at a point $x$ using cost evaluations at perturbations of $x$ with respect to a finite and fixed set of directions. Precisely, they assume that the gradient of the cost, $\\nabla l(x) = \\sum_{j=1}^{D}g_{\\epsilon}(l(x+\\epsilon d^j),d^j) - e_{\\epsilon}(x)$ for known directions $d^1,d^2,\\cdots,d^D$. In the setting where it is not possible to simultaneously use all the perturbation directions in each optimization step, so that cost evaluations $l(x_t+\\epsilon d^j)$ at a current solution $x_t$ is not available for $j=1,\\cdots,D$, they propose a descent method that reuses cost evaluations from previous optimization steps to estimate the gradient $\\nabla l(x_t)$ in round $t$. Then, under three main assumptions: strong-convexity, smoothness and an essentially-cyclic condition, they prove linear convergence of the last-iterate to a unique optimal solution. Their proof follows from an involved analysis of the descent method as a two-timescale dynamical system.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\nThe authors address an interesting problem in unconstrained convex optimization where the cost function and its gradient are unknown, however cost evaluations along a finite and fixed set of directions, when available, can be used to estimate the gradient. Moreso, their two timescale interpretation of the resulting descent method, and its analysis as a fast vs slow dynamical system, are quite interesting. Finally, from a quick glance at the technical content in the main text seem accurate.\n\n### Weakness\nThe paper is motivated by claims that multi-point zeroth other methods are too expensive, while existing single-point methods suffer from slow convergence. However, there is no mention of what kind of rates have been achieved in prior works. To properly validate the authors claim of “faster convergence” with their method, I believe it will be helpful to reference notable convergence rate results in related literature especially from Chen et al. 2022, Xiao et al 2023 and Zhang et al 2024 – which reportedly already achieve improved performance in terms of convergence of single-point methods.\n\nQUESTIONS:\n### Some questions\n1. Can the authors elaborate on what they mean by “model-free” in the context of their paper?\n2. Just for clarity, can the authors also explain the essentially-cyclic condition in Assumption 3 in lay terms, and how this helps in the analysis?\n3. The algorithm is easy to follow, but I had a conflicting opinion about the definition of the auxiliary variables in Equation 7. Consider the intended descent step $x_{t+1} = x_{t}-\\gamma\\nabla l(x_{t})$. Under Assumption 2, the previous expression would be equivalent to $x_{t+1} = x_{t}-\\gamma\\bigg(\\sum_{j=1}^{D}g_{\\epsilon}(l(x_t+\\epsilon d^j),d^j) - e_{\\epsilon}(x_t)\\bigg)$. However, since the error is unknown and cost evaluations may not be available, the authors propose the update $x_{t+1} = x_{t}-\\gamma\\sum_{j=1}^{D}g_{\\epsilon}(z_t^j,d^j)$ in Equation 8 where $\\\\{ z\\_t^{j} \\\\} \\_{j=1}^{D}$ are introduced as auxiliary variables with $z_t^j$ storing the cost evaluation along direction $d^j$ if available at step $t$. My confusion lies with the fact that $z_t^j$ is defined in Equation 7 to collect cost samples at $x_{t-1}$ i.e $z_t^j=l(x_{t-1}+\\epsilon d^j)$. Meanwhile, I expected that $z_t^j=l(x_t+\\epsilon d^j)$ so that in the best case: when all the cost samples are available and Assumption 2 holds exactly (with no errors), $g_{\\epsilon}(z_t^j,d^j)=g_{\\epsilon}(l(x_t+\\epsilon d^j),d^j)$ and Equation 8 simply executes a true gradient descent update.\n\n    It is interesting that the analysis still seems to check out nonetheless. Precisely, $z_t^j=l(x_t+\\epsilon d^j)$ is the equilibrium point of the dynamic system formed by definition of the auxiliary variables in Equation 10b.\n\n4. L260: Is this expression $F(0,t)=0$ obvious. Also what does $\\xi=0$ mean? It would help if there is some distinction between scalar and vector variables in the paper, for example the real value $0$ and vector of all zeros.\n5. L535: I assume the authors also use the expression in equation 14?\n\n### Some typos\n1. L109: is this missing an index? The LHS is a vector while the RHS is a scalar value.\n2. L132: Do you mean $l(x_t,\\epsilon d^j)$ instead of $\\nabla l(x_t,\\epsilon d^j)$.\n3. Equation 8: Should be $z_t^j$ instead of $z_t$.\n4. Equation 12b: Should be $\\tilde{x}_t$ instead of $\\tilde{x}$\n5. Equation 24: $V$ is a multi-variable function by definition but is defined here with single input variables.\n6. L255: $\\xi^t$ should be $\\xi’$.\n7. Equations 35 and 36: These are exactly the same expression.\n8. L540: you mean $L$-smooth?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}