Title: PseuZO: Pseudo-Zeroth-Order Algorithm for Training Deep Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: zeroth-order optimization

LLM fine-tuning
Abstract: Zeroth-order Optimization (ZO) has received wide attention in machine learning, especially when computing full gradient is expensive or even impossible. Recently, ZO has emerged as an important paradigm for memory-efficient fine-tuning of large language models (LLMs), circumventing the memory overhead of backpropagation. However, existing ZO gradient estimators exhibit dimension-dependent variance scaling as $\Theta(d)$, leading to dimension-dependent convergence rates without further assumptions on the objective function, which is prohibitive for large-scale LLM parameters. To address this problem, we present a Pseudo-Zeroth-Order (PseuZO) framework for optimizing composite objective functions, especially large-scale models: $ \min_{\mathbf{x} \in \mathcal{X}} \mathcal{F}(\mathbf{x})= \bbE_{\mathbf{z}} g\circ h(\mathbf{x};\mathbf{z}) $, where $h$ represents complex, high-dimensional representations and $g$ is a task-specific loss. While existing zeroth-order methods estimate gradients with final loss functions, our PseuZO algorithm estimate the Jacobian matrix of $h(\mathbf{x})$ with the model output $\mathbf{o}= h(\mathbf{x})$, and the gradient of the loss function on model output $\mathbf{e} = \nabla_{\mathbf{o}} g(\mathbf{o})$, and apply exponential moving average on Jacobian estimators to reduce the variance. Moreover, we use the sliding window technique to reduce memory costs. Our algorithm achieves an $O( \max \lbrace \alpha_1 L\epsilon^{-2}, \alpha_1 L \sigma_2^2\epsilon^{-4} \rbrace )$ convergence rate, where $\alpha_1$ is the effective dimension of $\mathcal{F}$.
  Experimental results demonstrate that PseuZO outperforms MeZO and MeZO-SVRG in classification, multiple choice and generation tasks in both full-parameter and PEFT fine-tuning settings by boosting convergence in the early stages of training. For instance, under the same computation time, with respect to SST2 task, PesuZO gets 9.8\% higher accuracy than MeZO (91.2\% v.s. 82.4\%). With the sliding window technique, our PseuZO achieves $70\%\sim80\%$ memory reduction compared to FO-SGD for different model sizes as PseuZO only introduced a small dimension-independent memory overhead, which enables efficient scaling of the model size. The code is available at https://github.com/YangBigMn/PseuZO.
$\newcommand{\bbE}{\mathbb{E}}$


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm PseuZO based on zeroth-order gradient for deep learning. Rather than directly using the loss value to compute the zeroth-order gradient, this paper turns to optimize a composite objective function $g(h(x,z))$ where $h$ represents the output of a neural network and $g$ represents a loss function. PseuZO computes the zeroth-order gradient for the function $h$ and uses the closed-form for the function $g$ which is with respect to $h$, then combines them to obtain the gradient for the model update. Since the first-order momentum consumes large memory cost, this work also designs a variant algorithm based on the sliding window. The authors make a convergence analysis for PseuZO. In the fine-tuning experiments, PseuZO achieves greater performance than MeZO across different llms. In addition, the proposed algorithm only introduces little extra memory cost.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThis paper proposes an optimizer PseuZO for training deep neural networks. The authors decouple the traditional loss function to a composite form. This is an innovative point which is different from related works.\n\n2.\tThe statement of the idea is clear and the organization of the paper is reasonable.\n\n3.  The authors provide the detailed theoretical analysis for the proposed algorithm.\n\n4.  The experiments cover multiple metrics and contents, including the time and memory cost and the ablation study.\n\nWeaknesses:\n\n1.\tThis paper lacks the explanation or theoretical analysis to show that why PseuZO works or achieves improvement compared to baselines. Maybe the closed-form of the outer function contributes some accurate gradient information to the final gradient in PseuZO. I think the authors could discuss this point.\n\n2.\tIn lines 81-84, the authors state that \"the convergence rate of PseuZO is not explicitly dependent on the parameter size\". However, in the upper bound of the iterations to find an $\\epsilon$-stationary point, the coefficient \"$\\alpha_1$ is the effective dimension\". According to formula (6) and Assumption 1, $\\alpha_1$ may be equal to the parameter size. Thus, I think there exists conflict within the statement here. \n \n3.  This work proposes the sliding window-based PseuZO. The authors consider to store the last hidden state with a much smaller size. This is not reflected in Algorithm 2. Algorithm 2 uses the notations $\\theta$ to denote the parameter and $x$ to denote the training data, which are also not consistent with the problem setting and Algorithm 1. It is necessary to introduce that $||\\cdot||_{op}$ in Assumption 1 means the spectral norm.\n\n4.  The experiments are mainly conducted on tasks in GLUE and SuperGLUE. There lacks some complex tasks like reasoning.\n\nQUESTIONS:\n1. In line 200, it says \"When $\\lambda$ approaches zero, PseuZO gradually degenerates to ZO method and thus we directly transform to MeZO instead\". In my opinion, when $\\lambda$ approaches zero, $u_k$ with large $k$ approaches zero, thus the stored gradient at time $k$ contributes little to the accumulated PseuZO gradient. Thus, the value of $\\lambda$ should only influence the weight in the accumulation step. How does it make PseuZO degenerate to ZO?\n\n2. In line 13 of Algorithm 2, why uses the zeroth-order gradient at time $k$ to multiply the outer gradient at time $t$?\n\n3. Lines 242-246 together with Table 6 are confusing. Does it mean that if using the same time consumed by MeZO to run 10k steps on OPT1.3B, PseuDO could run 9300 steps within 10 epochs, 8700 steps within 20 epochs? Why are the numbers of total steps and the steps within each epoch in the above results different from each other? Please explain this paragraph more detailedly.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a zero-order optimization framework called PseuZO for LLM training, which estimates the gradient with respect to model parameters based on the Jacobian matrix of the model output and the gradient of the loss function with respect to the model output. To reduce the variance, PseuZO applies exponential moving average ($\\lambda>0$ as the decay factor) on Jacobian estimators. Further, the sliding window technique and the hidden state storage strategy are adopted to control the memory usage introduced by the inner product of large-scale vectors in PseuZO. Besides, Periodic changing of $\\lambda$ is needed to accelerate convergence. Experiments on fine-tuning OPT models across multiple tasks and pretraining small vision models on MNIST and CIFAR datasets demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nQuality: The proposed method is technically feasible due to its clear implementation and the provided convergence guarantee.\n\nClarity: In general, the core ideas of this paper are easy to understand, and adequate information is given for readers to reproduce its results.\n\nOriginality: The idea of using Jacobian information to enhancing zero-order optimization is interesting.\n\nWeaknesses:\n\nClarity:\n1) Many spelling inaccuracies exist in this paper: In line 16, “meory” should be “memory”; in line 124, “themq” should be “them”; in line 261, “greaat” should be “great”.\n2) The explanation of function $\\mathcal{F}$ exists in Equation (4) (see line 135), while $\\mathcal{F}$ is cited in line 18 and 70. In line 162, the norm $|| \\cdot ||_{op}$ lacks definition.\n3) Equation (2) and Equation (3) are incorrect.\n\nQuality:\n1) This paper does not strictly verify the benefits of using Jacobian information of the model output in theory. Further, the ablation study (results in Table 6) shows that the better performance achieved by PseuZO is mainly due to the carefully designed scheduling of decay factor $\\lambda$. However, this scheduler can also be used in MeZO and no ablation experiment is given.\n2) The performance of MeZO reported in Table 1,2,3 is too low, making the claimed improvement doubtful (See Table 2 in [1], and Table 3 in [2]). Besides, experimental results on more models (like OPT-13B and LLaMA2-7B) and advanced zero-order optimizers (like SubZero [2] and LOZO [3]) should be given.\n3) The comparison of different optimizers on pretraining vision models shown in Table 7 seems unfair, because node perturbation and local learning methods are not universally adopted across all optimizers.\n\n[1] Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark. In ICML, 2024.\n\n[2] Zeroth-Order Fine-Tuning of LLMs in Random Subspaces. arXiv:2410.08989, 2024.\n\n[3] Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures. In ICLR, 2025.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pseudo zero-order optimization framework, PseuZO, to address the issue that the variance of gradient estimation in conventional zero-order optimization increases linearly with the parameter dimension, which leads to slow convergence in high-dimensional settings. PseuZO decomposes composite objective functions, explicitly estimates the Jacobian matrix of model outputs, and employs exponential moving average (EMA) to reduce variance. In addition, a sliding window technique is used to reduce memory overhead. Theoretical analysis demonstrates that the convergence rate of PseuZO depends on the effective dimension rather than the explicit parameter dimension. Experimental results show that PseuZO outperforms MeZO and MeZO-SVRG on multiple tasks and achieves significantly lower memory consumption compared to traditional first-order methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.\tBy decomposing the gradient of the composite function and leveraging Jacobian estimation of model outputs combined with EMA for variance reduction, the proposed method overcomes the dimensionality bottleneck of traditional zero-order optimization.\n2.\tThe authors prove that the convergence rate depends only on the effective dimension, rather than the parameter dimension, providing theoretical guarantees for high-dimensional optimization.\n3.\tThe superiority of PseuZO is demonstrated on multiple tasks, and its compatibility with PEFT techniques is also shown.\nWeaknesses\n1.\tExperiments are conducted only on the OPT model series. The effectiveness of the method on larger models or other architectures, such as Qwen, LLaMA, or DeepSeek, has not been validated.\n2.\tThe hyperparameter search space in the experiments is relatively limited, which may affect the fairness of the comparisons.\n\nQUESTIONS:\n1．\tIn the experiments, only 10,000 training steps are performed, and MeZO may not have fully converged. Could the authors provide comparisons after sufficient convergence?\n2．\tFor more complex tasks, such as MultiRC or DROP with longer input texts, does the method incur higher wall-clock time? How does model scaling affect the method’s efficiency?\n3．\tCould the authors include experiments on more recent models, such as LLaMA or Qwen series?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PseuZO, a pseudo-zeroth-order optimization algorithm designed to train large language models (LLMs) under constrained memory budgets. The core techniques of the proposed approach can be summarized as follows:\n\nTech 1: PseuZO decomposes the gradient using the chain rule (Eq. 5), where the gradient of the loss with respect to the last hidden state is computed in closed form, and the Jacobian of the last hidden state with respect to the remaining model parameters is approximated in a zeroth-order manner (Eq. 6);\nTech 2: A momentum-based variance reduction technique is employed (Eq. 7);\nTech 3: A sliding-window mechanism is introduced to further reduce memory usage (Section 3.4).\n\nThe authors provide a preliminary convergence analysis of the proposed method and evaluate its empirical performance across a variety of tasks, including classification, multiple-choice question answering, and text generation, under both full-parameter tuning and parameter-efficient fine-tuning (PEFT) settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper is overall well-constructed and clearly written, making it easy to follow.\n2. The empirical evaluation is thorough and demonstrates promising improvements over the MeZO baseline across various tasks and settings.\n\nWeaknesses\n\n1. The proposed approach relies heavily on prior work [1], particularly in its use of gradient decomposition and variance reduction via momentum (Tech 1 and 2 in the summary above), as acknowledged by the authors in Lines 104–106. The primary novel contribution appears to be the sliding-window mechanism for memory reduction. However, this design alone may not be sufficiently innovative to meet the novelty threshold expected at a venue such as NeurIPS.\n2. The source of improvement over MeZO is not clearly identified. From a theoretical standpoint, the convergence analysis in Section 3.3 is preliminary and does not explicitly justify the advantage of the proposed technical components. The convergence rate still depends on the effective dimension and, in the worst case, on the full model dimensionality. From an empirical standpoint, important ablations are missing—specifically, comparisons with (a) MeZO + Momentum and (b) PseuZO without Momentum would help isolate the contributions of the core techniques and clarify where the observed gains originate.\n3. The set of baselines is limited and could be significantly strengthened. Recent work in scalable and memory-efficient zeroth-order optimization has produced several relevant baselines that should be considered for a more comprehensive comparison, such as: (a) Forward Gradient [2], which avoids two-point estimation errors at the cost of slightly more computation; (b) HiZOO-L [3], which introduces additional forward passes and compressed diagonal Hessian storage; and (c) DeepZero [4], which accounts for model sparsity. Given the rapid progress in the field, relying solely on vanilla MeZO and MeZO-SVRG as baselines does not provide a strong benchmark.\n\nTypo:\n\nEq 3: <x, ...> => <\\nabla f(x), ...>\n\n[1] Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks, https://arxiv.org/pdf/2407.12516  \n[2] Gradients without Backpropagation, https://arxiv.org/abs/2202.08587  \n[3] Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer, https://arxiv.org/pdf/2402.15173  \n[4] DEEPZERO: SCALING UP ZEROTH-ORDER OPTIMIZATION FOR DEEP MODEL TRAINING, https://arxiv.org/pdf/2310.02025\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}