SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes
Authors: Yifan Yang
Zhen Zhang
Rupak Vignesh Swaminathan
Jing Liu
Nathan Susanj
Zheng Zhang
Venue: NeurIPS 2025 poster ()
Keywords: vision-language models
prompt-tuning
black-box optimization
zeroth-order optimization
Primary area: deep_learning

Abstract: Fine-tuning vision language models (VLMs) has achieved remarkable performance across various downstream tasks; yet, it requires access to model gradients through  backpropagation (BP), making them unsuitable for memory-constrained, inference-only edge devices. To address this limitation, previous work has explored various BP-free fine-tuning methods. However, these approaches often rely on high-variance evolutionary strategies (ES) or zeroth-order (ZO) optimization, and often fail to achieve satisfactory performance. In this paper, we propose a hybrid Sharpness-aware Zeroth-order optimization (SharpZO) approach, specifically designed to enhance the performance of ZO VLM fine-tuning via a sharpness-aware warm-up training. SharpZO features a two-stage optimization process: a sharpness-aware ES stage that globally explores and smooths the loss landscape to construct a strong initialization, followed by a fine-grained local search via sparse ZO optimization. The entire optimization relies solely on forward passes. Detailed theoretical analysis and extensive experiments on CLIP models demonstrate that SharpZO significantly improves accuracy and convergence speed, achieving up to 7\% average gain over state-of-the-art forward-only methods.

SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes
Yifan Yang', Zhen Zhang', Rupak Vignesh Swaminathan', ,Jing Liu", Nathan Susanj', Zheng Zhang'
University of California, Santa Barbara
AAmazon AGI
(yifanyang, zhen zhang) @ucsh.edu (swarupak, jlmk, nsusanj) @ amazonnccom
zhengzhang@ @cc...cb...eeuuuu
Project Page: https: yang. wet. http//ssarrp..g...oo github.......... 1o/
Abstract
Fine-tuning vision language models (VLMs) has achieved remarkable perfor- mance across various downstream tasks; yet, it requires access to model gradients through backpropagation (BP), making them unsuitable for memory-constraine inference-only edge devices. To address this limitation, previous work has explored various BP-free fine-tuning methods. However, these approaches often rely on varianar evolutionary strategies (ES) or zeroth-orde arothh-oddrr (ZO) optimization, and often fail achieve satisfactory performance. brid Sharpness-awee aware Zeroth- ooder optimization (SharpZO) approach, specifically designed to enhance the performance of 7O VLM fine-tuning via a sharpness- aware warm-up training. SharpZO features two-stage optimization process: a sharpness-aware ES stage that globally explores and smooths the loss landscape construct strong initialization, followed by fine-graines local search via sparse ZO optimization. The entire optimization relies solely on forward passes. Detailed theoretical analysis and extensive experiments CLIP models demonstrate that SharpZO significantly improves accuracy and convergence speed, achieving up to 7% average gain over catte-of-the-ee forward-only methods.
this paper, we propose a hy-
1
Introduction
In recent years, fine-tuning vision- vision-language models (VLMs) has achieved remarkable performance
across wide range of downstream tasks, including image classification [54] [55], object detection
II [2], and image segmentation [45] (24). Among these models, one of the most prominent CLIP
which has attracted significant attention for its powerful zero-shot recognition capabilities. To further improve the performance of VLMs in downstream tasks, previous work has explored the use of efficient, trainable prompt parameters [55]
34
for the prompt tuning of VLMs. However, these prompt-tuning techniques are heavily dependent on the availability of a backward computation engine, which is typically unavailable on memoryy-cosstriieed edge devices used Internet-of-Things (IoT)
40 4
applications [40] or wearable technologies []]
To address these limitations, recent studies have explored fine-tuning VLMs in backpropagation-fre free settings [38] [4 (43). These approaches optimize trainable prompts by leveraging high-variance black-box optimization techniques such as Evolutionary Strategies (ES) [1 and Saroth-Order as alternatives to the first-order (FO) methods used in white-box employ ES to update prompt parameters by evaluating sampled prompts through forward passes only, thereby eliminating the need for memory-expensing back propagation. More recently, ZO stochastic gradient descent (SGD) [4] methods have been adapted to VLM
(7O) optimization 31
32 53
scenarios. For instance, 49
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

D10
Sharpoo FickVP
Test Accuroc (ha
Training Loss.
ZP
Change Point
25 CLP
607
Sharz20
Sharpzo Backvit
5UA
410
dP
01
2503
.48
19:03
2000
AMC
SABC
12031
Training Stops
Training steps
Curve Training Loss and Testing Accuracy EuroSAT
Downstream Generalization
Figure 1: (a) Comparison between Sharp20 and other ZO coompt-tuning baselines. Shanpp2O demon- strates significantly lower variance than other ZO-based baselines like ZIP (32 and BIa [1). (b) Fine-tuned performance across all 11 tasks tested compared with ZIP and BIackVIP and BBT
39
All experiments are conducted using the CLIP model with a ViT-B/16 backbone.
By approximating gradients with just two forward evaluations, these ZO approaches avoid the high computational cost and instability of ES, yet still match its performance while requiring substantially fewer model queries [32].
fine-tuning in the work of BI and ZIP [3 32).
However, existing ZO-based VLM fine-tuning methods remain substantially inferior to packpropagation-basee based training. Their high variance and inherently local search dynamics make them prone to premature convergence. Previous work has attempted to improve the performance of ZO optimization by reducing the problem dimensionality through pruning [I5] 26 [50] and low-rank decomposition [46 32] of the trainable parameters. However, in widely adopted prompt-tuning settings, such parameter reduction offers limited benefit, as the number of trainable parameters is already inherently small in original trainable prompt.
In contrast to prior work that reduces the variance of ZO optimization by limiting the number of trainable parameters, our approach introduces a new perspective that focuses on initialization and the sharpness of the loss landscape. Specifically, Zeroth-Order optimization method that employs a two-stage framework to significantly reduce the variance of ZO gradient estimation and improve the performance of ZO-based VLMs prompt tuning.
propose SharpZO, hybrid Sharpness-Awar
The first stage performs warm-up training using sharpness-awars Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which provides both smoother loss landscape and a strong initial- ization for the second stage. Unlike gradient-based methods that follow local descent directions, CMA-ES enables effective global exploration by adaptively shaping the search distribution based on past evaluations (29). Moreover, incorporating sharpness not only improves model generalization but also improves the accuracy of the randomized gradient estimators used in stage ZO training, which unbiased only with respect to a smoothed version of the objective function 14].
In the second stage, we perform fine-grained local optimization using sparse Zeroth-Orde Stochastic Gradient Descent (ZO-SGD) method. To further reduce gradient estimation variance, we introduce novel Z-pruning technique specifically designed for noisy 7.0 settings, effectively reducing the dimensionality of the search space. Unlike conventional magnitude-
pruning used in previous sparse 70 method [15, 26) Z-pruning leverages gradient information to capture the influence of model non-linearity and applies Z-score-based normalization
12 to suppress outlier gradient estimates.
As shown in Figure D our method converges faster with significantly lower variance compared with other ZO prompt-tuning baselines, achieving up to 7% average improvement accuracy. Our main contributions are summarized as follows:
We propose SharpZO, novel hybrid sharpness- aware optimizer that fine-tunes VLMs using only forward passes. To our knowledge, this is the first ZO method that improves performance considering the sharpness-aware initialization.
In the first stage, we introduce a sharpness-aware CMA-ES that enhances generalization and reduces second stage 70 gradient estimation variance by smoothing the loss landscape.
In the second stage, we develop a sparse ZO fine-tuning method with a novel Z-pruning technique to suppress outliers in noisy gradient estimates.
We validate SharpZO through extensive experiments and theoretical analysis, demonstrating superior performance over existing Br-free baselines.

---

2 Background
2.1 Coordinate-wise Gradient Estimation or Randomized Gradient Estimation?
Mainstream ZO gradient estimation methods can be broadly classified into two categories: Coordinate- wise Gradient Estimation (CGE) and Randomized Gradient Estimation (RGE). In our SharpZO framework, we employ CGE to compute sharpness-related terms in stage and pruning metrics in stage 2, while RGE used to update parameters during the second-stage -tagee ZO-SGD optimization. Below, we provide background on both approaches and highlight their key differences in terms of estimation variance and computational efficiency.
Given a VLM with trainable prompt vector t E IRd we define the training cross-entropy loss as C(w). ZO estimated gradients Vw((w) are estimated via forward differences between function evaluations, where the perturbation of the trainable parameters to depends on whether the CGE or RGE method is used, which gives:
C(w + nu.) - o(w mmm)) u.] (CGE) C(w))
Σ O(w + ue:) [(w uey)
(RGE) v ((w)
())
2x
Here, /4 > is smooth parameter, u, Rdd denotes randomized perturbation vector perturbing all parameters at the same time RGE, and e; = [0, 0, ... 1, I.... 07 represents the i-th standard basis vector, which is used to compute finite-difference approximation of L(w) along single coordinate in CGE. To reduce the variance of the RGE gradients. it is common to average the gradients estimated over g different randomized perturbations, where is called query numbers. In contrast, the CGE method approximates the gradient by perturbing individual coordinates and estimating the directional derivative along each axis using finite differences.
Unlike FO methods that compute exact gradients VL(w) via BP, RGE methods estimate gradients in biased manner toward the exact gradients, which instead provides unbiased estimate of the gradient of a smoothed version of the objective, defined as (, = E., [(w + j.u)). contrast, CGE estimates directional derivatives along individual coordinates without applying such smoothing, resulting in greater sensitivity to sharp changes in the loss landscape.
Difference between RGE and CGE: We compare RGE and CGE primarily in terms of query complexity and accuracy. The number of function queries differs significantly between the two methods. Given the dimension of trainable parameter as d and the number of RGE query g. RGE requires (g) queries, whereas CGE incurs a higher cost of (dd) queries. Clearly, RGE offers much lower query complexity, especially when q < d in our case. Despite its higher query complexity, CGE achieves superior accuracy compared to RGE, as it directly approximates the true gradient without introducing smoothed objective function 5
2.2 Covariance Matrix Adaptation Evolution Strategy (CMA-ES)
Similar as ZO method, CMA-ES is another type derivative-free optimizer for continuous, black-box functions [1]. CMA-ES achieves truly global search by maintaining and adapting a full covariance matrix, which captures variable interactions and shapes an anisotropic search distribution; sampling population each generation then naturally balances broad exploration (through larger step-size) with focused exploitation (via covariance updates). In contrast, zeroth-order methods rely on local gradient approximations at single point and random perturbation directions.
At iteration CMA-ES maintains parameter or, az and Ct, where 0, is the or is the global step size, and C, the covariance matrix. To update these parameter at each iteration, a population of S candidates is drawn as
mean,
wi O, + o. N' Ci),
i=
and their fitness is evaluated via the black-box loss L(w().
After evaluating the population, the parameters are updated in three steps. First, the mean of the distribution 0, is shifted toward regions with lower loss L(w) by taking weighted combination of the top-performing candidates w'. This moves the sampling center toward promising areas in the search space while maintaining stochasticity. Second, the covariance matrix C, is adapted to capture both the overall spread and the correlations among the selected samples. Finally, the global step size
3

---

Original
After Stage
Stage I:
Sharpness- oware GMA-ES ES Giobol expanattenn
Optimization
Emootring Loos landscapa
SharpZO Framework Overview Update Oq. C and sample w"
landscape
Prompt
Sparse Perturbation juz
w(
Rendom Projection
Forward
4 42
2p
Sharpness
N( 8 d.Ce)
Stage S:hhappesss-wware
(d) Stage Sparse optimization
Figure 2: Overview the SharpZO method. (a) The overall training pipeline of SharpZO, consisting of two-stage optimization process. (b) Visualization of the smoothed loss landscape after Stage sharpness-awan CMA-ES optimization. (c) Training dynamics of the sharpness-awes aware CMA-ES method. (d) EGE-based gradient estimation during sparse ZO training in Stage 2.
Tt is adjusted through step-size adaptation mechanism, which regulates the overall exploration scale based on the recent success of the search. For detailed derivations and algorithmic formulations, please refer to [8].
2.3 Sharpness-aware Optimization
Sharpness-Award Minimization (SAM) 13 originally proposed improve generalization by smoothing the loss landscape and encouraging the optimization process to converge to flat minima. The key idea minimize the worst-case loss within neighborhood around the current parameters w by introducing an analytic approximation of the worst-case perturbation e* within radius Specifically, the SAM objective is formulated as:
min CSA (w). W
where
SAM
max C(w + ||
Following the objective function, the gradient approximation for SAM after dropping the second- order terms is given as Vww((w') w'te" by computing the worst-case perturbation e*. To estimate e", SAM approximates the inner maximization using FO Taylor expansion of the loss function. This yields the following analytic solution:
E" RJ arg max
(C(w) + T Selas
Unlike previous works that apply SAM to replace standard gradient descent cither throughout the entire training process [13] during the final few epochs [56]. we, for the first time, investigate the effectiveness of incorporating sharpness information into CMA-ES as an early-stage warm-up strategy enhance the performance of ZO fine-tuning.
3 The SharpZO Method
In this section, we detail the SharpZO method, which is designed to fine-tune VLMs using only forward passes. As illustrated in Fig. (a), our approach consists of main stages: sharpness- aware CMA-ES stage and sparse ZO line-grained search stage. We demonstrate that performing bhare sharpness- aware global search in the early steps of training significantly enhances the performance of ZO optimization both both in terms of convergence speed and final accuracy. In the following, we first summarize the problem setup of this work and then describe each stage our method in detail.
We consider a black-box VLM with loss function L(w) and dataset D = {(rm: yo. ) with a total of N samples and К classes. For models like CLIP, it is necessary to construct a text prompt for each class. Specifically, for a given class E {l..... K} and the model hidden dimension m, the class-specific prompt pa defined by concatenating a predefined initial text embedding Po E R (e.g., "a photo of a") with the class label embedding c², yielding p* [po. c* To make the prompt x trainable, introduce parameter p: that modifies the initial embedding Po. where t E [1,

---

is the training steps. This parameter is obtained via a random projection from a low- did-dmimenionll trainable matrix w. E R where < m is the latent dimension. Then, fixed randomized projection matrix A IR= is used to project ur into the embedding space, producing tü; A E IR to matching the shape of the original initialized prompt Po- Thus, the overall training objective becomes:
N
min N
Σ C((po + Pt, ct), (In. Un)), where p = w T
Next, we provide a detailed introduction to the two stages of the SharpZO method separately.
3.1 Stage 1: Sharpness-Aware CMA-ES Method
In this section, we first summarize the traditional CMA-ES method and then propose our sharpness- aware CMA-ES optimization for a warm-up training.
In the traditional CMA-ES method, given the population size S, the optimizer generates population of candidate solutions wi, i E [1, S). where w is obtained by sampling from the current multivariate normal distribution w 0, + o C,). Here, 0₁ is the weighted mean of the distribution, is the step size, is the population size and C' is the covariance matrix capturing the shape and scale of the search distribution. After evaluating the loss (fitness) L(w,, 'D) of these samples by forwarding the training samples DD along with the trainable prompt wij, E [1, S]. the parameters 0,, Tt, and C are updated accordingly (17).
Different from the previous CMA-ES method, we propose a new sharpness-aware CMA-ES method to smooth the loss landscape during the stage I warm-up training, which help to reduce the stage gradient estimation accuracy. Specifically, we add the worst-case perturbation e" during the sampling of CMA-ES method, where e' is computed within local Euclidean ball based on eq. (2) and gives:
VC
w ~ E + +0, + 80, Co), e [1,
e = = VVL
(3)
The effectiveness of this modification can be explained through the Taylor expansion of the Monte Carlo estimation for the loss E[C(0, + c" o)] used in the sharpness- aware CMA-ES optimizer, given o (0, 57 C.). Given the sampling strategy eq. (3) and omitting the first-order term by the fact EE [o] O, the expected fitness can be approximated as:
B[C +5 o)] C(O,+ + Ee)) 1B T
V2 (0C(0++ ) o~ NO, C.).)
As observed, in addition to optimizing the same term L(O, +e") as gradient-bases SAM methods, the effectiveness of the sharpness-aware CMA-ES approach is achieved with additional higher order term involving stochastic adaptation of the covariance matrix, which introduces an additional mechanism to explore and down-weigh, high-curvature directions.
Another challenge in applying sharpness- CMA-ES lies in gradient estimation. Specifically, we cannot directly access the gradient information needed to compute e" due to the absence of backpropagation. Unlike prior work [51] 48) which relies RGE-based gradient estimation with a large query budget 4, we adopt CGE for this purpose. CGE provides an unbiased estimate of the gradient along each coordinate, making it more suitable for computing the sharpness-aware perturbation term, as discussed in Section 2.1
The detailed formulation for estimating V V(ww) using
CGE is provided in eq.
3.2 Stage 2: Fine-grained ZO Method
After obtaining a strong initialization and a smoothed loss landscape through the first-stage global search, we proceed to optimize the parameters toward the global optimum using ZO-SGD method. In this stage, we perform fine-grained and efficient local search guided by RGE-estimated gradients. To further reduce the variance, propose new Z-pruning metries with pruning mask Ω to reduce the effective dimension during gradient estimation.
Unlike prior sparse 70 methods that rely solely on magnitude-based pruning metrics, we introduce novel pruning criterion tailored to the high-variance nature of 70-estimated gradients. In contrast to first-order sparse training approaches, which apply the pruning mask Q directly to model weights,
5

---

we upply the mask to the perturbation vector u. This strategy effectively reduces the dimensionality during ZO perturbation and results in the following gradient estimator:
(ww(w) L(w + Qu)) [(w μ Srr)
24
We note that query number of q 1. as defined in Eq. ①. is sufficient for stable training of our SharpZO method when preceded by the Stage 1 warm-up. This is significantly more efficient than prior works such as ZIP [32] and BLACK VIP 31]. which still exhibit high training variance even with query number of q = 5, shown in Fig. Next, we introduce how we construct the Z-pruning based pruning mask 92.
Z-pruning Metrics:
The Z-pruning metric is designed to minimize the loss degradation introduced by pruning, by considering the sensitivity of each parameter. Specifically, considering ow reflect the change of weights during pruning, the difference in loss between the dense and pruned models can be approximated via first-order Taylor expansion:
1 oow' Hčw + O(1löwl))
C(w + 6w) [(w)= v((w)
8w + [[[w)]Tww))
2
where H denotes the Hessian of the loss with respect to the parameters. Here, we can approximate the Hessian H using a Fisher matrix as H = (V):: and estimating gradients V ((w; 2) using the CGE method described in eq. as VL(w;;)) v). By considering the second-order term, we can obtain the Z-pruning metrics as:
[(w + uei) -(ww pe;) es (5)
= 110 z =(Bn> V VLw,, I),, where VL(w) = Σ
2/1
where =(g²) -g2-Na)/on denote the Z-score normalization given Mg and og as the mean and standard deviation of the gradient vector g VL. The Z-score standardizes the gradient magnitudes to mitigate the scale mismatch between trainable parameters and their gradients-a mismatch that is especially pronounced in ZO settings. We consider adapting the second-order term as the pruning score based on the practice of previous pruning work [37 [7]. In Section [.4.2] we demonstrate the effectiveness of the Z- Z-pruning method compared to dense and magnitude- based pruned ZO training.
3.3 Algorithms
We present the SharpZO algorithm in Algorithm I In practice, we first execute Stage I for a total of T. steps, followed by Stage 2 until the total training budget of '' steps is reached. The transition point T. determined automatically using strategy inspired by early stopping, based on the observed change in validation accuracy. Typically, T. < 'I' and is typically reached within 100 steps. As result, although Stage is more computationally expensive per step, the overall training efficiency remains high. During Stage 2, we update the pruning mask every K steps to balance computational cost and adaptability to the evolving optimization landscape. The learning rate for ZO optimization in Stage 2 is denoted by 17.
4 Theoretical Guarantee
In this section, give quick proof to show the convergence rate of SharpZO method. By comparing the SharpZO convergence rate with the baseline ZO-SGD rate given in MeZO (28), we highlight why our hybird smoothness-awanl setup significantly help improve the performance of VLMM fine- tuning. To align our analysis with VL.Ms/LLMM fine-tuning, we consider a non-convex optimization setup and the proof assume the loss landscape follows the Polyak-Lojasiewiez (PL) inequality, which has been widely considered in other ZO fine-tuning papers (28). First, we list the following assumptions for our analysis include the PL-inequality we just mentioned:
A1 (PL Inequality): The loss function l satisfies the there exists a constant y >> 0 such that for all w Rdd we have [[] μ (L(w) - L*), where L denotes the global minimum of the loss function.
(PL) condition. That is,
A2 (I (Lipschitz smoothness): The loss function L has L-Lipschitz continuous gradient. That is, there exists a constant I. 0 such that for all uns un E Rd, we have [[V[(wx VL(w,)) S L w. w. ||.

---

Algorithm SharpZ00: Hybrid Sharpness-Aware Zeroth-Order Optimization
Require: Initial prompt parameters wo, total steps T, transition step Te, pruning interval
I: for 1= to T do
2: if < To then
Stage I:
Sharpness-aware CMA-ES
3:
Sample candidate solutions wi using eq.
Evaluate fitness [wwi) each candidate
5:
Update CMA-ES parameters θ,, ds. and C₁ based on fitness values
6:
else
D Stage 2: Sparse ZO Optimization
7:
= or (t Te) mod K 0 then
8:
Update pruning mask Q using eq.
9:
end if
10: Estimate gradient ((w)) using the ZO oracle (eq. ut++1 +- W) - ((m))
12:
end if
13:
end for return Fine-tuned prompt vector wr
14:
Theorem 1. Under assumptions Al and A2, suppose the SharpZO algoritim first performs T, steps of global optimization using CMA-ES and then switches to ceroth-order gradient-bor based optimization until convergence. The convergence rate of SharpZ0 method can be give by:
( 221og log Tp
t TT++O +0
(6)
where is given by ussuming ((we) L- S Se, 11 the learning rate of ZO-SGD optimizer in stage
and I is the smoothness factor. Eq. (0) is obtained by ignoring the lower order terms for clarity.
Proof. Details of the proof can be found in Appendix D
Compared to the naive ZO-SGD convergence rate presented in 28 the SharpZO method leverages sharpness-aware initialization strategy that yields lower starting point for the second-stage ZO training, specifically of the form (1 μu11 2 = Au in eq. 13 in Appendix D Since we use a relative large step size during the first stage training, we can observe the original error gap As is linearly decreasing with high scaling factor. Moreover, as we can observe from the sharpness term in eq. (6), the integration of sharpness-aware optimization effectively clips the L-smoothness constant with the sharpness parameter p, thereby reducing the effective sharpness in the 70 training phase.
5 Experiments
In this section, we present experimental results to evaluate the performance of the proposed SharpZO method across variety of downstream tasks using CLIP models with different architectures. Specifi- cally, we compare the proposed SharpZO method with zero-shot (ZS) inference and other BP-free baselines like BBT [49]. BIackVIP 31], and ZIP 32 (Detailed descriptions for tasks and baslines method can be found in Appendix B). Our results demonstrate that SharpZO not only achieves (ToTA)
superior accuracy but also improves efficiency, as measured by the time-to-test-acene metric 7 Additionally, we provide a comprehensive ablation study to analyze the contributions of individual components in Section 1.4.1 and Section 5.4.2 Further implementation details and extended experimental results-includdig
evaluations across various model architectures and hyper- parameter choices, as well comparisons with state-of-the-an prompt-tuning methods that involve backpropagation, such as CraFT [43] are provided in Appendix
Training Detail: For the VLM model, utilize CLIP with both ResNet [19 and ViT [10] backbones as the visual encoder, and Transformers 41 as the text encoder. The CLIP weights are initialized from the official pretrained checkpoints and remain frozen during training. The prompt generator use initial prompt with length 4, and hidden dimension 512. Parameters in w are initialized from Gaussian distribution (0, 0.02).
Here, we manually tuned the change points from stage to stage by choosing from a set of parameter between 100 to 500. However, we also tried to adapt an early stopping criterion to automatically
7

---

Table 1: Few-shot performance across 11 datasets using CLIP models with ResNet and ViT backbones,
trained for 20K steps. indicate results reported in prior works [31 43 B2]. We additionally reproduce
the ZIP results, as the original paper restricted the query budget to 5K. Bold values highlight the best
performance, demonstrating the superiority of Sharp7.0 over all BP-free baselines.
Backbome
Methods
Image Vet
Pets
Flo
FGVC
DTD
Euro
Cars
Food
SUN
Cal
UCF
AVG
ZS-CLIP
58.18
85.77
66.14
17.28
42.32
37.56
55.61
77.31
58.52
86
61.46
38.77
BBT
61.74
88.73
72.53
12.07
54,33
69.01
60.24
78.44
64.34
90.05
67.91
65.40
RN50
BlackVIP
60.33
85.99
65.
17.37
42.73
58.
56.70
77.23
59.17
86.37
60.11
60.84
7IP
61.30
89.53
68.41
19.98
47.40
63.10
58.61
78.98
62.86
90.63
64.05
64.08
SharpZO
63.29
89.51
79.50
23.97
60.58
80.77
60.58
79.28
66.17
91.24
72.43
69.76
ZS-CLIP*
66.73
89.21
71.34
24.72
44,39
47.60
65.32
86.06
62.50
92.94
66.75
65.23
BBT
70.15
92.70
82.41
29.49
59.26
70.48
70.19
86,42
70.33
94.75
70.48
72.42
VIT-B/16
BIackVIP
67.10
89.70
70.60
24.78
45.20
73.10
65.60
86.60
64.70
93.70
69.10
68.20
ZIP (Officall*
66.20
94.00
70.40
26.80
47.80
64.60
71 11
86,40
63.30
94.00
69.80
70.57
7IP (Rep)
68.35
93.18
73.00
28.32
$4.26
74.19
67.5 58
87.01
67.43
94.97
72.51
70.98
SharpZ0
71.60
94.06
88.02
32.34
63.95
79.42
72.50
87.13
70.86
95.09
77.08
75.64
Table 2: Comparison of robustness to distribution shift between SharpZ0 and other baselines. The best results among BP-free methods are highlighted in bold.
ResNet-50
VIT-B/16
Method
ImageNet
.V2
-Sketch
-A
Avg
ImageNet
.V2
-Sketch
Avg
ZS-CLIP
58.2
51.3
33.3
21.7
56.0
40.6
66.7
60.8
46.2
41.8
74.0
57.2
CoOp
63.3
55.4
34.7
23.1
56.6
42.4
71.7
64.6
47.9
49.9
75.1
59.4
BRT
61.7
54.0
33.9
23.2
58.3
42.4
70.2
63.0
47.9
49.5
76.1
59.1
BlackVIP
60.2
52.3
33.3
21.5
57.7
41.2
65.5
59.2
44.6
42.5
73.1
54,9
ZIP
61.3
53.7
33.7
23.9
57.6
42.2
68.4
59.7
45.5
47.1
75.2
56.9
SharpZo
63.3
54.8
35.2
24.5
58.7
43.3
71.6
63.8
45.0
50.3
76.6
58.9
decide this point: the algorithm switches to stage if the validation accuracy does not improve by more than 0.01 over the best recorded accuracy for 10 consecutive steps, which can achieve similar results. Detail cyyper-paramene setup for SharpZO method on various tasks can be found in Table. All experiments use a 16-shot setup unless otherwise specified.
in Appendix C.2
5.1 Results on Few-Shot Classification
We first compare our SharpZO method with SOTA BP-free prompt-tuning baselines across 11 downstream tasks. To explore the effect of different model architectures, we evaluate all methods using CLIP models with both ResNet-50 and ViT-B/16 viusal encoder backbones. The results are summarized in Table Based on these results, we draw the following conclusions:
Sh significantly outperforms all other BP-free methods. As shown in Table SharpZO con- sistently surpasses other ZO prompt tuning approaches in terms of classification accuracy. Compared to the SOTA ZO prompt tuning method ZIP, SharpZO achieves an absolute average performance gain of 5% and outperforms ZIP among all tasks on the CLIP model with ViT-B/16 backbone. The performance of SharpZO is approaching first-order method like CoOp, which shows the potential of deploying ZO method in real-world application. These improvements are driven by the reduction of gradient estimation variance and bias with the sharpness-awand aware warm-up training.
SharpZO performs robustly across diverse model architectures. Unlike prior ZO prompt-tuning methods such as ZIP and SeAkVIP-whice often struggle to converge on certain tasks like Flow- 10s AT,OS and UCF101 when using CLIP models with ResNet backbones SharpZO method consistently delivers strong performance across a wide range of architectures and tasks. Additional results using architectures such as ResNet- 101 and ViT-B/32, presented in Appendix C.I. further demonstrate the robustness of SharpZO varying model backbones.
proposed
SharpZO exhibits lower training variance. As illustrated by the optimization curves in Fig. [a), SharpZO achieves markedly more stable training-int standard-deviation bands are substantially narrower than those of other ZO methods such ZIP and Black
5.2 Robustness to Distribution Shift
In this section, we further evaluate the robustness of the SharpZO method under distribution shifts. Results comparing SharpZO to other BP-free basclines summarized in Table Compared to the

---

Figure 3: Comparison between the naive CMA- ES with the Sharpness-awan (S-aware) CMA- ES method on EuroSAT dataset.
Figure 4: Comparison between the naive ZO optimization and sparse ZO optimization with various pruning metrics on EuroSAT dataset.
(a) influence Stage
(b) Influenca
00.0
80
80
Test Accuracy 196)
Test Accuracy 199)
75
PC
70
70.0
Nageitada Pruntng
2-scere Pruting)
Denie
S-pwore CM6-15
EMALE5
CM4-5
00.0
200
10000 15000
1400
Training steps
Training steps
Training
ofate-of-the-art ZO method ZIP, SharpZO achieves an absolute improvement 2.0% on ResNet- 50 and 2.8% on ViT-B/16 (averaged over all distribution shift benchmarks) for the ImageNet test accuracy. These findings highlight the strong out-of-distributio generalization ability of Sharp7.0 under varying types of distribution shifts.
5.3 Time-to-test-acccuacy Efficiency
In this section, we evaluate the training efficiency of the proposed Sharp70 method. We focus on training time rather than memory usage, as all zeroth-order (ZO) baselines exhibit comparable memory con- sumption due to their forward-only nature. Specif- ically, we measure the wall-clock time required to reach a common evaluation accuracy threshold, fol- lowing the protocol of [7]. The threshold is selected such that attainable by all baselines. The re- sults are summarized in Table 3 and tested single Nvidia A 100-40G GPU.
Methods
IN
Pets
DTD
Euro
BIIckVIP
172.6
714.8
132.0
201.5
ZIP
19.0
126.3
6.0
251.4
SharpZ0
15.3
2.4
2.6
12.7
Table Time-to-test accuracy comparison be- tween different BP-free prompt tuning meth- ods on multiple dataset. The time is recorded in minutes.
As shown in Table the SharpZO method achieves faster convergence compared to other ZO prompt tuning baselines, which is consistent with our theoretical analysis Section Beyond the benefits of the proposed hybrid sharpness-awart optimization scheme, the improved training speed of SharpZO also stems from its lower per-step query count and significantly faster forward pass.
Specifically, unlike ZIP and BlackVIP, which require 10 queries per step to reduce training loss, SharpZO only requires 2 queries per step during Stage 2. Moreover, ZIP incurs substantial overhead due to its complex reconstruction process in forward pass, taking approximately 0.53 seconds per forward pass, whereas Sharp70 requires only 0.0069 seconds. Consequently, the average per-step training time of SharpZO is markedly lower than other 7O baseline like ZIP.
5.4 Ablation on Components in Different Stages
5.4.1
Influence of Stage Sharpness Aware Optimization
In this section, we aim to validate the effectiveness and illustrate the influence of our sharpness-aware CMA-ES method to both stage and stage training in SharpZO. Specifically, we compare the training curve between the naive CMA-ES method and sharpness-aware (S-aware) CMA-ES method for both stage in Fig. (a) and Fig. 3 (b), respectively. The experiments are conducted on the EuroSAT dataset using the CLIP model with ViT-B/16 backbone. For the convenience of comparison, the transition point from Stage 1 to Stage 2 optimization fixed at 500 steps.
As illustrated in Fig. 3 (a), the sharpness-awar CMA-ES method consistently achieves faster convergence rate and superior final accuracy compared to the naive CMA-ES method stage 1, which shows better generalization ability. More importantly, the sharpness-awan training benefits the second-stage stage convergence of 70 optimization as observed from Fig. 3 The sharpness-aware warm-up training leads to more stable Stage 2 training curve with reduced variance, which can be

---

attributed to the implicitly clipped smoothness factor introduced by the Sharpness-aware updates, as discussed in the theoretical analysis in Section
5.4.2 Effectiveness of Stage Sparse ZO Optimization
We evaluate the effectiveness of our proposed Z-pruning strategy for sparse ZO optimization in Stage Specifically, we compare it against magniude- baaeed pruning and dense training on the EuroSAT dataset using the CLIP ViT-B/16 backbone. Results are shown in Fig.
Our findings show that sparse ZO training with Z-pruning reduces gradient variance and improves both accuracy and convergence speed compared to dense training. In contrast, magnitude-based croming- used in prior work [15 perffrrss poorly in prompt-tuning due to the limited number of trainable parameters (512). which makes accurate pruning critical. Moreover, magnitude-basee pruning operates solely on weight values, ignoring critical nonlinear interactions within the model. This limitation is particularly impactful in prompt-based tuning, where the prompts are prepended the input and play a disproportionately large role the model's behavior compared to standard weights.
5.4.3 Comparison with Method Involving Backpropagation
In this section, we compare our method with black-box tuning baselines that involve backpropagation, such as CraFT 43 The CraFT method introduces collaborative fine-tuning framework that jointly optimizes both the prompt and the adapter, using ES for the former and first-order (FO) methods for the latter. Although CraFT achieves strong performance, its requirement of backpropagation limits its environments, such as mobile devices and edge devices, where gradient access via backpropagation is not available.
applicability memory-constraine
is important highlight that reliance on backpropagation presents significant challenges for deployment on edge inference devices, as these devices are typically equipped with inference-only ASIC's and do not support gradient computation. Unlike large-scale multimodal models like LLaVA [25], the CLIP model is particularly relevant for edge computing scenarios. Despite the inherent limitations of backpropagation basidd methods in such contexts, we include CraFT our comparison to demonstrate that our proposed SharpZO method can achieve even better performance without relying on backpropagation, and in a more efficient manner.
The comparison results on few-shot task accuracy and training memory, are summarized in Table 4 As shown, SharpZO consistently outperforms CraFT across all evaluated tasks. Moreover, SharpZO also requires less training memory, as it avoids storing backpropagation graphs, further enhancing its suitability for edge deployment.
Table 4: Comparison between SharpZO and black- box fine-tuning baseline involving backpropagation. * represent accuracy results obtained from original CraFT paper 43.. We bold the best results during the compasion.
Test Accuracy
Memory (MB)
Methods
Imagenet
Pets
DTD
Euro
Imagenet
Pets
DTD
Euro
CraFT*
68.21
91.94
63.28
72.07
3297.5
3130.9
3128.7
2178.8
SharpZ0
71.60
93.46
63.95
79.42
3132.3
3032.5
3057.9
2075.2
Conclusion
This paper has introduced SharpZO, a hybrid ZO fine-tuning method comprising two optimization stages. In Stage 1, SharpZO employs sharpness- aware CMA-ES algorithm to conduct global search for optimal regions while simultaneously smoothing the loss landscape. In Stage 2, SharpZO performs fine-grained sparse ZO optimization for local optimization. Compared with prior BP-free fine-tuning approaches, Sharp70 provides a segh-performanc, inference-only fine-tuning solution tailored for VLMss. Future work may explore the extension of SharpZO to full-model fine-tuning for both text- and multimodal LLMs.
10

---

References
[1] Youhei Akimoto, Yuichi Nagata, Isao Ono, and Shigenobu Kobayashi. Bidirectional relation between ema evolution strategies and natural evolution strategies. In Parallel Problem Solving from Nature, PPSN XI: 11th International Conference, Kraków, Poland, September 1 11-15, 2010, Proceedings, Part I 11, pages 154-163. Springer, 2010.
[2] Anne Auger and Nikolaus Hansen. Tutorial cma-es: evolution strategies and covariance matrix adaptation. In Proceedings of the 14th annual conference companion on Genetic and evolutionary computation, pages 827-848, 2012.
[3] Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness- awape minimization improves language model generalization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 7360-7371, 2022.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- food-101-an 101-mining discriminative components with random forests. In Computer vision-ECCY 2014: 13th European conference, zurich, Switzerland, September 6- 12, 2014, proceedings, part VI 13, pages 446 461. Springer, 2014.
[5] Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng I Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. arXiv preprint arXiv:2310.02025,, 2023.
[6] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[7] Cody Coleman, Daniel Kang. Deepak Narayanan, Luigi Nardi, Tian Zhao, Jian Zhang, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. Analysis of dawnbench, time-to- accuracy machine learning performance benchmark. ACM SIGÓPS Operating Systems Review, 53(1):14-25, 2019.
[8] Erika Covi, Elisa Donati, Xiangpeng Liang, David Kappel, Hadi Heidari, Melika Payvand, and Wei Wang. Adaptive extreme edge computing for wearable devices. Frontiers in Neuroscience, 15:611300, 2021.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255 Icce, 2009.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogmition at scale. In International Conference on Learning Representations (ICLR), 2021.
[11] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vasion-language model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14084-14093, 2022.
[12] Nanyi Fei, Yizhao Gao, Zhiwu Lu, and Tao Xiang. Z-score normalization, hubness, and few- shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 142-151, 2021.
[13] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
[14] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23(4):2341-255 -2668, 2013.
[15] Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, et al. Zeroth-order fine-tuning of llms with extreme sparsity. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization ( (WANT@ ICML 2024), 2024.
11

---

[16] Nikolaus Hansen and Anne Auger. Evolution strategies and cma-es (covariance matrix adapta- tion). In Proceedings of the Companion Publication of the 2014 Annual Conference on Genetic and Evolutionary Computation, GECCO Comp '14, page 513-534, New York, NY, USA, 2014. Association for Computing Machinery.
[17] Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (ema-es). Evolutionary computation, I 111) 2003.
[18) Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolu- tion strategies. Evolutionary computation, 9(2):159-195, 2001.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition (CVPR), pages 770-778, 2016.
[20] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[21] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages $340-8349, 2021.
[22] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages
2021.
[23] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[24] Zihan Li, Yunxiang Li, Qingde Li, Puyang Wang, Dazhou Guo, Le Lu, Dakai Jin, You Zhang, and Qingqi Hong. Lvit: language meets vision transformer in medical image segmentation. IEEE transactions on medical imaging, 43(1):96 107, 2023.
[25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances neural information processing systems,
2023.
[26] Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsich, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order Ilm fine-tuning. arXiv preprint arXiv:2402. 15751, 2024.
[27] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv: 136.5555 2013.
[28] Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038-53075, 2023.
[29] Gregory Morse and Kenneth O Stanley. Simple evolutionary optimization can rival stochas- tic gradient descent in neural networks. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pages 477-484, 2016
[30] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEF, 2008
[31] Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
12

---

[32] Seonghwan Park, Jachyeon Jeong, Yongjun Kim, Jaeho Lee, and Namhoon Lee. ZIP: An efficient zeroth-order prompt tuning for black-box vision-language models. In The Thirteenth International Conference on Learning Representations, 2025.
[33] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[34] Alec Radford, Jong Wook Kim, Christopher Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference Machine Learning (ICML), 2021.
[35] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[36] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11):1-7, 2012.
[37] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[38] Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuan-Jing Huang, and Xipeng Qiu. Bbtv2: Towards gradient-free free future with large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3916-3930, 2022.
[39] Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language- Language-moeel-ag -oode--as-a-service In International Conference on Machine Learning, pages 20841-20855 PMLR, 2022.
[40] Nazli Tekin, Ahmet Aris, Abbas Acar, Selcuk Uluagac, and Vehbi Cagri Gungor. A review of on-device machine learning for iot: An energy perspective. Ad Hoc Networks, 153:103349, 2024.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2017.
[42] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in neural information processing systems, 32, 2019.
[43] Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Connecting the dots: Collaborative fine-tuning for black-box vision-language models. In International Conference on Machine Learning, pages 50931-50943, PMLR, 2024.
[44] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485- 3492.. IEEE, 2010.
[45] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained rision-language model. In European Conference on Computer Vision, pages 736-753 Springer, 2022.
[46] Yifan Yang, Kai Zhen, Ershad Banijamali, Athanasios Mouchtaris, and Zheng Zhang. Adazeta: Adaptive zeroth-order rensor-train adaption for memory-efficienst large language models fine- tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 977-995, 2024.
[47] Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M. Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, Nathan Susanj, Zheng Zhang, Jack FizzGerald, and Abhishek Kumar. Wanda++: Pruning large language models via regional gradients. In ICLR workshop in Sparsity in LLMs (SLLM), 2025.
13

---

[48] Feiyang Ye, Yueming Lyu, Xuehao Wang, Masashi Sugiyama, Yu Zhang, and Ivor Tsang. Sharpness-awer aware black-box optimization. The Thirteenth International Conference on Learning Representations, 2025.
[49] Lang Yu, Qin Chen, Jiaju Lin, and Liang He. Black-box prompt tuning for vision-language model as service. In IJCAI, pages 1686-1694, 2023.
[50] Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunz- mann, and Zheng Zhang. Mazo: Masked zeroth-order optimization for multi-task fine-tuning of large language models. arXiv preprint arXiv:2502. 1513, 2025.
[51] Huaqin Zhao, Jiaxi Li, Yi Pan, Shizhe Liang, Xiaofeng Yang, Wei Liu, Xiang Li, Fei Dou, Tianming Liu, and Jin Lu. Helene: Hessian layer-wise clipping and gradient annealing for accelerating fine-tuning Ilm with zeroth-order optimization. arXiv preprint arXiv:2411. 2024.
[52] Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, BG Vijay Kumar, Anastasis Stathopoulos, Manmohan Chandraker, and Dimitris N Metaxas. Exploiting unlabeled data with vision and language models for object detection. In European conference on computer vision, pages 159-175. Springer, 2022.
[53] Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, and Zheng Zhang. Quzo: Quantized zeroth-order fine-tuning for large language models. arXiv preprint arXiv:2502. 12346, 2025.
[54] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-land languae models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816-16825, 2022.
[55] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009):2337-2348, 2022.
[56] Zhanpeng Zhou, Mingze Wang. Yuchen Mao, Bingrui Li, and Junchi Yan. Sharpness- minimization efficiently selects flatter minima late in training. The Thirteenth International Conference Learning Representations, 2025.
14

---

A Limitations
While SharpZO demonstrates strong empirical and theoretical advantages for forward-only VLM fine-tuning, several limitations remain. First, the method currently tailored for prompt-tuning scenarios with relatively low-dimensional parameter spaces: its scalability to full-model or multi- modal fine-tuning remains unexplored. Second, the sharpness-aware CMA-ES warm-up stage requires coordinate-wise gradient estimation (CGE), which may be computationally expensive for higher-dimensional settings. a result, the SharpZ.0 method proposed in this work is a better fit. under the parameter-efficient fine-tuning setup.
B Further Detail Regarding Tasks and Baselines
Datasets: Following the experimental setup of prior VLMSs fine-tuning works 43 32
we evaluate
Sharp70 on 11 diverse image classification benchmarks under a few-shot learning scenario. These datasets cover a broad range of tasks: generic object recognition with ImageNet 9 and Caltech 101
42). fine-grained image classification with OxfordPets [33]. StanfordCars [23] Flowers 102 [30], Food101 4) and FGVCAircraft (27]. satellite image classification with EuroSAT [20], texture recognition with DTD (6, scene classification with SUN397 E4 and action recognition with UCF101 (3). To assess the robustness of SharpZO under distribution shift, further evaluate it on four widely-used Sketch 42, ImageNet-A [2). and ImageNet-R (2n)
(OOD) variants of ImageNet: ImageNetV2 335, ImageNet-
Baselines: To benchmark the performance of Sharp70 against SOTA methods, we mianly consider five baseline approaches:
Zero-shot (ZS): This baseline manually crafted prompts to directly evaluate the pretrained
CLIP model without any additional adaptation.
BBT 49 BBT employs naive CMA-ES-based optimizer to update the trainable prompt parame- ters. As the original BBT is designed for LLMs, we adopt its prompt generator structure and adapt it to the VLM fine-tuning setting.
Black B1: BlackVIP uses naive ZO-RGE estimator to jointly optimize both textual and visual prompts in a black-box manner.
7IP 32:: ZIP improves upon naive 70 prompt tuning by reducing the number of trainable
parameters via low-rank decomposition of the prompt space.
CraFT 4 : CraFT introduces trainable adapter appended to the output of the CLIP model. It
jointly optimizes both the prompt parameters and the adapter using a combination of CMA-ES and gradient-based methods. As CraFT requires access to backpropagation, provide separate comparison with in Appendix
Additional Experimental Results
C.1 Robustness across different model architectures
We further evaluate the performance of SharpZO across different model architectures and compare it with other baselines, with results summarized in Table 5 As shown, SharpZO demonstrates architecture-agnosicc effectiveness, consistently outperforming previous backpropagation-free (BP- free) methods across all four evaluated architectures. In particular, SharpZO achieves an average absolute performance improvement of 2.25% and 4.19 % over the ZIP and BIackVIP methods, respectively.
Table 5: Ablation study for model architectures with Imagenet dataset.
Methods
RN50
RN101
Vit-B/16
Vit-B/32
Avg.
7S-CLIP
58.18
61.62
66.73
62.05
62.15
BlackVIP
60.33
62.00
67.10
61.10
62.63
ZIP
61.30
63.67
68.35
64.97
64.57
SharpZO
63.29
65.40
71.60
66.98
66.82
15

---

C.2 Hyper- Search
To guide future applications of the SharpZO method, we conduct ablation studies on several key prpee-parameters, including the scaling factor for the sharpness term, and the sparsity ratio used in sparse ZO optimization. The results are summarized in T
Based on our experiments, the optimal scaling factor for the sharpness term should be around 0.05 or 0.1, which is consistent with the choice in the original SAM paper for the SAM-SGD algorithm. The sparsity ratio should be larger than 0.5. Conversely, a sparsity ratio that is too small will hurt the representational capacity of the prompt parameters, since the number of training parameters is already low enough during prompt tuning. This observation differs from previous ZO full-model fine-tuning work.
Table 6: Ablation studies for different applicable pruning metrics and sparsity,
MethodsiSpariity 10%
30%
50%
70%
90%
Magnitude
78.07
77.14
76.12
77.06
77.09
Z-Score 78.91
79.02
79.42
79.15
79.21
Table 7: Ablation studies for different scaling factor p.
0.001
0.01
0.05
0.1
0.5
Accuracy
77.74
79.03
79.22
79.42
76.32
Another hyperparameter that influences performance is the scaling factor p. which adjusts the weight of the sharpness-ave aware term during the sampling process in Stage optimization. We conduct an ablation study with different values of p. as shown in Table
We outline the configuration details for each comparative baselines. Specifically, the hyper-parametc setup of individual tasks for Sharp70 method are presented Table [ To search these hyper- parameters, we select 3-5 candidate values (00.1, 0.2, 0.4, 1.0] for CMA ES step size, [le - 3, le 4, 5e for ZO scale and [le- 1, le - 3, 1e-5] for ZO learning rate) and choose the one yielding the best performance with the other hyper-ypparrre parameters fixed. All on a 5-shot validation set extracted from the official validation set or splitted from the training set (e.g. ImageNet). For fair comparison,
search are performed
the original hyperparameter settings provided in the when running the experiments. In contrast, for the ablation studies, we adopt consistent parameter setup across methods to ensure comparability. Specifically, like to note that during the stage 2 of our method, we set the query number as instead of 5 used in previous baselines like ZIP and BIackVIP, which is enough for the convergence of SharpZO method.
baseline papers 32 31 43
would
Table 8: Hyper-parameter setup for Stage I and Stage 2 in the SharpZO paper.
Method
Dutaset
Pets
Flo
FGVC
DTD
Euro
Cars
Food
SUN
Cal
UCF
IN
Step size a
0.1
0.4
0.2
0.4
0.4
1.0
0.1
0.4
0.1
1.0
70-CGE scale Rage
1e3
16.3
1e33
le-5
1o:3
Ie.3
le.3
1o-3
1633
1e.3
1e-5
Implicit population
Sharp70
Intrinsic dimension
$12
(Stage
Context tokers number m
Scaling factor
Population size S
Change point (step)
100
500
400
300
500
100
100
200
200
200
Learning rale 1)
le.3
1e:3
16.3
16.3
1e.3
1e.3
1e.3
1e-1
16.3
1e.3
le-1
ZO-CGE Hege
SharPZO (Stage
RGE scale
le.3
Pruning interval
Number of query
Pruning
D Proofs
To prove Theorem I we begin by establishing Lemma which characterizes the convergence of the first-stage sharpness-aware CMA-ES method by leveraging its interpretation as a natural-gradient descent algorithm [[]. We then apply the result of Lemma I ass the initial condition for analyzing the convergence the second-stage 70 optimization. Finally, by composing these two phases, we obtain the overall convergence rate of the SharpZO algorithm.
16

---

D.1 Proof of Lemmal
Before presenting the Lemma we first introduce some background knowledge regarding the con- nection between the CMA-ES update and the natural gradient descent, details about the mathematical relationship can be refereed to [6].
Considering the minimization
arg min B[[(P) Oc. Ci] O,C,
under the sampling distribution of eq. (3) natural-gradient descent step with step-size reads
Oc+1 0. Fo Vo. (o.B([(P) Oc. Ci],
C++1 C. Vc, E(C(P) O2, Ct].
(8)
where F is the Fisher information matrix Fo, - -pp-(( distribution defined in eq.
T log (P))))) 0e 80
given the sample
3 Here we have used that the natural gradient with respect 0g satisfies:
Vo. [[[(P)) C, Vo. E[C(P)).
To simplify the subsequent convergence proof, we note that our target is the expected fitness ((m) of the sample center mi 0r, whereas C, affects only the sampling spread and not directly the objective value. In the idcalized infinite-samples regime of CMA-ES one shows
c [²L(m) = H,
so that C₁ implements a Hessian-inverse preconditioner. Consequently, in our proof we focus solely on the mean update (7 (with C, x H) and omit carrying the detailed covariance dynamics through the convergence bounds.
Based eq. (3) we sample P3 based both the current mean value of the distribution parameter O, and the sharpness aware term E" obtained by optimizing the maximize problem max |ellaSS L(P within the nearly region around the current parameter Thus, by simplifying the second and higher order term with the variance of 2, we can obtain the gradient of expectation for the loss (., + 6" + =)] as:
V E[0 + + +2)] V. [(O, + + 0
Putting the above equation into eq. 77. can obtain the natural gradient updating equation for the stage of our SharpZO method, gives:
Oc+1 0, o F Vo, [[C(P O,, C.]
02 oF (V + + (
Here, inspired by the proof of Therorem 4. of original SAM paper 3 we divide the natural gradient
step of our sharpness-awes aware CMA-ES method into two steps:
VL(O,)
04+ 0,
(V.
0,41 0, oF (V (6F))
(10)
Lemma (Per-step eтror bound for sharpness-aware CMA-ES). Under Assumptions A2, the sharpness-aware CMA-ES method gives a per-step error bound for the updating process as:
C(0,41) I ((O,) [ 2T. V + T? T??p²
+ T.02 p²
Proof. We begin our proof for the first stage defined eq. @.. By the assumption of L-smoothness,
we have:
L(A 1) 1 = (())) + (VL())), 2+
0,) + 1 2+ -0,12..
(+
Since 0+ - -:
C(A 1+7
L(0.) + ((V L0²
(1)
17

---

For the second stage in eq. F (v)(()) which gives:
( we employ the similar idea and denote natural gradient as VlL(6,) ~
[(0++1) S ((0,) - (V)())) VC 1)) (+))) ())+))+++ "0" VC
1)(
(+2
( 2 ((.))) It [ + (())) V V2 1 1)
+ Loc
VC 1)|
4+2
2
1+2
S ((0)) [ + -22 + L0¹ V2 .t1'1**
< ((0.) - [ + + Lo VV (P' ((0,)] + 0L00 p²
S _(.)) -
+ O² p²
where (a) is given by the fact (a. 6) 2 |a/12 Žl1a b112 with VL(O.), b= VC(0 1) and (b) is given by eq.
++2
D.2 Proof of Theorem
Now, we begin the proof of the global convergence rate of SharpZO method. Before we start, we first prove a per-step error bound for the stage 2 sparse 70 training in Lemma Then, we perform inductive step based on the per-step error bound of the stage and include the results in Lemma] [ as an initialization point of stage training. Different from the proof in previous ZO fine-tuning paper the optimization problem, we consider the
[27 that consider an "effective' rank for the dimension true dimension d, the trainable parameter in our prompt tuning case is much lower than the full model fine-tuning case. The proof Lemma is given as follows:
Lemma 2 (Per-step Error Bound for 7O-SGD). Given the Assumption A2 and the 70-RGE gradient estimation follow eq. (, by setting the learning rate 1J S 2L(d+43. we have:
L² (d+ (d+
B[C(w)+1) me] S L(w.)
where d the true parameter dimension of the trainable prompt ur and L is the smoothness factor, I is the ZO perturbation seal. The standard devation of the stochastic gradient estimation "is is defined as 7 = [B[]] V ((w.)) fyp (mn))))) given the unbiased estimator 4 L(w:)) for the smoothed objective function (w))
Proof. Let w, be the parameter at iteration i and we consider ZO-SGD using gaussian smoothing estimator defined eq. (4). Based on properties of L, Theorem 3.1 (c) of []. the variance of estimator satisfies:
4 L ( +)3
E [(:))()) < 2(d [(w))) + +
where is smoothness factor T. is assumed
Given the learning rate > 0, then from the smoothness of L, the standard descent lemma gives:
B[[(w++1) w:] S L(w,) V.(w:))) 11 + -"
1 " (L(w))1
n L2 + 6)9)
< C(w,) n| [[(
2 (2(d 41[[
Rearranging:
L²
B[[(w++1) w.) < L(w))
Lnydd + (o7
(d + 6)3).
4
Choose 7 S शा( that Lny(d + 4) 1.2 Then:
B(L(w++1) wi] S [(w)) " ("" I22²
(d + 6)3).
18

---

Next, we proceed to prove Theorem by performing an inductive argument based on the result of Lemma 2
Let the total number of steps Stage be denoted as T2 == T Te- To facilitate the analysis, we define two suboptimality gap measures:
A (1) ((0:) L" which denotes the optimality gap of the distributional mean 0₂ used
in Stage (Sharpness-awarc CMA-ES);
A (2)
((w)) L used in Stage 2 (ZO optimization).
guarantees that the distributional mean Or, satisfies convergence bound on A7 Using a second-order Taylor expansion of the loss function around Or.- we can relate the Stage initialization gap (2)) to A Pa
At the transition point To, Lemma |
via:
To
(1) ( -(pr.)))) - -- - T + VL(OT)I + (0² Tr(OT.))
(11)
This inequality provides the initial condition for the inductive proof in Stage 2. where we now track the evolution of A!?) for Te. ..... T, as governed by Lemma Here, first bound T. By Lemma for each we have
(1)
((0++1) S ((0,) - - | + +30'p0
Subtracting L" from both sides yields
A411 S A. } - (1-) (VL( +C, C: 7.
Under the PL inequality [V((0)]] 2 2/r A,, it follows that
- - { 2Lo²) (2 +C
= [1 - 1-111 - 2Loo A +C.
Set
$ = p (1-
0 0< (assuming 0² < 1/(2L)).
Then the recursion becomes
4 S (1 (1- (1 CC
Unrolling this for t 0, 1...., To 1 gives
A 11) - 7T= A (1)
+ C Σ
T.-1
E)- (1 (1 ( + ] 1 (1 (1-
C
1=0
Substituting back C = -12pp² + 1'002p²
and & = 1u(1 - 2Lo²) completes the proof:
< (1 - 5) Te 0 + 1.2p² 1.302p² p²
1 - -1-
(12)
Te
p (1 - 2Lo2)
Now, we begin to prove the global convergence rate for the SharpZO method. Let's focus back into the bound given in Lemma By the PL-inequality assumed in Assumption A1, we have:
E [((w++)
w
< [(w,) 3 (l Crar L) r?
< L(w,) пи (L(w,) L*) + Cvar (d, L) 17²
where Cvar (d, T.) Md+4))+ e2p² (d+ is some constant. Taking full expectation and with 4,2)
E[[(w:)] - L* gives the one-step contraction
A
(2) ++ < (1 nu) A
Cvar (d, L)
19

---

Given the current step we define 12 == -t- Te and unroll this linear recursion for = Tes.... To +t2 yields
() S (1 n A (d, Ld (1 (1- (1 (1-n1)²*)
1
(a)
S (1 (1-mu)* + PlI 11+ L (p2+
Cvarld, L) (1- (1 mppi)²)
+ oo. Tr(Cr.))) +
(6)
+ L2p + 1²m1²0² [1 1 -1)
S (1 mn)² ((1 -) (1)
(1 21182) pll VV )1 + L (p² + 8 Ir(Cr:))) +
Gvar(d,L)))
(1 (1-(1=)))*
where (a) is given by eq. 11 and (b) follows the bound of A
(1) proved in eq. 12
Finally, by
ensuring A have:
tu =0
In B
77
where
(1- A000 + p2pp + 133ppp2
(1 (1- + +
(p² 202 + 87. Cr:).
L
&
2
(13)
L Tr Cr.
= (1 1000 + 20((1 + 1S - 2) + 2 L) + +1l V((r.))
(14)
2
and
Cvar (d, L)
Ovar (d, L)= L(d+4)) ddp (d+6)*
B
(15)
Here, if we focus the influence of smoothness factor L and ignoring the lower-order terms for convenience, we can write the convergence rate l2 as:
L"
(2(L) ~ 0 log
(16)
1/
20

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes]. , [No] or [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
the checklist
of the factors in their evaluation.
"[No] " or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist", Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
21

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the
implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only lested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We prodive detailed description for the assumption and proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
We included the detiled hyper- paree--rrretter setup in the appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways.
For example, if the contribution is novel architecture, describing the architecture fully
22

---

might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release the code, the anonymous code is provided in the supplemental material with documentation
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
mublic/guidese hhbpll///giidessoooeehunmssssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
claipss.cc/ublic///c hpppuuiiiguuudesss..eeppppe Codeeoooiiisseeeee ssssonoiiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and basclines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
23

---

Justification: We use public dataset and we have offered detailed instruction on the hyperpa- rameter of the data processing.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them. The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In the training curve we provided, we mark the variance interval by shallow
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma eтror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eтrоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide wall data and memory information the description.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwniiaeee/pppp///atciiaa
Answer: [Yes]
24

---

Justification:
We have checked.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
25

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the related papers.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pittasess.papessiiiii paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provided such details
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
26

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (attps://neurips htt.a/rrrpp..c///.oeeeeeee////ss for what should or should not be described.
27


Reviews:
[{"review_id": "loN71IZqeW", "reviewer": "Reviewer_VzLh", "summary": "This paper proposes a two-stage zero-order prefix tuning method for the CLIP. The first stage utilizes sharpness-aware CMA-ES for creating a smooth loss landscape and then the second stage performs the major zero-order optimization.", "questions": "Please address my concerns in the above session.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- The SharpZO method is clearly introduced and easy to follow. Moreover, the SharpZO method shows a significant performance advantage than other zero-shot optimization methods in both accuracy and speed.\n- Page 2 Line 38: Why would a high variance lead to suboptimal convergence? I assume a high variance would lead to the failure of convergence.\n- The official baseline result for ZIP in Table 1 seems wrong. For Flowers102, the reported baseline in the ZIP paper is 70.4%, but in this paper it is 92.3%.\n- In the Table 2, SharpZO does not perform the best on ImageNet-Sketch, thus shouldn't be highlighted in bold.\n- The hyper-parameter selection in Table 7 is not detailed, *i.e.*, is the parameter search performed on the test set?\n- The configuration for few-shot learning is not included.\n- Authors did **not** include any information about how they evaluate the statistical significance of their results (*e.g.*, the number of random seeds adopted). Only the ablation experiments shows the variance of results. For all other experimental results that involves a comparison with other baseline methods, the statistical significance required in the NeurIPS checklist is not justified. As zero-shot optimization methods are highly stochastic, a sufficient repeated evaluations with different seeds are necessary.\n- For CMA-ES in the Figure 3, there is a huge accuracy drop and variance gap between the end of the Figure 3(a) and the beginning of the Figure 3(b). However, for the S-aware CMA-ES, there is no such difference. I would like to know what leads to such a difference.\n- For comparison with gradient based method, prefix tuning should also be included justifying the need for zero-order optimization.", "was_revised": true, "final_justification": "Authors have properly addressed all my concerns. Indeed, their method shows much better results than other zero-order optimization methods. Though the accuracy is not as good as gradient-based methods, their method has significantly less resource consumption. Thus, I would like to raise my rating towards accept. Thank you.", "rebuttal": "Thank you for your thoughtful and detailed feedback. We greatly appreciate your recognition of the strengths of our work, as well as your constructive comments regarding its weaknesses. Below, we address each of the points raised in your review.\n\n## Weaknesses\n\n> Page 2 Line 38: Why would a high variance lead to suboptimal convergence? I assume a high variance would lead to the failure of convergence.\n> \n\nThanks for pointing this out. We agree that high variance can slow down or misdirect updates. The issue of convergence to suboptimal regions in line 38 is more related to the limitations of local search dynamics. We will revise the sentence to clarify this distinction in the final version.\n\n> The official baseline result for ZIP in Table 1 seems wrong. For Flowers102, the reported baseline in the ZIP paper is 70.4%, but in this paper it is 92.3%. In the Table 2, SharpZO does not perform the best on ImageNet-Sketch, thus shouldn't be highlighted in bold.\n> \n\nApologies for these typos—these errors likely occurred during table reformatting. The right number for the ZIP method should be 70.4%. Since it’s an error during table reformatting, the average accuracy is not influenced. We have corrected these errors\n\n> The hyper-parameter selection in Table 7 is not detailed, *i.e.*, is the parameter search performed on the test set?\n> \n\nYes, we tune hyper-parameters based on test set accuracy. For each hyper-parameters, we select 3–5 candidate values ([0.1, 0.2, 0.4, 1.0] for CMA_ES step size, [1e-3,1e-4, 1e-5] for ZO scale and [1e-1, 1e-3, 1e-5] for ZO learning rate) and choose the one yielding the best performance with the other hyper-parameters fixed. We will include these details in the revised version.\n\n> The configuration for few-shot learning is not included.\n> \n\nThanks for pointing this out. We use a 16-shot setup in all experiments, consistent with prior work such as ZIP, to ensure a fair comparison. We will add this information in the revised paper.\n\n> Authors did **not** include any information about how they evaluate the statistical significance of their results (*e.g.*, the number of random seeds adopted). Only the ablation experiments shows the variance of results. For all other experimental results that involves a comparison with other baseline methods, the statistical significance required in the NeurIPS checklist is not justified. As zero-shot optimization methods are highly stochastic, a sufficient repeated evaluations with different seeds are necessary.\n> \n\nWe agree that further repeated evaluations with different seeds are necessary. In addition to the standard deviation shown in the Fig. 1, 3, 4 in the paper, we conducted experiments using four different seeds and report the mean and standard deviation, as shown in the following table. Due to time constraints, we performed this evaluation on the four datasets below, but we will include standard deviation results for all tasks in the final version.\n\n|  | EuroSAT | DTD | OxfordPets | ImageNet |\n| --- | --- | --- | --- | --- |\n| SharpZO(mean+std) | 80.77±1.53 | 60.58±0.86 | 89.51±1.14 | 63.29±0.65 |\n\nAs we can observe, the overall randomness is limited to a standard deviation around 1 in our fine-tuning case. This degree of randomness is similar to the results reported in previous ZO LLM fine-tuning paper (”Fine-Tuning Language Models with Just forward Passes”, Table 18)). \n\n> For CMA-ES in the Figure 3, there is a huge accuracy drop and variance gap between the end of the Figure 3(a) and the beginning of the Figure 3(b). However, for the S-aware CMA-ES, there is no such difference. I would like to know what leads to such a difference.\n> \n\nThis is an interesting point. In short, the different behavior should be attributed to the different sampling strategy used in S-aware CMA-ES (refer to Eq. (3)), which includes a gradient-related term in the first stage of training. In contrast, it is expected to observe some disturbance in the training curve when switching optimizers with the naive CMA-ES method, as the search dynamics of the two optimizers are fundamentally different: CMA-ES learns a distribution, whereas ZO relies on locally estimated gradients. This change also helps improve the overall accuracy of the SharpZO method.\n\n> For comparison with gradient based method, prefix tuning should also be included justifying the need for zero-order optimization.\n> \n\nThanks for the suggestion. It’s possible that you were referring to first-order prompt tuning (as the original prefix tuning is applied to the attention mechanism, which is not a good fit for CLIP model with possible ResNet Backbone), which has been studied in the CoOp paper [1]. Below, we provide an additional comparison between SharpZO and CoOp.\n\n|  | EuroSAT | DTD | OxfordPets | ImageNet | AVG |\n| --- | --- | --- | --- | --- | --- |\n| CoOp | 84.05 ± 1.05 | 63.26 ± 0.22 | 87.06 ± 0.88 | 63.00 ± 0.18 | 73.48 ± 0.39 |\n| SharpZO | 80.77±1.53 | 60.58±0.86 | 89.51±1.14 | 63.29±0.65 | 69.76  |\n\nWe can observe the our BP-free SharpZO performs on-par or better than the BP-based prompt tuning method CoOp on dataset like OxfordPets and ImageNet. We also agree there is still a gap between SharpZO and CoOp on other datasets. This is limited by the stochastic nature of zero-order fine-tuning, which in return, bring in the advantage of very low memory cost and inference-only fine-tuning availability. \n\n[1] Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. IJCV, 130(9):2337– 2348, 2022b."}, {"review_id": "IZIO7nBg0y", "reviewer": "Reviewer_yzjx", "summary": "This paper proposes a novel backpropagation-free fine-tuning method for VLM's prompt optimization. Their method is based on a 2-stage hybrid optimization process between an evolutionary strategy, CMA-ES and zero-order (ZO) optimization. They introduce interesting and effectiveness changes to both stages that result in a better performance over the SOTA. Extensive experiments and empirical validations provide strong evidence for the effectiveness of their method.", "questions": "1. When estimating $\\epsilon*$, using CGE, do you use $w_i$ for the estimation? Which would mean that $\\epsilon*$ actually also depends on $i$. \n2. While SharpZO only requires 2 queries per step in Stage 2, it would seem like Stage 1 requires a significant number of forward passes and if stage 1 were not fixed to 500 steps, how would the ToTA change? especially in comparison with the other methods. \n\nMinor: \n- line 154, is $E[z]$ meant to be $E[o]$?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. This paper is well-written (despite a few minor typos) that has a clear and easy to follow structure. Each part of their method, theory and experiments were explained comprehensively with sufficient background, making it smooth for the reader to follow. \n2. Besides the novelty in combining 2 different optimization strategies, ES and ZO, the authors also propose new and effective changes to each stage, further strengthening their method. (Though, I'm not as familiar with other ZO methods, so I defer the final judgement on its novelty to other reviewers and this simply remains my opinion. )\n3. There is some theoretical guarantee for the convergence of their method, providing sufficient theoretical justification for their performance gains.\n4. Extensive experimental results and ablations are convincing of the effectiveness of SharpZO. \n\nWeaknesses: \n1. In providing the theoretical guarantee in section 4, it would be more clear to state the convergence rate of the baseline ZO-SGD as well, this is for the reader to easily understand and believe that your convergence rate is faster. \n2. It would be good to provide the average results in Table 2, since you do reference that on average, you increase by 2/2.8% over the SOTA. \n3. In lines 272-274, you seem to be referencing the results in section 5.4.1. On the first read through the paper, it might seem out of place since the reader does not know that you will be comparing naive CMA-ES methods to your sharpness aware version. Moving it to a section below 5.4.1 or referencing that you will compare them in a later section would improve the reader experience.", "was_revised": true, "final_justification": "My initial assessment was already quite positive and the authors have addressed my questions and concerns thoroughly during the rebuttal. I think this is a well-written paper that will contribute greatly to the community and should be accepted.", "rebuttal": "Thank you for your thoughtful and detailed feedback. We greatly appreciate your recognition of the strengths of our work, as well as your constructive comments regarding its weaknesses. Below, we address each of the points raised in your review.\n\n### Weaknesses\n\n> In providing the theoretical guarantee in section 4, it would be more clear to state the convergence rate of the baseline ZO-SGD as well, this is for the reader to easily understand and believe that your convergence rate is faster.\n> \n\nThis is a great point. Unlike the traditional way of presenting convergence rates, we highlight the highest-order term of the smoothness factor $L$ to emphasize its influence on zeroth-order optimization. Compared to the baseline ZO-SGD convergence rate $t(L) \\approx \\mathcal{O}(\\frac{L^3\\mu^2}{\\epsilon})$, our analysis reveals that the smoothness factor $L$ is further reduced by an additional factor (less than 1) related to the first-stage optimization. Since the theoretical number of iterations required for convergence is proportional to the smoothness of the loss landscape, our method demonstrates a faster convergence rate compared to baseline ZO-SGD. We will further discuss these points in the revised version\n\n> It would be good to provide the average results in Table 2, since you do reference that on average, you increase by 2/2.8% over the SOTA.\n> \n\nThanks for the suggestion. We will include the averaged results to better reflect the overall improvement compared to previous methods.\n\n> In lines 272-274, you seem to be referencing the results in section 5.4.1. On the first read through the paper, it might seem out of place since the reader does not know that you will be comparing naive CMA-ES methods to your sharpness aware version. Moving it to a section below 5.4.1 or referencing that you will compare them in a later section would improve the reader experience.\n> \n\nThanks for the suggestion. We will move this part of the discussion to Section 5.4.1 to improve the reader's experience.\n\n### Questions\n\n> When estimating **\\epsilon^**using CGE, do you use **w_i** for the estimation? Which would mean that **\\epsilon^**actually also depends on **i**.\n> \n\nSorry for the confusion here. The computation of the perturbation $\\epsilon$ should be completed before the CMA-ES sampling at each step, which means the value of trainable matrix $w_t$ should be the same as the latest mean value $\\theta_t$. Thus, a better way to present the right part in eq. 3 should be $\\epsilon^*=\\rho\\frac{\\nabla\\mathcal{L}(w_t)}{\\|\\nabla\\mathcal{L}(w_t)\\|_2}$, where $w_t=\\theta_t$. So, this computation process don’t depend on $i$. We will make this point clear in the revised version.\n\n> While SharpZO only requires 2 queries per step in Stage 2, it would seem like Stage 1 requires a significant number of forward passes, and if Stage 1 were not fixed to 500 steps, how would the ToTA change? Especially in comparison with the other methods.\n> \n\nWe agree stage 1 requires a significant number of forward passes. If we increase the number of Stage 1 step, the ToTA will inevitably exceed that of other methods in some point. \n\nHowever, this point ( ~1800 steps) is far beyond 500 steps based on our experiments. Here, we provide an example on EuroSAT task. In our setup, the total number of training steps is fixed at 20,000 for all methods. For SharpZO, Stage 1 training takes approximately 5 seconds per step, while Stage 2 training is significantly faster at 0.15 seconds per step. In comparison, ZIP and BlackVIP require roughly 5 times more queries than SharpZO’s Stage 2, resulting in about 0.6 seconds per step, based on our empirical measurements. To ensure SharpZO run slower than ZIP, the switch from Stage 1 to Stage 2 must occur **later than around 1800 steps**.\n\nAlso, we would like to clarify that across all 11 datasets we tested, Stage 1 consistently converges within 500 steps. The actual number of change point for some of the tasks determined by the system are listed as below:\n\n|  | ImageNet | caltech101 | eurosat | food101 | stanford_cars |\n| --- | --- | --- | --- | --- | --- |\n| Change Point | 125 | 105 | 145 | 80 | 65 |\n\n### **Minor:**\n\n> Line 154, is **\\mathbb{E}[z]** meant to be **\\mathbb{E}[o]**?\n> \n\nThanks for point this typo out. We will correct it in the revised version."}, {"review_id": "3iicTyu8Kj", "reviewer": "Reviewer_qycS", "summary": "This paper proposes an effective and efficient backpropagation-free (BP-free) prompt tuning method named SharpZO. The method consists of two stages. The first stage employs a sharpness-aware evolutionary strategy that combines coordinate-wise gradient estimation with sharpness-aware optimization to construct a smoothed initialization. The second stage performs fine-grained local optimization via sparse zeroth-order (ZO) optimization using randomized gradient estimation. The authors also provide a theoretical convergence analysis to justify the superiority of the proposed method. Extensive experiments demonstrate that SharpZO significantly outperforms existing BP-free methods in both accuracy and training efficiency, as measured by wall-clock time.", "questions": "Please refer to Weaknesses.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow. As someone not familiar with BP-free tuning methods, I appreciate that the paper is self-contained and provides sufficient background to understand the key concepts.\n\n2. The proposed method is elegant and intuitively convincing, which makes me believe it is likely to work well in practice. In addition, the authors provide a theoretical convergence analysis to further justify its effectiveness.\n\n3. The empirical improvements are significant in both accuracy and training time, which makes me believe this work has the potential to be influential.\n\nWeaknesses:\nI do not find any major weaknesses in the paper. Rather, I have a few questions or suggestions that I would like the authors to response:\n\n1. As someone not from the field of BP-free optimization, I would appreciate a brief introduction to the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to provide necessary background. (e.g. why we cannot directly optimize the prompt embedding \\bar{p}?).\n\n2. In line 143, $C$ should be typeset in boldface.\n\n3. The proposed method is not inherently limited to prompt tuning and could be applied to other tuning frameworks such as linear probing. I would be interested to see how SharpZO performs when combined with linear probing.\n\n4. While SharpZO is a BP-free method and understandably compared with other BP-free baselines, its strong performance warrants additional context (I noticed that its performance is significantly improved). I suggest including results from BP-based methods such as CoOp [1] and ProGrad [2] as reference points to better contextualize the remaining performance gap with BP-based methods. And whether add objective of [2] can further improve the performance of SharpZO?\n\n[1] Learning to Prompt for Vision-Language Models, IJCV 2022.\n\n[2] Prompt-aligned gradient for prompt tuning, ICCV 2023.\n\nTo summarize, this is a solid paper with many valuable insights. I am open to raising my score if the authors adequately address the questions and suggestions above.", "was_revised": true, "final_justification": "All my concerns have been addressed", "rebuttal": "Thank you for your thoughtful and detailed feedback. We greatly appreciate your recognition of the strengths of our work, as well as your constructive comments regarding its weaknesses. Below, we address each of the points raised in your review.\n\n### Weaknesses\n\n> As someone not from the field of BP-free optimization, I would appreciate a brief introduction to the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to provide necessary background. (e.g. why we cannot directly optimize the prompt embedding \\bar{p}?).\n> \n\nThank you for the suggestions. In one sentence, CMA-ES treats the trainable parameters as a multivariate normal distribution and updates its parameters (mean and covariance) to search for global optima. We will add a brief introduction to CMA-ES in the Background section of the revised paper.\n\nWe update the prompt embeddings in a low-dimensional space to reduce the variance in both Stage 1 and Stage 2 training, as both CMA-ES and ZO optimization exhibit dimension-dependent convergence rates. The effectiveness of reducing variance via low-rank projection has been demonstrated in prior work [1][2].\n\n[1] ZIP: An efficient zeroth-order prompt tuning for black-box vision-language models\n\n[2] Adazeta: Adaptive zeroth-order tensor-train adaption for memory-efficient large language models fine-tuning\n\n> In line 143, should be typeset in boldface.\n> \n\nThanks for reporting this. We will correct in the final version.\n\n> The proposed method is not inherently limited to prompt tuning and could be applied to other tuning frameworks such as linear probing. I would be interested to see how SharpZO performs when combined with linear probing.\n> \n\nThis is an interesting point. To explore this, we attach a classifier on top of the CLIP model to do linear probing with SharpZO (SharpZO-LP) with ZO-SGD (ZO-SGD-LP) and a first-order method using Adam (Adam-LP). The results are summarized as follows:\n\n| Vit-B/16 | EuroSAT | DTD | OxfordPets |\n| --- | --- | --- | --- |\n| Adam-LP | 57.51 | 59.69 | 87.98 |\n| ZO-SGD-LP | 25.67 | 31.98 | 66.51 |\n| SharpZO-LP | 35.55 | 39.77 | 84.98 |\n| SharpZO |  79.42 | 63.95 | 94.06 |\n\nAs we can observe, the linear probing is more sensitive to the optimization noise with the shallow and unregularized classifier, compared with prompt tuning. Even though the SharpZO can improve the performance compared with ZO-SGD, there is still a large gap between Adam method.\n\n> While SharpZO is a BP-free method and understandably compared with other BP-free baselines, its strong performance warrants additional context (I noticed that its performance is significantly improved). I suggest including results from BP-based methods such as CoOp [1] and ProGrad [2] as reference points to better contextualize the remaining performance gap with BP-based methods. And whether add objective of [2] can further improve the performance of SharpZO?\n> \n\nThanks for the suggestion. We agree with your point and will include the BP-based method results in the revised version. We summarize the key comparison results here and will expand the comparison in Table 1 in the revised version:\n\n| RN50 | EuroSAT | DTD | OxfordPets | ImageNet | AVG |\n| --- | --- | --- | --- | --- | --- |\n| CooP | 84.05 | 63.26 | 87.06  | 63.00  | 73.48  |\n| ProGrad | 83.74  | 63.97  | 89.00  | 63.45  | 74.28  |\n| SharpZO | 80.77 | 60.58 | 89.51 | 63.29 | 69.76  |\n\nWe can observe the our BP-free SharpZO performs on-par or better than the BP-based prompt tuning method CoOP on dataset like OxfordPets and ImageNet. We also agree there is still a gap between SharpZO and CoOp on other datasets. This is limited by the stochastic nature of zero-order fine-tuning, which in return, bring in the advantage of very low memory cost and inference-only fine-tuning availability. \n\nHowever, Regarding the combination of the objective proposed in [2] with SharpZO, we believe it may not be appropriate. In SharpZO, we employ ZO-RGE to estimate gradients by assuming a fixed update direction randomly generated at each step and adjusting the magnitude to reflect the quality of that direction. In contrast, the objective in [2] requires computing the angle between two gradients. Since our approach involves randomly generated directions, calculating such angles does not provide meaningful information. Therefore, we believe the training objective in [2] is not well-aligned with the principles of our method and will not provide additional benefit."}, {"review_id": "X1zuZzxkLD", "reviewer": "Reviewer_ZKiu", "summary": "This paper proposes SharpZO, a hybrid sharpness-aware zeroth-order (ZO) optimization method designed for fine-tuning CLIP, using only forward passes. The work is motivated by the limitations of BP-free tuning in resource-constrained environments.  This approach is entirely forward-pass based, making it attractive for inference-only settings. The authors support their method with theoretical convergence analysis and extensive empirical validation across 11 datasets, showing strong performance improvements over previous BP-free methods.", "questions": "1. How were the training and test datasets split? Did the authors follow the same protocol as CoOp?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper provides a solid theoretical analysis that offers valuable insights into the convergence behavior of SharpZO.\n\n2. The experimental evaluation is thorough and well-executed, featuring comparisons with strong baselines, detailed ablation studies, robustness tests under distribution shifts, and efficiency analysis.\n\nWeaknesses:\n\n1. The transition point between training stages is determined using validation accuracy. However, the paper does not clarify how validation accuracy is computed—especially for datasets that lack an official validation split, such as ImageNet. Furthermore, evaluating validation accuracy during training could introduce significant overhead, potentially making the method less suitable for the low-resource settings that this paper aims to support.\n\n2. Although the method is motivated as a general solution for VLMs, the experiments are conducted solely on CLIP. This raises concerns about the generalizability of the proposed approach. The authors are encouraged to evaluate SharpZO on other vision-language models such as LLaVA or SigLIP.\n\n3. The paper focuses primarily on few-shot learning. However, in real-world edge device scenarios, models are often expected to handle distribution shifts across disjoint class sets. A base-to-new generalization setting would better reflect such conditions. Including results in this setting would enhance the practical relevance of the proposed method.", "was_revised": true, "final_justification": "Although I am concerned about the use of a validation set in the few-shot setting, which I believe contradicts the premise of limited labeled data in this setting, the technical contributions of this paper seems sufficient to me. Therefore, I maintain my positive score of 4.", "rebuttal": "Thank you for your thoughtful and detailed feedback. We greatly appreciate your recognition of the strengths of our work, as well as your constructive comments regarding its weaknesses. Below, we address each of the points raised in your review.\n\n### Weaknesses\n\n> The transition point between training stages is determined using validation accuracy. However, the paper does not clarify how validation accuracy is computed—especially for datasets that lack an official validation split, such as ImageNet. Furthermore, evaluating validation accuracy during training could introduce significant overhead, potentially making the method less suitable for the low-resource settings that this paper aims to support.\n> \n\nFor datasets like ImageNet, we manually select validation samples from the training set. In our paper, we adopt a 5-shot per class sampling strategy to construct the validation set. We will add this detail in the revised version.\n\nTo ensure that validation evaluation does not become a bottleneck, we use a simple trick: since the prompt weights only affect the text encoder in the CLIP model, we pre-compute and cache all image features for the validation set. This avoids repeated computation of the vision backbone, which is typically the most time-consuming part of inference.\n\nWith this trick, a single validation accuracy evaluation only takes **0.009 seconds** taking EuroSAT dataset as an example. Even with 50 evaluations throughout training, the overhead is minimal and does not significantly affect overall efficiency.\n\n> Although the method is motivated as a general solution for VLMs, the experiments are conducted solely on CLIP. This raises concerns about the generalizability of the proposed approach. The authors are encouraged to evaluate SharpZO on other vision-language models such as LLaVA or SigLIP.\n> \n\nThank you for the suggestion. We are happy to provide additional experimental results using the SigLIP model to demonstrate the generalizability of our method. The experiments follow the same setup as in the ZIP paper to ensure a fair comparison within the limited timeframe of the rebuttal. As observed, the SharpZO method achieves better performance than the ZIP method on the SigLIP model, highlighting its strong generalization ability.\n\n|  | EuroSAT | DTD | OxfordPets |\n| --- | --- | --- | --- |\n| ZIP* | 48.7 | 49.4 | 82.0 |\n| SharpZO | 53.4 | 56.5 | 84.6 |\n\n*result obtained from ZIP paper\n\n> The paper focuses primarily on few-shot learning. However, in real-world edge device scenarios, models are often expected to handle distribution shifts across disjoint class sets. A base-to-new generalization setting would better reflect such conditions. Including results in this setting would enhance the practical relevance of the proposed method.\n> \n\nThis is a great point. Following the implementation of the Table 3 of ZIP paper, we train the model on base class and evaluate them on the new classes.  We show some key results here and will expand the comparison in the revised paper:\n\n|  | ImageNet | EuroSAT | DTD | OxfordPets |\n| --- | --- | --- | --- | --- |\n| ZIP* |  65.6 | 64.4 |  51.0 | 97.0 |\n| SharpZO | 67.1 | 70.4 | 53.5 | 95.2 |\n\n*result obtained from ZIP paper\n\nAs we can observe, in the base-to-new generalization setting, our SharpZO method still demonstrates a clear advantage compared to SOTA ZO fine-tuning methods in most of the tasks.\n\n### Questions\n\n> How were the training and test datasets split? Did the authors follow the same protocol as CoOp?\n> \n\nYes, we follow the same protocol as CoOp for loading dataset."}]

Author rebuttal:

Meta-review: The paper introduces SharpZO, a hybrid sharpness-aware zero-order (ZO) optimization method for vision-language model prompt tuning using only forward passes. It combines sharpness-aware CMA-ES for initialization with sparse ZO optimization, supported by theoretical justification and empirical validation. Strengths include clear methodological contributions, strong empirical performance gains over prior BP-free methods, reduced resource consumption, and demonstrated efficiency advantages. Weaknesses identified by reviewers concerned limited discussion of statistical significance, reliance on validation sets in few-shot settings, absence of comparisons with gradient-based baselines in the initial submission, and questions about generalizability beyond CLIP. However, during rebuttal, the authors provided clarifications, corrected table errors, added statistical evaluations, extended comparisons with gradient-based methods, and reported additional experiments (e.g., SigLIP, base-to-new generalization), which resolved most concerns. The discussion confirmed that the method achieves both accuracy and efficiency benefits while maintaining practical applicability. Given the sound methodology, thorough rebuttal, and potential impact on BP-free fine-tuning, the TC recommends accepting this paper.
Decision: Accept (poster)