Collaborative Discrete-Continuous Black-Box Prompt Learning for Language Models
Authors: Hualin Zhang
Haozhen Zhang
Zhekai Liu
Bin Gu
Yi Chang
Venue: ICLR 2025 Poster (2025)
Keywords: Black-box prompt learning
discrete optimization
Primary area: optimization

Abstract: Large Scale Pre-Trained Language Models (PTMs) have demonstrated unprecedented capabilities across diverse natural language processing tasks. 
Adapting such models to downstream tasks is computationally intensive  and time-consuming, particularly in black-box scenarios common in Language-Model-as-a-Service (LMaaS) environments, where model parameters and gradients are inaccessible. Recently, black-box prompt learning using zeroth-order gradients has emerged as a promising approach to address these challenges by optimizing learnable continuous prompts in embedding spaces, starting with \textit{randomly initialized discrete text prompts}.  However, its reliance on randomly initialized discrete prompts limits adaptability to diverse downstream tasks or models. To address this limitation,
this paper introduces ZO-PoG, a novel framework that optimizes prompts through a collaborative approach, combining Policy Gradient optimization for initial discrete text prompts and Zeroth-Order optimization for continuous prompts in embedding space. By optimizing collaboratively between discrete and continuous prompts, ZO-PoG maximizes adaptability to downstream tasks, achieving superior results without direct access to the model’s internal structures.
Importantly, we establish the sub-linear convergence of ZO-PoG under mild assumptions.
The experiments on different datasets demonstrate significant improvements in various tasks compared to the baselines.

Published as a conference paper at ICLR 2025
COLLABORATIVE DISCRETE-CONTINUOUS BLACK- BOX PROMPT LEARNING FOR LANGUAGE MODELS
Hualin Zhang Haozhen Zhang²- Zhekai Liu
Bin Gu
Yi Chang2451
Deppatment of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence Sccooo of Artificial Intelligence, Jilin University 'School of Mathematics, Jilin University *International Center of Future Science, Jilin University
'ngineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE zhanghual i 1nn.nlalinnn@@00mmm gma 1 com haozhen23 1s. jlu. edu. cn Liu_Zheka @oot look. com js g con yichange jlu. edu. en
ABSTRACT
Large Scale Pre-Trained Language Models (PTMs) have demonstrated unprece- dented capabilities across diverse natural language processing tasks. Adapt- ing such models to downstream tasks is computationally intensive and time- consuming, particularly in black-box scenarios common Language-Model-as a-Service (LMaaS) environments, where model parameters and gradients are in- accessible. Recently, black-box prompt learning using zeroth-order gradients has emerged as promising approach to address these challenges by optimiz- ing learnable continuous prompts embedding spaces, starting with randomly initialized discrete text prompts. However, its reliance randomly initialized discrete prompts limits adaptability to diverse downstream tasks or models. To address this limitation, this paper introduces ZO-PoG, novel framework that optimizes prompts through a collaborative approach, combining Policy Gradient optimization for initial discrete text prompts and Zeroth-Orde Order optimization for continuous prompts in embedding space. By optimizing collaboratively between discrete and continuous prompts, ZO-PoG maximizes adaptability to downstream tasks, achieving superior results without direct access to the model's internal struc- tures. Importantly, we establish the sub-linear convergence of ZO-PoG under mild assumptions. The experiments on different datasets demonstrate significant im- provements in various tasks compared to the baselines. Our code is available at: http://githbb.ommm/aaggaaiii///t
INTRODUCTION
I Large-scale Pre-trained Language Models (PTMs) have demonstrated exceptional capabilities across broad spectrum of natural language processing tasks (Devlin, 2018; 8; Raffel al., 2020; Brown, 2020; Zhang et al., 2021; Zeng et al., 2021; Sun et al., 2021; Fedus et al., 2022), particularly since the advent of models like GPT-3 by (Brown, 2020). Their popularity stems from their ability to generalize to various downstream tasks with minimal supervision, often through few-shot or zero- shot learning.
However, commercial models often limit users to interacting with PTMs solely via APIs, offer- ing no access to the underlying parameters or gradients so we cannot optimize the model by the traditional method. In reality, adapting PTMs to specific tasks remains computationally intensive, especially in black-box scenarios common in Language-Modes -eeee-Mood------reeeeee (Zhao et al., 2022; Sun et al., 2022b), where direct access model parameters and gradients is stricted. Moreover, the growing size and complexity pose significant challenges for practical adap- tation and deployment. Fine-tuning PTMs requires extensive computational resources and access to model parameters, which is often impractical for many users due to cost and privacy concerns.
(LMaaS) environments
"quual contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2025
To overcome these challenges, black-box prompt learning has emerged promising solution. This approach seeks to modify and optimize input prompts to improve task performance while avoiding the need for full model access or fine-tuning. Prompt learning focuses crafting or optimizing input prompts to guide the PTM in producing task-relevant outputs. A successful prompt can trans- form the model's behavior various downstream tasks, improving performance without modifying the model. In scenarios where only function evaluations are available, derivative-free optimization (DFO) algorithms have been widely used. In a black-box setting, this optimization needs to be derivative-free, as gradients are not available.
Building on these insights, several approaches for black-box prompt learning have emerged, such as Black-Box Tuning (BBTT) (Sun et al., 2022a). which optimizes continuous prompts through ferivative-free methods in embedding space. Since the dimensionality of the original embedding space could be tens of thousands while DFO methods suffer from high variance in high-dimensional space. BBT proposed dimensional subspace based on the fact that common PTMs already have a low intrinsic dimen- sionality (Li et al., 2018; Aghajanyan et al., 2020). This leads to an effective way of utilizing DFO algorithms for black-bot box prompt learning. Despite these advances, these methods encounter notable limitations.
solve this problem by optimizing the continuous prompts in low-
Continuous prompt tuning often starts with randomly initialized discrete prompts. which can result in suboptimal performance and limit adaptability across different tasks and models. The discrete- to-continuous transfer lacks guidance from task-specific signals, reducing the efficiency and adapt- ability of the learned prompts. Besides optimizing continuous prompt in embedding space, recent research has also proposed effective methods for optimizing discrete prompts directly (Deng al., 2022: Diao et al., 2022). These approaches have demonstrated promising results, offering pathways to enhance discrete prompt optimization. However, while these methods offer a structured way to optimize discrete prompts, they often struggle to efficiently translate task-specific information into the continuous embeddings used by language models or effectively guide model behavior in black-box settings where internal parameters are inaccessible.
This brings us to a critical question: can we harness the strengths of both discrete and continuous prompt optimization to create a more adaptable and robust prompting framework? In order to tackle the above problem, this paper presents zeroth-order and policy gradient method (ZO-PoG) a collab- orative framework that alternates between optimizing discrete prompts and continuous prompts in low-dimensional intrinsic spaces. For discrete prompt optimization, we first reparamterize the cate- gorical distribution of the PTM vocabulary with Gumbel--Softma trick (Maddison et al., 2016; Jang et al., 2016) and use the policy gradient method (Diao et al., 2022) for gradient approximation in the black-box setting. This allows to optimize in parameter space rather than token probabilities, mitigating bias. For continuous prompt optimization, we use zeroth-order gradient to optimize in the lower-dimensional subspace by using a random projection. By doing so, ZO-PoG bridges the gap between discrete and continuous prompt learning, ensuring that prompts are tailored to maximize performance across diverse downstream tasks in black- environments.
The main contributions of this paper are summarized as follows:
We introduce ZO-PoG, prompt learning framework that alternates between optimizing discrete and continuous prompts. This approach is the first to jointly optimize both discrete text prompts and the continuous embeddings for black-box PTMs, enhancing the adapt- ability and performance of the prompts across different downstream tasks.
We formally establish the convergence of our framework, showing that ZO-PoG achieves a sub-linear convergence rate under mild assumptions. Our analysis highlights how vari- ance and biases are controlled by hyperparameters and mini-batch sizes, ensuring that our method remains effective for tuning PTMS without accessing model parameters.
Through extensive experiments various datasets, demonstrate that ZO-PoG signif- icantly improves the performance of PTMs. Our findings also underscore the importance of prompt initialization, showing that optimized starting prompt is crucial for achiev- ing better performance, thereby enhancing the overall effectiveness of our collaborative optimization strategy.
2

---

Published as a conference paper at ICLR 2025
CMA-ES
x
Hor's swecsome <MASKC
Waite my lime Tuss--MAA -MAK
48
like this movic. .MASI
Black-bo
Inpur Embeekding
BBT
Zenoth-Doder gradient
ZO-PuG
x RY
softmms
That's porsome! was <MASK
Wasie lime was <MASK>
&llogg m
like the <MASK-
Altemating optimizaation
Cups
Sampting
Inpes Batheddling
A
Black-box
Policy gnddeet
Figure I: Overview of BBT (Up) and the proposed ZO-PoG framework (Down). Alternating optimization process between discrete and continuous prompt tuning in the ZO-PoG framework. The discrete prompts are first optimized through policy gradient in the parameter space to enhance interpretability and task-specific alignment. Subsequently, the continuous prompt representations refined using zeroth-order gradient optimization in low-dimensional intrinsic space.
2 RELATED WORK
Language Models Service. Language Models Service (LMaaS), which early intro- duced by Zhao et al. (2022); Sun et al. (2022b), describes a diagram where models are accessible through APIs without direct access their internal parameters. This paradigm has grown promi- nence with systems like OpenAl's ChatGPT, Google's PaLM. Recently, several lines of research in the scope of LMaaS has focused on adapting pre-trained models various downstream tasks. One approach is through text prompting, where manually or automatically designed prompts help tailor model outputs (Brown, 2020; Kojima et al., 2022). In-context learning further enhances this by providing labeled examples within prompts quickly adapt the model (Chen et al., 2021; Xie et al., 2021). The capabilities PTMs also evaluated across multiple benchmarks (Liang et al., 2022: Chang et al., 2024; Bubeck al., 2023).
Black-box Prompt Learning. Black-box prompt learning methods aim to modify input prompts to guide the PTM toward producing task-relevant outputs. A key challenge lies in optimizing these prompts without access to gradients or internal parameters, leading to reliance on derivative-free optimization (DFO) algorithms. One line of this kind of works focused optimizing the soft prompts in continuous embedding space. One notable work in this area is Black-Box Tuning (BBT) (Sun et al., 2022b), which optimizes soft prompt with Covariance Matrix Adaptation Evolution Strategy (CMA-ES). BBT addressed the challenge of high dimensionality in the embedding space by focusing on a diver-dimensional subspace. leveraging the low intrinsic dimensionality of PTMs. BBTv2 (Sun et al., 2022a), an improved version of BBT. optimizes prompts across all layers of pre-trained language model using gradient-free divide-and-con -nd--uuuee algorithm, achieving few-shot learning performance comparable to full model tuning with significantly fewer tunable parameters. Another line of work focused on optimizing discrete prompt tokens black-box scenarios, which more suitable for black-box large language models with inference APIs like ChatGPT, GPT-4, Claude, etc. These work can mainly be categorized by reinforcement learning (Deng et al., 2022; Zhang al., 2023), ensemble (Hou et al., 2023), iterative search (Prasad et al., 2022), policy gradient optimization (Diao et al., 2022), Bayesian optimization for instruction optinization( (Chha et al., 2024; Hu al., 2024), multi-armed bandit (Pryzant et al., 2023; Lin et al., 2023; Shi et al., 2024).
3

---

Published as a conference paper at ICLR 2025
3 METHODOLOGY
In this section, we start by introducing the problem formulation in Section 3.1, where we define the task of optimizing prompts for pre-trained language model (PTM) in a black-box setting. Specif- ically, we focus optimizing input prompts to improve the model's performance on downstream tasks. Then, in Section 3.2, we present the discrete prompt optimization approach using policy gradi- ent estimations to find optimal prompts over the vocabulary space. Next, we proceed to Section 3.3, where we describe continuous prompt learning through zeroth-order gradient (ZO), which allows for fine- dine-grained adjustments in the prompt embeddings without requiring access gradients. Finally, we present our alternating optimization strategy in Section 3.4, where we collaboratively optimize both discrete and continuous prompts, enabling effective performance downstream tasks.
Notations.
Throughout this paper, we use bold uppercase letters (e.g., A) to denote matrices and bold lowercase letters (e.g.. z) to denote vectors. We use T [L 12...., Ln] to represent the prompt tokens of length n and each token I, is chosen from the PTM vocabulary V. We use p = [pi..... .P.] E R to denote the probability distribution of the prompt tokens over the vocabulary. Each p3 should satisfy the categorical distribution, i.e., Vi = 1, n. == {p : ||p|| = 1,0 3 p 3 1). We use e(-) ym - IR D to denote the map from prompt tokens of length n the prompt embedding of dimension D. We use ||
to denote the Euclidean norm of vector.
3.1 PROBLEM FORMULATION
Given a pre-trained language model (PTM) in black-box scenario, aim optimize the input prompts to enhance the model's performance on specific downstream tasks. Specifically, for a batch of primitive texts X and labels Y. the general goal of prompt learning to find the optimal prompt p = arg min p [pf X), Y), where D is the dimensionality of the prompt embedding space and f represents a black-box PTM where the inputs are combined with learnable prompts and the primitive texts X. Directly optimizing the prompt in the embedding space is impractical due to the high dimensionality, which can reach tens of thousands. Sun et al. (2022b) proposed to optimize over the subspace of the original embedding space with much lower intrinsic dimensionality, Specifically, the objective is
z' = arg min C(f(Az + X),
where z E Rdd, A E TR. Dxd is the projection matrix (with d < D), D= (X, (X,Y) Y} is the total dataset, and Po E R is the initial prompt, which are embedded tokens chosen randomly from the PTM vocabulary.
However, random initialization may lead to a suboptimal prompt. A more effective approach is to optimize the initial prompt Po- Directly optimizing the initial prompt in the embedding space presents the same challenge of high dimensionality. To address this, we propose to optimize the initial prompt Po in the discrete probability space of the PTM vocabulary, rather than the continuous embedding space. Specifically, instead of selecting prompt tokens uniformly at random from the aim to learn distribution over the vocabulary, allowing for more targeted and
PTM vocabulary, meaningful token selection. The details of the optimization method of the probability distribution will be discussed in Section 3.2.
Based on the above considerations, we will optimize the initial discrete prompt po by learning a probability distribution over the vocabulary rather than selecting prompt tokens randomly. This allows us to improve the selection of tokens and guide the model more effectively in black-box settings. Following this, we will alternate between optimizing the discrete prompt Po and optimizing the continuous prompt z in the lower-dimensional intrinsic space, ensuring an efficient and scalable solution to prompt learning black-box scenarios. The main objective of this work becomes:
min E (S(Az po: E + e(T);
P(T).z
In the above objective, the prompt p can be decomposed into three components: an initial discrete prompt embedding Po R obtained by sampling tokens from the PTM vocabulary with trainable distribution, a continuous intrinsic low dimensional vector z E Rd. and a random matrix A E Dxd that projects back to the original high dimensional prompt embedding space.
4

---

Published as a conference paper at ICLR 2025
3.2 BLACK-BOS DISCRETE PROMPT LEARNING WITH POLICY GRADIENT
To optimize discrete text prompts in black-box scenarios, we adopt a policy gradient approach. Specifically, we aim optimize the distribution of discrete prompt tokens over given vocabulary, where each token is represented by probability distribution. Let p .p,.....P! .Pn] R denote the probability distribution of n discrete prompt tokens over the vocabulary of size [V], where P; E C {p ||p|| 1,0 p 1).
Gumbel-Softmax Trick for Smoothing Optimization. Instead of directly manipulating the prob- ability distribution, we reparameterize the categorical distribution using the Gumbel-Softmax trick (Jang et al., 2016; Maddison et al., 2016). This allows us to transform the optimization process into one over continuous parameters, reducing bias and ensuring smooth optimization. To be specific,
log(as, +9
Ps.j softmax (log(ar)) + Di.1 )
exp
(2)
Σ exp
where Cii denotes the j-th component of column vector at 9t.j is the Gumbel random variable, which is sampled from the Gumbel(0, 1) distribution and is the temperature parameter controlling the smoothness of the approximation. After reparameterization, the optimization objective becomes:
min F PT-P(Tio) X),);
a.x
For abbreviation, we denote L(f cT)) as L(T) for fixed in the rest of this subsec- tion.
Since accessing the model's internal parameters is often restricted, the problem involves optimizing the discrete probability distribution with just forward passes through the black-box models. To achieve this, we will adopt the policy gradient estimator (PGE) to optimize the above objective. According the policy gradient theorem (Sutton al., 1999), can estimate the parameter's gradient with respect to a; by:
V. Er(((T) = Σ ((TVI,PPTTTO) Σ C(T) P(T(a)
P(T V P(TIa)
TEVn
TEV=
= Σ P(T]a) log P(T (P())) log T P(sylas)
TEV-
= =Ep(ra))((()V log P(L)a.))
Variance Reduction in Policy Gradient Estimation. One of the primary challenges in applying policy gradient methods is the high variance in gradient estimates, which can lead to unstable updates and slow convergence. To address this, we implement a variance-reduced policy gradient estimator (VR-PGE) (Williams, 1992; Dong et al., 2020; Zhou et al., 2021; Diao et al., 2022), which reduces variance by subtracting a baseline from the reward function during updates. The estimated gradient is calculated by:
- carbo)) Va, log P(ts(x)),
or
-
(4)
j=1
where (k)) l 1,.... II are sampled independently from P(T with parameter a E R|xn
3.3 BLACK-BAL BOx CONTINUOUS PROMPT LEARNING WITH ZEROTH-ORDER GRADIENT
In addition to optimizing discrete prompts described in Section 3.2, continuous prompts provide more fine- fine-grained mechanism to guide pre-trained models (PTMs) in black-box settings. Given that the internal gradients of black-box models are inaccessible, zeroth-order gradient (ZO) (Nesterov & Spokoiny, 2017) is an effective approach for optimizing continuous prompts without needing gradient information.
For abbreviation, we denote C(f(Az + x)T); as L(z) for fixed e(T) in the rest of this subsection. Then, aim to optimize low-dimensional continuous vector Rd in the intrinsic
5

---

Published as a conference paper at ICLR 2025
subspace of the prompt embedding space. The Zo gradient estimator approximates the gradient of the loss function using a symmetric difference:
C(z yu)- L(z lu) u,
92
(5)
2p
where j is the smoothing parameter and E R2 is distributed in M Ia).
3.4 ALTERNATING OPTIMIZATION BETWEEN DISCRETE AND CONTINUOL'S PROMPTS
In this section, we describe our proposed optimization algorithm, which alternates between optimiz- ing discrete and continuous prompts. This method takes advantage of the unique properties both prompt types effectively enhance performance in black-box models. The algorithm leverages the strengths policy gradient for discrete prompt optimization and zeroth-order gradient estimation for continuous prompts, achieving efficient prompt learning in black-bos settings.
In each iteration, the optimization process alternates between two key steps: 1) Discrete Prompt Optimization: This involves learning distribution over discrete prompt tokens from the model's vocabulary. Each token in the prompt is sampled from a Gumbel-Softmax distribution, and policy gradient methods are used to update the distribution parameters a E X 2) Continuous Prompt Optimization: After discrete prompts are optimized, the continuous prompt, represented low- dimensional vector z E Rd in the prompt embedding space, is optimized using zeroth-order gradient estimation. This is crucial in the black-box setting where internal gradients unavailable.
By alternating between discrete prompt optimization and continuous prompt optimization ensures that both the categorical structure of tokens and the fine-grained continuous embeddings are op- timized tandem. This complementary approach maximizes the model's performance on down- stream tasks by balancing discrete token selection with smooth embedding adjustments.
Algorithm Black-Box Prompt Learning via Zeroth-Ordee and Policy Gradient Method
Input:
Parameters of categorical distributions ai, On, temperature parameter т, prediction model f. PTMs embedding function e(-), downstream dataset DD mini-batch size B, learning rates Ma 722 and sample times J1, Tz.
1: Initialize a
2: for in 0 to T
do
3: Draw mini-batch S; (Xi, Yi) fzi, Vi 1 from D
4: for in to In do
5: Sample Ji GS(a), r). , ~ GS 7)
(k)
Po,: e(t (k)
(k) e 1, J,... , (1)
6:
7:
Cavy = +. Σk C(f(Az' + +Po.: X.)
(N),
8:
for 1 to do
-ar f1-1 Eh (C(f(Azz (k) Po.:: (,), - Lavg)Va log P() lof)
9:
10:
a; [+1 + a Maga
Update discrete vocabulary distribution
11:
for to Ia do
12:
Sample 11. ~ GS(a 1+1 ,),,,,),----)) Fnnd ~ GS(att1,7)
13:
Po.: = e(t())
1,0
(11) = e(v()())
(0) v(((1
14:
Sample u, (o,
L(f(A(z' + yu!
(1) - (.) - ((f(A jou(") Po. X²). Y.) u.
Po. 1
(
15:
2/
1 Ei 9z
16:
9x
In
17:
*+1 = x 11292'
>. Update continuous prompt
Return:
of .... 1/ zt

---

Published as a conference paper at ICLR 2025
4 CONVERGENCE ANALYSIS
In this section, we provide detailed analysis of the convergence behavior of our proposed frame- work. Our goal is to demonstrate the convergence properties of the alternating optimization process that integrates the policy gradient method for discrete prompt learning and the zeroth-order method for continuous prompt optimization.
We begin by establishing several basic assumptions about the smoothness and bounded variance of the loss function. Under these assumptions, derive the convergence rate of our method and analyze the impact of both the policy gradient and zeroth-order gradient on the overall convergence. Specifically, we make the following assumptions:
Assumption (Smoothness). For convenience, we will use [(a,z) z) to represent the loss function (A + Po; )), Y), where D = (X.Y) is the whole dataset, f is the prediction function, and Po is sampled from the token distribution which is generated by az Suppose that L()) is block-wise smooth with gradient Lipschitz constant Lax, that for Vox, a' E V
TR.
V z) - Vaf(a', 2)ll S Lallox- - a'll-
(6)
and block-wise wise smooth with gradient Lipschitz constant Ly that is for Vz, z' E &R, R VaL(a, z) VzCiox x')ll I|| I Tz2liz- z'||-
(7)
Assumption 2 (Bounded Variance). Assume that the stochastic gradient has bounded variance, i.e.,
E Bix.w.)epllaaaa(( z) Vaf(a, 2)11 S da. E (E.DDDD VxCi(ar - VaL(a, 2) 1] S oz Va E Rlvxx
Vz E R
(8)
(9)
Assumption There exists a positive constant < oa such that [Ls((x2)) < C, Vo RIV xn E Rdd
Assumption 4. In the policy gradient method, we optimize au where is from 1 to n. The elements in parameter vector a; can be written by Og, has lower bound denote:
S Oi.j.
Remark 1. In our experiments, after each iteration, we truncate elements aij to 1 x 10-11 if it falls below this threshold. This precaution prevents the gradient as from becoming excessively large or meaningless during optimization, which can occur due to the properties of the logarithm.
Proposition 1. Under Assumption 2, 3 and the stochastic policy gradient Ia, has bounded variance
Es EeBTorprlllo. -
20 < 2T Va,fs(aa +2 V Ls,,, Va, 2)12 <
20
B
Where is foxed
Variance from mini batch S
where o² 8C2V1 S (I and Oc. ZE>0.
g2g2
Remark 2. This proposition demonstrates that the variance of the policy gradient is bounded under the assumptions provided. The bounds depend on the bati-batch size B and the sampling times I, indicating that the variance can be controlled by choosing larger mini- mini-batches and improving the gradient approximation technique.
Based on these assumptions, we establish the following convergence result:
Theorem 1. Under Assumption 1, and let (a",z') TT) be the sequence generated
1
1
o /nkko
by running Algorithm 1, set aa S
Tae S
B
2La
40(d + +4)L2z'
2 m
ROZ 2
2
and T' =0 (0" ['mmar)
-(((mann)
O
1 0
we have
2
Limax
T-1
Σ
C( 1 C( xt)l13) < O
E
(10)
where Lmax max (d+4) L) Limin = min Las (d + Lz and k = Lmmaa/ Limin- Remark 3. The above theorem establishes thut Z0-PoG achieves an e-stationary point in expecta- tion with total query complexity (forward passes of the PTMs) T = O
/AK
7

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
EXPERIMENT SETUPS
Datasets. For performance evaluation, we chose 5 commonly utilized datasets from the GLUE benchmark (Wang et al., 2018): CoLA (Warstadt al., 2018), MNLI (Williams et al., 2017), QNLI (Wang et al., 2019), SNLI (Bowman et al., 2015), and WNLI (Levesque et al., 2012). These datasets encompass various typical language understanding tasks such as natural language inference.
Baselines. In our comparative analysis, we evaluate our proposed method against the following black-box prompt learning techniques under the same experimental conditions: Manual Prompt (MP) executes zero-shot evaluations on LLMs with human crauunerratedd templates, providing imitial per- formance metrics. BBT (Sun et al., 2022b) optimizes continuous prompts within random low- dimensional subspace utilizing the Covariance Matrix Adaptation Evolution Strategy. BDPL (Diao al., 2022) is a policy gradient- based black-box discrete prompt learning method, which is men- tioned in section 3.2. SSPT (Zhang et al., 2024) builds upon the BBT optimization paradigm by employing subspace learning and selection to identify the optimal ulrra-lw-ddiensionn subspace, replacing the random subspace.
Implementation Details. The experiments are executed on a cluster of NVIDIA A40 GPUs. We employ RoBERTa-large (Liu et al., 2019), GPT2-XL (Radford et al., 2019), and Llama3 (AI@ Meta, 2024) as our backbone models, and all pre-trained weights are sourced directly from HuggingFace. All experiments performed under the few-shot learning setting. We assemble the training and development sets by randomly selecting m instances for each class from the original training data. Comprehensive details of the input templates and hyperparameters used in our experiments can be found in Appendix B.
Table 1: Comparison of test accuracy (tean-istd) on RoBERTa-LLage in 16-shot (per class) set- ting. The best results are underlined. The bold represents the optimal result in the prompt learning approach. Len represents the prompt length.
Len
Method
CoLA
MNLI
QNLI
SNLI
WNLI
MP
48.51
41.61
51.91
38.00
42.25
BBT
56.1813.34
40.55:10.08
12.97=11588
37.3911.33
50.23
BDPL
38.83=1.00
13.22.11.63
55.25-2.39
38.04:1..30
39.91
20
SSPT
44.65+7.552
41.72-40.40
53.10+2.23
36.84412.99
46.01 +3.54
Ours 16.29
0..70
55.44 1.98
39.12-0.229
53.99 +0.81
BBT 49.76.14.51
41.95
52.83
38.80.L0.35
16.99
BDPL 42.73.14.64
40.98
54.74
38.48.10.69
43.19
50
SSPT
19.60+1.20
42.4
53.74=0.14
41.52+22.05
51.17+4.07
Ours
56.18-7.59
42.53+0.81
55.33-0.32
41.99++0.00
56.34+1.41
Table 2: Comparison of test accuracy (mean=istd) on GPT2-XL in 16-shot (per class) setting. The best results are underlined. The bold represents the optimal result in the prompt learning approach. Len represents the prompt length.
Len
Method
CoLA
MNLI
QNLI
SNLI
WNLI
MP
67.88
37.25
50,30
35.16
47.89
BBT
30.9720.10
33.81+2.300
50.73+0.30
33.25-00.09
55.40=1..3
BDPL
45.29
38.18+00.3
51.96=0.81
35.61+1.08
42.25
20
SSPT
49.59.13.54
37.9312.11
50.97-10.99
36.2510044
48.8311.63
Ours 53.69-1.774
3..7
52.21
37.17
10.81
BBT 31.83+0.110
33.62+0061
50.72+0.17
33.2740.12
55.40+11.3
BDPL 49.15+11.82
37.2310.35
11.75+1.06
35.1710.93
41.78
50
SSPT
53.1841.29
37.81+2.02
51.17+124
37.8441034
53.5242.82
Ours
58.87-2..55
39.58-10.00
51.85-10.49
37.38
56.34
8

---

Published as a conference paper at ICLR 2025
Table Comparison of test accuracy (mean-sstd) Llama3 in 16-shot (per class) setting. The best results are underlined. The bold represents the optimal result in the prompt learning approach. Len represents the prompt length.
Len
Method
CoLA
MNLI
QNLI
SNLI
WNLI
MP
60.50
33.98
52.65
38.01
57.75
BBT
45.8917.32
33.3010..5
49.16.10.30
31.8811.08
47.42.15.33
BDPL
45.83+7.12
37.33++3.1
55.11=1.21
34.4
54.93+2.44
20
SSPT
41.87+2.13
38.25++3.44
54.24+1.09
36.70++3.00
53.52+2.82
Ours 48.13=5.93
38.84:12.46
55.90-2289
38.26:+2.85
59.15
BBT 44.84
33.70.+0.24
50.29-10.3
33.20++0.39
51.24
BDPL
19.98=1.26
37.06+1.65
13.80=1.76
37.32+3.17
57.28=0.81
SSPT
44.97=9.63
14.
52.20=2.66
34.01
55.40=1163
Ours
66.50-16.88
37.19
55.60
38.12.10.86
58.22.1.15
5.2 MAIN RESULTS
We perform experiments with different prompt lengths and report the mean test accuracy over random seeds. The comparative results for each of the backhone models detailed in Tables 1, 2, and 3, respectively. Notably, the proposed collaborative optimization method outperforms other prompt learning methods across all datasets. For instance, when employing the RoBERTa-large model on WNLI with a prompt length of 50, ZO-PoG achieves an improvement of 5.17% over the best baseline result. The experimental results confirm the effectiveness of jointly optimizing both the discrete initialization prompt and the continuous prompt. When using GPT2-XL and Llama3 as backbone models on the CoLA dataset, all prompt learning methods perform worse than manual prompts. This can be attributed to the AA task's focus on grammatical acceptability, which aligns better with the pretraining objectives of encoder-only models like RoBERTa. Decoder-only models like GPT2-XL and Llama3, pretrained for generative tasks, less sensitive to grammatical cor- rectness unless explicitly encoded in the prompt. Additionally, learned prompts, such as those from BDPL, may include grammatically incorrect tokens, further impairing the performance of decoder- only models. In contrast, manual prompts, designed with grammatical correctness, provide clearer guidance for these models, explaining their superior performance CoLA. Additionally, the sults for prompt lengths of 20 and 50 do not demonstrate a strict correlation between prompt length and performance. We believe that shorter prompt lengths may constrain representational capac- ity, whereas longer prompt lengths increase the number of parameters, making optimization more challenging. We conducted an ablation study on the effect of prompt length Appendix C.
55
52-
50-
09/A
1911
ONLI
Whi1
03.
ONL
WX11
0.1.A
M8211
OR1
Whit
PuSERTT
OPT2.AL
Unmaa
55
50-
40
63
CAlA
1811
DNLI
CoLA
GNL
Shi1
WH1
CoLA
M801
Gh11 [Imma]
Whil
GF77-0)
Figure 2: Ablations of the prompt lengths of (top) and 50 (bottom).
trick on RoBERTaLarge, GPT2-XL and Llama3 with

---

Published as a conference paper at ICLR 2025
5.3 ABLATION STUDY
Effect of Gumbel-Softmax Trick. We removed the Gumbel-Softmax trick from ZO-PoG and re- placed it with the policy gradient estimation method used in (Diao al., 2022). The comparative results are presented in Figure 2. The results indicate that the Gumbel-Softmax trick has a positive impact on the overall optimization in ZO-PoG.
Effeet of Optimization Component. We separately removed the discrete prompt optimization component (w/o Policy Gradient) and the continuous prompt optimization component (w/o Zeroth Order) from ZO-PoG. The experimental results for the prompt lengths of 20 and 50 are presented in Figure 3. can be observed that ZO-PoG consistently maintains the best performance across all backbone models. This demonstrates that ZO-PoG is capable of effectively combining both prompt learning approaches, significantly enhancing the performance of L.MSs on downstream tasks.
Test Accuracy
60
60
40
20
20
0
CoLA
MNLI
QNLI
SNLI
WNLI
CoLA
MNLI
QNLI
SNLI
WNLI
NoBERTa-Large
RoBERTa-Large
60
60
Test Accuracy
40
40
20
20
CoLA
MNLI
QNLI
SNLI
WNLI
CoLA
MNLI
ONLI
SNLI
WNLI
GPT2-XL
GPT2-XL
60
60
Test Accuracy
40
40
20
20
CoLA
MNLI
QNLI Llama3
SNLI
WNLI
CoLA
MNLI
QNLI Llama3
SNU
WNLI
w/o Policy Gradient
Zeroth Order
Ours
Figure 3: Ablations of the collaborative optimization components RoBERTa-Larg, GPT2-XL and Llama3 with the prompt lengths of 20 (left) and 50 (right).
CONCLUSION
In this paper, we propose novel black-box prompt tuning framework for PTMs named Zeroth- Order and Policy Gradient (ZO-PoG) method, which optimizes both discrete and continuous prompts in the black-box setting. During the tuning process, we alternately optimize the discrete prompts using the policy gradient in its parameters space and continuous prompts using the zeroth- order gradient in the parameter's 's low-intrinsic space. Our experiments show that ZO-PoG signifi- cantly improves the Large language model's performance in various downstream tasks while reduc- ing the computational expense the Black-box setting. Moreover, our convergence analysis proves the effectiveness of our method.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported part by the National Key R&D Program of China under Grant NN0.2023YFF0005000,, National Natural Science Foundation of China through grants (No.U234122)) No.62076138)
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model finennuning. arXiv preprint arXiv:2012.13255, 2020.
Al@Meta. Llama model card. 2024. URL https://github: wttp://www....oettaaaaaaaaeaaaae
1lama 2aama3/blobmmaee Balllo//main/MOEEL CARD md.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Mamning, A large anno- tated corpus for learning natural language inference. arXiv preprint arXiv:1008.05326, 2015.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 2023.
Sparks of artificial general
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang. et al. A survey on evaluation of large language models. ACM Transactions Intelligent Systems and Technology, 15(3):1-45, 2024.
Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. Instructzero: Eff- cient instruction optimization for black-box large language models. In Forty-first International Conference on Machine Learning, 2024.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning. arXiv preprint 1rXiv:2110.078/4 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1416 2021.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang. Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205. /2488, 2022.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018
Shizhe Diao, Zhichao Huang, Ruijia Xu, Xuechun Li, Yong Lin, Xiao Zhou, and Tong Zhang. Black-box prompt learning for pre-trained language models. arXiv preprint arXiv:2201.085: 2022.
Zhe Dong, Andriy Mnih, and George Tucker. Disarm: An antithetic gradient estimator for binary latent variables. Advances neural information processing systems, 33:1 13:18637-18647, 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Bairu Hou, Joe O'connop Jacob Andreas, Shiyu Chang, and Yang Zhang. Promptboosting: Black- box text classification with ten forward passes. In International Conference on Muchine Learning, pp. 13309-13324, PMLR, 2023.
Wenyang Hu, Yao Shu, Zongmin Yu, Zhaoxuan Wu, Xiangqiang Lin, Zhongxiang Dai, See-Kiong Ng, and Bryan Kian Hsiang Low. Localized zeroth-order prompt optimization. arXiv preprint M:Xiv:4433002993 2024.
11

---

Published as a conference paper at ICLR 2025
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriizttion with gumbel-softnax. arXiv preprint arXiv:1611.01 144, 2016.
Takeshi Kojima,
Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213. 2022.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning, 2012.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv: 1804.0888, 2018.
Perey I. Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:221 1.022/2/100111 10, 2022.
Xiaoqiang Lin, Zhaoxuan Wu, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng. Patrick Jaillet, and Bryan Kian Hsiang Low. Use your instinct: Instruction optimization using neural 2023.
bandits coupled with transformers. arXiv preprint arXiv:2310.0220
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoerr and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rXf::197.111622, 2019.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint 1oXiv:6611.00712 2016.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-566, 2017.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based in- struction search for prompting large language models. arXiv preprint arXiv:2203.07281, 2022.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1 170::111667776600 2020.
Chengshuai Shi, Kun Yang. Zihan Chen, Jundong Li, Jing Yang. and Cong Shen. Efficient prompt optimization through the lens of best arm identification. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. Bbtv2: Towards gradient-free future with large language models. arXiv preprint a2Xiv:2005./220) 2022a.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for sariag--mmool-----erviviie In International Conference on Machine Learning. pp. 20841- 20855. PMLR, 2022b.
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang. Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2107.02137 2021.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
12

---

Published as a conference paper at ICLR 2025
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Jan 2018. doi: 10.1 10.18653/v1/w18-1 w1 6.
URL http: /xx doi org/10. 1. 18653/v1,
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel R Bowman. Superglue: A multi-task benchmark and analysis platform for natural language understanding. Advances in Neural Information Processing Systems, 32:3261- 3275, 2019.
Alex Warstadt, Amanpreet Singh, and SamuelR. Bowman. Neural network acceptabilitty judgments. arXiv: Computation and canguage,arXivv Computation and Language, May 2018.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprini arXiv: 7004.1444...66 2017.
Ronald J Williams. Simple statistical
algorithms for connectionist reinforcement
learning. Machine learning, 8:229-2:1 256, 1992.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.0200 2021.
Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. Pangu-alpha: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint vrXiv:2104.22669 2021.
Haozhen Zhang, Hualin Zhang, Bin Gu, and Yi Chang. Subspace selection based prompt tuning with nonconvex nonsmooth black-box optimization. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4179-4190, 2024.
Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. TEM- PERA: Test-time prompt editing via reinforcement learning. In The Eleventh International Con- ference Learning Representations, 2023.
Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, et al. Cpm: A large-scale generative chinese pre-trained language model. Al Open, 2:93 99,999998909 2021.
Mengjie Zhao, Fei Mi, Yasheng Wang. Minglei Li, Xin Jiang, Qun Liu, and Hinrich Schuetze. LM- Turk: Few-shot learners as crowdsourcing workers in a -anguuge-model-as---sseere
framework. de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings In Marine Carpuat, Computational Linguistics: NAACL 2022, pp. 075-692, Seattle, United of the Association for
States, July 2022. Association for Computational Linguistics.
Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, and Tong Zhang. Efficient neural net- work training via forward and backward propagation sparsification. Advances in neural informa- tion processing systems, 14:15216-15229, 2021.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A CONVERGENCE RATE ANALYSIS
A.1 AUXILIARY LEMMAS
Lemma 1. (Notation in Nesterov Spokainy (2017)) Denote: f E Cl.1(RR), if: [v](() VV (1))l S L|y - x||, where z, y E Rd. This condition is equivalent to the following inequality:
I,²
(f(v) f(x) - (Vf(z), - x)1 S lly - =|l?,
T, E Rd.
(11)
2
Lemma 2. (Lemma 1 in Nesterov & Spokoiny (2017)) We often need bounds for the moments:
Mp R: 22 | -ull'e-=lllll" =1lll² -uu,
where &. = SRa e 2|llll du. For E [0, represents the dimensional of the space, then we have:
Mp S do/2.
p 2 2, then have two-side bounds:
d!!2 Mp (p + dyp/?.
(12)
Lemma 3. (Theorem 3 in Nesterov & Spokoiny (2017)) If f differentiable at z, then
E. (190(2)11) - (d + 4) ()
(13)
Denote: 90(x) = f(xx u) where f( u) is a directional derivative of function f(x along E R", and ff(o)) == 1 Jaa (Vf(x), u) e || udu.
Lemma 4. f E C (IRd,, which means that [V) vf(y) < L1| 2, E Rd. Then, let random vectors U, Rd have stand Gaussian distribution (O, Ia) and return
(x(x) = H( + puss - [(x puss) U4.
2μ
Finally, we will have conclusion:
Eu (11x(()))P) S 222 (d 2(d 4)]
(14)
2
Proof. For the symmetric oracle 9r. we have
[s pm)) f(x 22) [5 pu)- [(x) (V[(x), (f(x (mm [(x) (V](x), mu))
+ f2V p2)
(0)
x 21f(x - pm) f(x) - (V) + (V)(z), mu))
+ fx (m) f(x) + (V (x), mw) + (V wuu?)
(1)
2
2
2 1| + (Vf(z), pm)
+2
L||u|| + (V (c))
2
2| μ (V) u)
2
Then, take the expectation with respect to a following the Gaussian distribution:
E. (11x,(z))1²
Eu
(If(x + pm) - f(x 11uu1
4u2
(a)
2
1.221u116) + Fm
Em
2
4
(12)
2
<
L2
+ 6)3 + 2(d ())
where inequality (a) holds because Elja- b
2
14

---

Published as a conference paper at ICLR 2025
Lemma 5. (Theorem 1 and Lemma 3 in Nesterov & Spokoiny (2017)) f C..l (RR) with constant
L, and x E R then for the bias with respect to the zeroth-order function:
[u(()) (())
- Ld.
(15)
2
For the bias with respect to the zeroth-order graddin:
[Vfx(=)-V(aa - vf(())] - 2 Ld + +3)3/2
(16)
where:
fu(z) = //e f(x + pu)e b1llll du;
Vf((z) = 1 / f(x + (u) - pm)) e - lull? udu.
JR"
2/1
Proposition 2. Now, let's 's analyze the boundedness of the zeroth-order gradient's 's variance. For any I RRdd we have:
- S 4(d [Vf(2)] 4(d + 4)02
1² 8B
(d 6)3, (17)
where f((x) and f(x) are the estimated gradient by the zeroth-order gradients. For v fi(x), we will sample one random disturbance us per sample; us follows Gaussian distribution. Then denote: 02 E. (x(x)) fz))) to represent the sample variance of the data sets.
Proof. Let Is(-) be indicator function, [s) { 1, i S
where S represents the mini-batch
0,
if $ S'
B
2
B
B(B 1)
set and denote 1 == Iss (i). Then we have E, (I3)
and E, (I,I,)=
when
n
1
n - 1)
i + j. Let
V (1)) vs (x)):
E
S (x)) - 4, (x))
E
Σ
Zz
E
2
B
B
YES
B
B(B - 1)
ET? ||
Σ
L (Zis 23)
FF
2
Σ (
B
71
n(n - 1) b4
1
E
B
B(B -
12
|
B(B 1)
Σ 7/
BB
11
n(n
1)
nnnn - 1)
E,
n
B
(a)
E,
Σ
2 1
Eu fz) (z) {f()))
nnn
1)B
B
B
1=1
[m1
$
E.
B
(14)
112
E,
(d + 2(d 2(d+4)
2
(
12
(.2 6)3 + 4(d 4(d+ 4) E, (I]V)(() (/ [V{
+
2B
B
4(d
+ 4)
[1+ 4(d + 4)0²
112
1.2 (d + 6)3
B
B
2B
n -
1 n 1
1 1
where incquality (a) holds because
<
< inequality (b) holds be-
nen 1)B
B
n(nn 1)
B n
cause Ella- + b1|2 < 2E |a||2 + 2
15

---

Published as a conference paper at ICLR 2025
A.2 PROOF OF PROPOSITION 1
Proof. The variance between the policy gradient and true gradient can divided into two parts, The first variance part comes from the sampling process with fixed mini-batch S and fixed P(T", and the second variance part comes from stochastic mini-batch S = {I,, ys } (Notice that the mini-batch we use in both zeroth-order and policy gradient methods are the same)
Es Eger+peryllee 1laar Va, 2)1 S2 BT Va,Cs(a, 2)1l +2 Es V Ls( Vo, 2)1?
(18)
(19)
tixed
Variance from mini-batch 8
Then we analyze the first part of the variance, where S is fixed.
Var(g = Var
a log P(t(t)) (Cs(a(th) -Cave)
((s(a(l)) Ls(a (=)((())) Vas log P((!4))
Var
k=1
E
([s(al(),)) Ls(aa)) .2)) Va, log P(( Va, (a)
57k
Σ (Cs(a(l)),)) Ls((s))),,)) -2)(a(()a) Va, log ((t - Cs(a,zz
E
( Va. log P(t(()) Va, Cs(az))
jzk
1
Σ (Cs(((()) -c)((a)))))
(2) Va. log p(t ) Va.Cs
7
jZk
3
Bait) (0 Mef || (Cs(a -
-) (s((a())22 .2) Vos log P(t(1)) |?
1(1 1)
(2)
4C²
Va log P(t /(
E Ba! (4) (00,),00) K4
1(1 1)
3
4C
( 8C
( 1)
722212
Above, inequality (c) holds because Ella b112 Ela]] Inequality (d) holds because Assumption 3, there exists a positive constant C' < oo such that (Cs 2) < C, Vav z. Incquality (e) holds because the j-th component of Va. log P(t.) could be solved explicitly by:
1
V a., P(t.) = (1 - Pijj.
j ja
TOG.j.
V Vas, log P(t a.) Va.,, logpis, =
1
V Voi, log P(t+) -Pi.i TOc..
j # j:.
Then we have
Va. log P(e!"))) s |V| max (1 (1 Pt.j.) Toi.j.
V
Pi.j
ToG.;
T2
The last inequality (f) holds when I 2 2, where |V] means the total number of the tokens in the vocabulary list.
16

---

Published as a conference paper at ICLR 2025
8 Denote o7 o²= r2g2 then we have:
E Pa- 2)s(a, 0²
1
On the other hand, under assumption 2, it easy to get Es| V La, ((a d This completes the proof of Proposition 1.
A.3 PROOF OF THE THEOREM
Notation recap:
C(x, z) is an abbreviation of the loss function. We utilize the cross-entropy loss function during the experiment.
(, z) represents Gaussian smoothing of the original loss function:
L,( 2) = { L(ar 2 + mu)e 1llu du.
Rd
where ya hyper-parameter and x x= S fpc 21lull du.
Va (-) means that the block-wise true gradient with respect to a and, VzC(.), (.C,()) represents that the block-wise true gradient with and zeroth-order gradient with respect to z. Where a is the parameter that can create discrete prompts distribution, optimized by the policy gradient algorithm and is the continuous prompt in the low-dimensional space optimized by the zeroth-orde gradient.
Iz means the estimated gradient with random mini-batch, and the VzC(-) represents the estimated gradient calculated by a total dataset.
9hr represents the policy gradient in the parameter space.
and n/ represent the learning rates with respect to VzL() and VaC(.) respectively. d represents the dimension of the parameter space, and represents the length of the ini- tialized discrete prompts. And each token is independent with others, so we can add their variances together directly.
Lz. La represent the Lipschitz constants with respect to L-Smooth functions L,(+) and C-(-) respectively.
B is the size of the mini-batch, represented sampling times in the policy gradient every iteration.
(X₁, Yi (3 (i and I is the
For convenience, let L([(Az + Po: X),Y) == L(o,, z), where represents the Black-box API, total dataset D (X,Y):
Og the variance from SGD during continuous prompt optimization, 0g is the variance
induced by policy gradient, and aa is the policy gradient variance from mini-batch.
Proof. According to the lemma 5, the value of the veroth-order function (Gaussian smooth: Origin function with Gaussian kernel) has a bias compared with the true function value:
(a, 2) - C(a, z) - 212 2
Lzd.
(20)
We can also know that (a, 2) with respect to z is a I,z Lipschitz smooth function and L(a,, with respect to a, is a Lo Lipschitz smooth function from the Assumption
2)
According to (12) in Nesterov & Spokoiny (2017), can now that L, is Lipschitz smooth for Vz, x', oc:
VVLC 2)
VaL, z'Ill T.uliz 2/11,
(21)
17

---

Published as a conference paper at ICLR 2025
1 K
L(a, + pu) C(a, 2 wu) =lllull udu, and
where (2Lm(a,z)
2p
Sgee-tllul du.
Firstly, because of the smoothness of the C() with respect to z from the inequality (7) and the bias from the zeroth-order gradient from the inequality (15):
(16) (aa'tl,!t1) L((xtt1 24) -
(fulatt).** ,2t12) + 112
2 ludd - (Lulat-, 22)
122
2 Inad
Eu(att) ,2t11)=== - (
,2¹) + 1²² Lidd
(21(11)) $ (VeLulatt) 22'), 1( - z²)
L
2011z" 1zzt+11 - zz + 1²² Lgd.
2
Secondly, according to smoothness of the L()) with respect to a according to the inequality (6) in the assumption 6, we use equation (11):
Le 2
[(alt),2 S L(a ,x') + (2)) -2') +
11 =11112 - + 1² Lnd
(6)II) S L(a', 2²) + (VaL(a', 2¹), a -a²) + La
22),
||a a²11
L Ix 1zz z/11 + 42 Lgd.
+ (VeLu(a**) z²), z 1--1-1111 z²)
After rearranging the formulation above:
= z 1= - L(x', z²)
< (VaL(a', a +1 a²) La Ella - a²11
I E
+ E V VaLu(a" 1+1 ",,,'112)) 2z))z²+1 -z²)
El|zzt 21112 3 + 11² Lyd
aaa la
T E El 112 + 112
Na V VaL(a', 2²
Bllgor
E
1/ Lf (tt11 z² z²)112 +
2
2
(
Na VaL(a', 2² + nia L. E(lg.r - VaLla' 2² +|| VI C(a , 21)133)
Varicance (19)
,2))|12 3n Lz ,,aa=,,,,))1
B B( vaL(a 2²)11 + (V=((a** ,2)) 22 (a 22)
2
Variance (17)
Flfa- Fa 1 then use (14)
+ (a 1+ 22))(3) 11+ + 12² T.2d
(
20a 9
202
31
(Na Ni La) El V L(a' z²) 2²)112 + n/a na
E|| V 122))
172
12
B
312L2 B
(4 (4+ 2B) 4)
VaL(a't1,2")) 22)11+
4(d + 4)02
1
L2dd + 6)3
1
+11
12² Land
B
B
B
202
20
3r
<- (1a - 1ha La) El VaL(a' x ||| + 1/a La 11
1
E| V 2 22))²
1/2
T2
B
2
31 Ln E 6(d+ 6(d+4))\VzL("" 1+1 21)112 + 4(d + 1)0 2
+ 2
+ 24
L2g0
+ +x2Lad
B
20²
202
S (Na na La)
E
V. C(a''' z z²)11 + 1/L
12
B
3n2 L.n
1/4
E
2
E
V C(a
1+1 zz'11 - | C(Caattt 1+| zt) VeCula 12,))
2
+ 2
3r Lz F.
6(d + V 2
4(d + 4)02
24 212 11,21 ) ) +12
B
18

---

Published as a conference paper at ICLR 2025
20
20
S-(16 - "la (a) EE VaL(a', 2 )12 + +1/a La
T2 +
B
31 L
11²
1/2
E
E 22 0+1 ,2')]|
[2 + +3)3
2
4
312 1'
4(d
+ 4)02
+
(6(d+4)llV V +11 ,2))11++
2/
Tige))
11 L.gd
B
(Na - Na La) E Va z' 1)1 + Na Lan n
20 9
20
1/2
10(d + 4) 4)) BlV ,z'|l
=-
1
B
2
3
1² L2(d 3)3 6(d + 4)
L2
+
1/2
3p
1/ Lgd + nº Lzd.
2
Inequality (g) holds because Ellu + b12 < 2B||a|| + 2 and Ella + b + c|2 S 3B|aa| + 3TB|b||2 + 3/|/
Incquality (h) holds because each gradient with respect to a; would have variance, then the lower bound of the variance need to multiply by n because the token length is n, and they are independent.
Let 1Ja
and Vh- 40(d + T)
we have
2La
EL(att1 (C(att1z((+1) L(a'-,a z') S
11a E V aaala' 22)222)))1=
The EVEV 22)01 ,z)11²=
no
g
4
2L
no? a
02
+
+ L L
+ Lndu2 + Lndp²
BLaB
(d+ 4)LBB
no?
E V C(a',z'))1
E| V2L(a'+1 ,,))1
Alia
160(d + 4)
2T
no72
of
Tz 12?
(d + 4) L 2Tzzd
BLaB B
Rearranging the formulation, we have
ATaa ElVaalaa' z')1
E VzC(a"* 1+1 ,z')1
160(d 4)
no
no² 2
(d + o L)
<L(a' EC(att1 21t2) +
+ 12d22122
2L
2L
2L B
Let Lmax max{_aa LLx (d + 4) Liin La (d + 4 1) we have
E| V zf 20) 160L
E| L(a 1+1 zz)ll
4Lmax
no² no 2
02
SL(a ('(a',z))BLLiC ++1 ",,,tt11""" ((+1) 2T
+ 3 Imm
2T
T
Limax and multiple sides with 160 we have Lmin
Let K
E VaLia' z' 12 V/ +1 221112
80nk0? 2
80nko2
160Rg²
< <160Lmax (C(a", z²) E ++1 21)) +
g
a
480L
max maxdux
12
B
B
Summing them up from t = t 1, we have
(IIVa + (20()))))))))
80nko?
80nno2 2
160ko
160 (C -BC(aat
1+1 221)
+
9
+
48 2 dp².
12
B
B
19

---

Published as a conference paper at ICLR 2025
160 (C(ae) C(aa,2% --)- -(
80nn0
80nxo α
160kx
T
+ 72
B
B
2
+
480T
max
dur²
Let I O
(nniog (Vinio B = o
max
nno
2
RO
)
.14
0
and T
2
Limax
Limaz O
we have
62
E (lIVal(a' 22)11 + VV ,²))),))))) = 0(2))
B MORE EXPERIMENTAL DETAILS
B. MANUAL TEMPLATES
Table 4: Input templates, and output label words used in RoBERTa-large. (S) represents the sen-
tences in the dataset. [MASK] represents the mask token.
Task
Dataset
Input Template
Output Label Words
acceptabilityy
CoLA
(S) correct? [MASK].
no, yes
NLI MNLI
(S2) entailment? [MASK]. no, maybe, yes
NLI
QNLI
(S2) entailment? [MASK]. no, yes
NLI
SNLI
(S2) entailment? [MASK] no, maybe, yes
NLI
WNLI
(52) entailment? [MASK]. no, yes
Table 5: Input templates, and output label words used in GPT2-XL and Llama3, (S) represents the sentences in the dataset.
Task
Dataset
Input Template
Output Label Words
acceptability
CoLA
(S) correct?
no, yes
NLI MNLI
(S2) entailment?
no, maybe, yes
NLI
QNLI
(S2)
entailment?
no, yes
NLI
SNLI
(Sg)
entailment?
no, maybe, yes
NLI
WNLI
(S2)
What is the relation?
no, yes
B.2 HYPERPARAMETERS
Table 6: Main hyperparameters used in our algorithms.
Hyperparameter
RoBERTa-large
GPT2-XI.
Llama3
query limit
4000
2000
1000
train batch size B
32
16
prompt length
{50, 20}
{50, 20)
{50, 20)
small positive constant μ
0.01
sample times 7, sample times Tz
20
10
temperature т
1.0
vocabulary size [
100
intrinsic dimensionality d
500
20

---

WNLI Published as a conference paper at ICLR 2025
C MORE ABLATION STUDIES
Effeet of Optimizing Initial Prompt. To evaluate the impact of optimizing the initial discrete prompt, we conducted ablation study where we compared the performance of ZO-PoG using optimized and randomly initialized initial prompts. Specifically, we investigated whether learning a task-specific distribution for selecting initial prompt tokens, instead of random selection, contributes to performance improvements across downstream tasks. To achieve this, we replaced sampling from Gumbel-Softme -Softmax with randomly selected tokens from the PTM vocabulary. Both setups employed the same continuous prompt optimization pipeline to isolate the impact of initial prompt optimiza- tion. The comparative results are reported in Table and Table 8. The comparative results indicate that models initialized with optimized discrete prompts consistently outperformed those using ran- dom initialization across all datasets.
Table Comparison of the average accuracy (%) between randomly sampled Pa and optimized the Po (ZO-PoG) on RoBERTa-Large in 16-shot (per class) setting.
Len
Method
CoLA
MNLI
QNLI
SNLI
random Po (seed = 14)
52.64
41.65
54.57
39.06
47.89
random Po (seed 42)
46.69
41.31
53.14
39.65
52.11
20
random Po (seed= = 81)
55.23
42.54
51.18
38.32
42.25
ZO-PoG (mean)
56.79
43.29
55.44
39.12
53.99
Table 8: Comparison of the average accuracy (%) between randomly sampled Po and optimized the Po (ZO-PoG) Llama3 in 16-shot (per class) setting.
Len
Method
CoLA
MNLI
QNLI
SNLI
WNLI
random Po (seed = 14)
50.43
34.55
53.21
37.41
56.34
random Po (seed = 42)
37.20
38.05
53.49
37.09
43.66
random Po (seed = 81)
43.53
35.39
53.41
36.16
59.15
ZO-PoG (first run)
41.71
41.16
53.14
39.44
56.34
ZO-PoG (second run)
49.28
39.08
55.67
35.01
57.75
20
ZO-PoG (third run)
53.40
36.26
58.91
40.33
63.38
random Po (max)
50.43
38.05
53,49
37.41
59.15
ZO-PoG (max)
53.40
41.16
58.91
40.33
63.38
random Po (mean)
43.72
36.00
53.37
36.89
53,05
ZO-PoG (mean)
48.13
38.84
55.90
38.26
59.15
In order to further verify the sensitivity of prompt initialization, we also tried the following different sampling strategies for Pu: 1) we sample the same token for each position of the fixed length prompt. 2) we uniformly randomly sample from the vocabulary (the same way as that sampled with random seeds). 3) We sample the continuous tokens of fixed length from the PTM vocabulary (this is the same way as implemented in BBT). We have also tried to directly sample Po in the embedding space as initialization through 4) Gaussian distribution (with the same mean and variance value for the sampling of random matrix A) and 5) uniform distribution within a fixed region (mean standard deviation of the sampling of random matrix A). 6) Orthogonal initialization: each token embedding in are orthogonal with each other. We report the accuracy with three different random seeds in Table
Effeet of Prompt Length. To investigate how the prompt length affects the performance of our proposed ZO-PoG framework, we experimented with prompt lengths of 10, 20, 30, 40, 50 tokens on the performance of Llama for QNLI dataset. The results in Table 10 show that the relationship be- tween prompt length and model performance non-linear. Short prompts may demonstrate limited representational capacity, leading suboptimal performance and longer prompts do not necessarily produce better result. The observed trends suggest that a moderately long prompt provides a good trade-off between capacity and learnability.
Effect of Optimization Strategy. To verify the necessity and effectiveness of alternating opti- mization, conducted an ablation study to compare three optimization strategies: Alternating
21

---

Published as a conference paper at ICLR 2025
Table 9: Accuracy (%) on RoBERTa--Lage in 16-shot (per class) setting with different initialization strategy.
CoLA
MNLI
QNLI
Seed
Initialization Strategy
81
14
81
prompt with same tokens
46.40
44.87
45.93
41.37
42.64
41.40
55.30
52.17
$2.70
Uniform sampling (Vocabulary) continuous token Vocabulary Gaussian sampling (embedding) Uniform sampling (embedding) Orthogonal initialization
46.69
55.23
41.31
42.54
54.57
53.14
51.18
45.54
49.19
42.57
43.29
36.18
43.68
51.38
54.37
52.00
50.24
49.28
53.4
38.31
38.34
37.54
52.04
52.44
53.96
48.03
43.53
54.75
38.67
42.22
35.59
52.39
50.52
51.33
42.57
37.87
53.98
35.84
40.89
39.68
53.54
52.02
52.83
SNLI
Seed
Initialization Strategy
14
42
81
14
42
81
prompt with tokens
36.99
40.06
38.59
42.25
42.25
38.03
Uniform sampling (Vocabulary)
39.06
39.65
38.32
47.89
52.11
42.25
continuous token in vocabulary
38.91
35.15
34.33
42.25
56.33
53.52
Gaussian sampling (embedding)
35.38
35.87
36.23
45.07
50.7
42.25
Uniform sampling (embedding)
38.84
34.22
33.58
52.11
54.93
Orthogonal initialization
37.38
34.00
35.01
54.93
45.07
54.93
Table 10: Ablation study of prompt lengths on Llama3 in 16-shot (per class) setting with the QNLI dataset.
Prompt length
10
20
30
40
50
80
Dataset
QNLI
53.17
55.90
56.13
55.83
55.60
54.73
optimization: The proposed strategy alternates between discrete and continuous prompt optimiza- tion across multiple iterations. 2) Single-round Optimization: This approach involves performing one round of soft-prompt optimization followed by one round zeroth-order optimization without alternating iterations. 3) Joint optimization: Both soft prompt and discrete probability distribution are optimized simultaneously using zeroth-order optimization. The results in Table 11 and 12 ver- ify the design of the alternating optimization strategy. By leveraging the strengths of both policy gradient and zeroth-order optimization iteratively, ZO-PoG achieves superior performance
The comparison of the average accuracy between one round ZO + one round Policy gradient and ZO-PoG on Llama3 on 16-shot (per class). The best results are underlined. Len represents the prompt length.
Table 11:
Len
Method
CoLA
MNLI
QNLI
SNLI
WNLI
round round
43.50
37.48
50.58
34.29
54.93
20
ZO-PoG (ours)
48.13
38,84
55.90
38.26
59.15
one round + one round
42.25
36.13
55.09
33.70
54.93
50
ZO-PoG (ours)
46.50
37.19
55.60
38.12
58.22
D EXPERIMENTAL RESULTS ON BLACK-BOX LARGE LANGUAGE MODELS
D. ADAPT ZO-PoG TO BLACK-BOX LARGE LANGUAGE MODELS
Modern black-box large language models (LLMs), such as GPT-4, offer powerful capabilities but restrict user access to their internal components, such as the embedding space. These models accept only discrete input tokens, which limits direct optimization in the continuous embedding space. Additionally, feedback from black-box APIs is available only through forward evaluations, further complicating optimization.
22

---

Published as a conference paper at ICLR 2025
The comparison of the average accuracy between joint optimization with zeroth-order optimization and ZO-PoG on RoBERTa-Large in 16-shot setting. The best results are underlined. Len represents the prompt length.
Table 12:
Len
Method
CoLA
MNLI
QNLI
SNLI
WNLI
joint optimization
52.33
41.00
54.19
35.47
46.95
20
ZO-PoG (ours)
56.79
43.29
55.44
39.12
53,99
joint optimization
53.05
41.92
55,56
35.74
46.00
50
70-PoG (ours)
56.18
42.53
55.33
41.99
56.34
Algorithm/ ZO-PoG with black box Large Language Models
Input: Parameters of categorical distributions 01,," Onn, temperature parameter т, black-box
large language models f. surrogate embedding model c(-), downstream dataset D, mini-bate size B, learning rates Tl Th and sample times /,, Iz.
1: Initialize a ... a., z
2: for in 0 to T - 1 do
3: Draw a mini- mini-batchhee S₁ = (X,, Yi} ] = fo,, ye M from
4: for & in 1 to 1 do
5: Sample 11.1) GS(af, T), spht ( GS(a
Рo,1 e( (k)
(k)
(1) e(Vij j.... 11
6:
7:
Layg Σ Ck=1 [ff(e (Az' ' Po, (k)
) X.),
8:
for i in 1 to do
9a. 1-1 E (C(f(e 'Azz + + X1), Yi) - Va. log P(t. (2) (at)
9:
10:
a [+1 a - naTa:
b Update discrete vocabulary distribution
11:
for / in to L do
12:
Sample ( (1)
GS 1+1 T), Int GS(a .T)
13:
Po.: c(t
(1))) ccV(1111 v(!!!
14:
Sample u, ~ Ia)
_ = L(f(e (A + +ni ()) + : X)),
15:
( = ((f(e-(((()) - pp ()) + Po. X,)
16:
9 (C - / u (!)
17:
1 Eta Oz.
(1)
18:
O2 T
19:
z' = z' - T'29u'
Update continuous prompt
Return:
of .' TT, IT.
To address these challenges, we propose an enhanced version of ZO-PoG tailored for discrete-token- only LLMs. The new algorithm leverages a surrogate embedding model e(-) y - 2 approx- imate the black-box box LLM's embedding space and enables efficient optimization using projection function that maps continuous embeddings back to discrete tokens. We could adopt pre-trained open-sourced as the surrogate embedding model (e.g., RoBERTa, GPT-2). This approach ensures the method can adapt to real-world L.LM scenarios without compromising performance.
The proposed algorithm consists of a preprocessing step for surrogate embedding computation and an alternating optimization framework that integrates both discrete and continuous prompt optimiza- tion. Let Vg denote the full vocabulary of the surrogate embedding model. Before the training of the algorithm, we will first need to compute the embedding {e(v)]e E V. for all tokens in V.- Then could construct lookup table &, {(v, e(v)) During training, we define a projection function e (.) RR + V that map a learned embedding p E T back to the nearest the most similar token in V.- In this work, choose the cosine
V. for efficient access during training.
23

---

Published as a conference paper at ICLR 2025
similarity as the projection function, i.e.
(p) = arg max (p))(u)) VE |
(22)
which is a token-wise projection operation.
D.2 EXPERIMENTS ON MATH TASK
To provide a detailed evaluation of ZO-PoG's effectiveness on tasks requiring advanced reasoning. we conducted experiments on challenging mathematical problem-solving dataset GSM8K (Cobbe et al., 2021) in the 64-shot setting. In addition to comparisons with prompt learning baselines, we have also included comparisons with instruction optimization methods InstructZero (Chen et al., 2024) and TRIPLE (Shi et al., 2024). For prompt learning methods, the prompt prefix is optimized, which is concatenated with manual prompt with Chain-of-Thoupe (CoT), i.e., Let's think step by step. We choose open-sourced GPT-2 as our surrogate embedding model. For instruction opti- mization baselines, the entire instruction is optimized. The input templates are shown in Table 13. In order to enhance exploration, we also adopt the single-point gradient estimation for ZO-PoG in this task, i.e., replacing Line 17 of Algorithm 2 with 92'
(!) - //u u The experimental results on the GPT-4 are presented Table 14. Our method achieves the most significant performance im- provement compared Manual Prompt with the Chain of Thought in the context of prompt learning baselines and also demonstrates superior performance over instruction optimization methods.
Table 13: Input templates used in GSM8K dataset. (Q) represents the questions in the dataset, \n represents the line break.
Method
Input Template
Prompt Learning
[Prompt] Let's think step by step. \n Question: (Q) |n Answer:
Instruction Optimization
[Instruction] \n Question: (Q) \n Answer:
Table 14: Averaged scores of baselines and Zo-PoG on the GSM8K dataset using GPT-4.
Method
MP CoT
BBT
BDPL
TRIPLE
InstructZero
7O-PoG (ours)
Dataset
GSM8K
86.48
78.22
85.77
70,26
89.16
89.94
24


Reviews:
[{"review_id": "33Wtsm3pqE", "reviewer": "Reviewer_sg5f", "summary": "This paper introduces a framework for prompt optimization in LLMs that combines discrete and continuous prompt tuning. This framework alternates between optimizing discrete prompts through policy gradient methods and continuous prompts via zeroth-order gradient optimization. The authors establish the sub-linear convergence of ZO-PoG under assumptions of Smoothness, Bounded Variance, Bounded Loss, and Lower-Bounded Parameters. This framework was evaluated on five datasets (CoLA, MNLI, QNLI, SNLI, WNLI) from the GLUE benchmark using RoBERTa-large, GPT2-XL, and Llama3 as backbone models. The code has been provided.", "questions": "Given that the loss function of LLMs is unbounded, I'm wondering if Assumption 3 can be relaxed in theoretical analysis to better reflect practical scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. ZO-PoG combines discrete and continuous prompt optimization. Discrete prompts are refined through policy gradient in the parameter space, and continuous prompts are adjusted using zeroth-order gradients. This dual optimization approach enhances the adaptability and efficiency of prompt learning.\n2. The authors provide a theoretical analysis showing ZO-PoG’s sub-linear convergence, validating its efficiency. Sub-linear convergence means that ZO-PoG requires fewer iterations to achieve satisfactory performance, making it a computationally efficient solution for black-box prompt learning.\n3. The framework is tested on five datasets and demonstrates improvements over other black-box prompt learning methods. Results from the ablation study confirm that each component positively contributes to ZO-PoG’s overall performance, further validating its design choices.", "weaknesses": "1. Assumption 3 requires the loss function to be bounded. Does this assumption hold for the loss function in LLM fine-tuning? If not, could this assumption be relaxed in the theoretical analysis presented in this paper?\n2. This paper evaluates ZO-PoG on five datasets from the GLUE benchmark, while methods like BBT and SSPT report results on additional datasets. It would be valuable if the authors could provide performance comparisons of ZO-PoG on a broader range of datasets to offer a more comprehensive evaluation.\n3. I recommend that the authors discuss the query complexity of zeroth-order optimization following the theorem, as this would provide a clearer understanding of the computational efficiency of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kTcUbzL9td", "reviewer": "Reviewer_yHre", "summary": "This paper proposes a new approach to black-box prompt learning by jointly optimizing both discrete text prompts and continuous embeddings, with a convergence analysis. Experiments on five commonly used datasets demonstrate its superiority.", "questions": "See the weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and easy to follow\n- The proposed method is technically sound \n- The experimental results show the proposed method has a significant performance improvement.", "weaknesses": "- My main concern lies in that why the authors did not employ the true black-box models, such as GPT-4 and Claude-3.5, as their backbone models. Since the proposed method aims at black-box prompt learning, it would be more convincing to show how it works with these leading black-box API.  Is it still necessary to optimize the prompts for such powerful models in terms of cost versus benefit? It would be much better to include these results. \n- Why choosing a random matrix $\\textbf{A}$? Is it good enough? If not, why not optimizing it or how to choose a better one?\n- How does the performance change as the prompt length increases? Better show more results on different lengths.\n- Does this method works only on natural language tasks? How does it behave on math or code tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "azNzk1Wfro", "reviewer": "Reviewer_PhZG", "summary": "In the paper \"Collaborative Discrete-Continous Black-Box Prompting Learning For Language Model,\" the authors propose a new method for learning prompts, considering LLMs as a black box model. Under this assumption, the algorithms only have access to token and output (thus possible computation of a loss); therefore, it is impossible to access gradient and modify embeddings contrary to classical prefix tuning.\n\n\n\nSimilar to the CMA-ES approach, the authors propose to learn prompt embeddings in a low dimension and then project it to the input dimension of the model. The input embeddings correspond to the addition of the continuous prompt and representation of tokens of the vocabulary  $Az + p_0$($p_0$ sampled from the vocabulary). Contrary to previous approaches, authors propose to learn $z$ using the zeroth-order approximation.\nIn addition, the authors propose to learn the distribution of discrete prompt tokens over the vocabulary (sampling p_0). The distribution is approximated using the gamble softmax distribution, where parameters are estimated using a policy gradient with a baseline (REINFORCE).\n\nThe authors propose to analyze the convergence of the algorithm proposed in section 4. The approach is then evaluated on different GLUE tasks, and performances are compared with the manual prompt approach (designing the tokens of the prompts manually) and other black box prompting approaches. Three LLMs are compared: RoBERTa large, GPT2-XL, and Llama3. \nAn ablation study is set up, removing the Gumbel softmax (using a policy gradient approach from [1] instead), removing the discrete prompts optimization and the continuous prompt optimization.   \n\nFor all experiments, the settings were tested at 20 and 50 prompt lengths.\n\nThe contributions are the following : \n * A new algorithm for black box prompting.\n * A new approach to select (sample) $p_0$ for discrete prompts  using  Gumbel softmax (contrary to the previous approach, choosing random token vocabulary)\n* State-of-the-art results in black box prompting.\n* Convergence analysis of the proposed method", "questions": "* In the 5.1 section, the authors observed a significant difference depending on the model when compared to the manual prompt (first result line of the tables). Particularly, GPT2-XL and mostly Llama3 have, for most datasets, lower improvement using black box rather than manual prompt, particularly on the CoLA dataset. The authors state that it is due to the nature of the CoLA corpus. Does the fact that Roberta is an encoder only and the others are decoders (different pretraining tasks) have a role in this difference? The latter are thus more likely to get better performances on grammatically correct prompts.\n* What is the prompt size for manual prompts? How did you select manual prompts?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The approach and related works are well-described and motivated\n* New algorithm  for black box prompting combining previous ideas with new one proposed\n* Proposal of optimizing discrete prompt using gumbel softmax\n* State-of-the-art result for approach in a black box setting\n* Relevant Ablation study removing some part of the algorithm to judge the effect of the different component\n * Justification of the algorithm", "weaknesses": "* Limited dataset and configuration (20 and 50-length prompt) evaluation\n* The related works section could have been extended", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qAsBL0RXgB", "reviewer": "Reviewer_hHWU", "summary": "This paper studies the problem of soft prompt optimization for pre-trained language models. In some previous works, random initialization is applied to a soft prompt which will be added to a low-dimensional vector's random projection. The authors proposed to optimize this random initialization as they believe this may lead to suboptimal performance. Technically, an alternating optimization method is proposed where the random initialization is optimized via a policy gradient method over token distributions, and the low-dimensional vector $z$ is optimized through zeroth-order optimization. Detailed convergence analysis is provided and empirical results are shown to justify the effectiveness of the proposed method.", "questions": "See the questions above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This work improves previous literature with special consideration on optimizing the initialization of soft prompts. The mixing of discrete prompt and soft prompt optimization is also innovative.\n- Theoretical analysis of convergence is given to justify the design and principle of the proposed alternating optimization method.", "weaknesses": "- The authors stated their focus is on PTMs that can be interacted solely via APIs. However, most commercial models nowadays do not open embedding access,  which makes the soft prompt tuning method not practical and thus less significant. Especially in the experimental section, only a few white-box models are included and no black-box models are not considered.\n- The recent literature on prompt optimization is not included. For example, InstructZero [1], ZOPO [2], and TRIPLE [3] were proposed to use a derivative-free method (Bayesian optimization or zeroth-order method) to optimize soft or discrete prompts. ZOPO and TRIPLE also project the discrete prompts into an embedding space to conduct optimization. Those methods need to be at least discussed and possibly compared.\n\n[1] Chen, L., Chen, J., Goldstein, T., Huang, H., & Zhou, T. (2023). Instructzero: Efficient instruction optimization for black-box large language models.\n\n[2] Hu, W., Shu, Y., Yu, Z., Wu, Z., Lin, X., Dai, Z., ... & Low, B. K. H. (2024). Localized zeroth-order prompt optimization. \n\n[3] Shi, C., Yang, K., Yang, J., & Shen, C. (2024). Best arm identification for prompt learning under a limited budget. \n\n\n- A strong motivation for this work is to improve the random initialization which is claimed to lead to a suboptimal performance. However, no empirical or theoretical justification is given. I suggest the authors conduct some ablation studies to demonstrate the sensitivity of prompt optimization performances given random initialization (number of random initialization or different random seeds). \n- Why alternating optimization is necessary? Could the author demonstrate the difference between <just using one round of soft-prompt optimization + one round of zeroth-order optimization> and <the alternating optimization strategy>? I also feel a joint optimization is feasible here, where the low-dimensional $z$ and the encoded probability of the initialization prompt can be jointly optimized by zeroth-order optimization. These points should be helpful for justifying why the alternating optimization is designed.\n- Assumption 1 normally does not hold in practice, especially when the loss function only produces discrete values for some NLP tasks, e.g., accuracy. I understand this assumption has to be made for the theory to work, but at least some level of justification should be given.\n- Limited empirical results.\n    - More representative baselines proposed in recent years should be considered.\n    - Only one benchmark (i.e., GLUE) is considered. I expect to see comparisons on other more practical benchmarks, such as mathematical reasoning task GSM8K and some text generation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)