Title: Localized Zeroth-Order Prompt Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Prompt Optimization

Large Language Models

LLM

Instruction Optimization
Abstract: The efficacy of large language models (LLMs) in understanding and generating natural language has aroused a wide interest in developing prompt-based methods to harness the power of black-box LLMs. Existing methodologies usually prioritize a global optimization for finding the global optimum, which however will perform poorly in certain tasks. This thus motivates us to re-think the necessity of finding a global optimum in prompt optimization. To answer this, we conduct a thorough empirical study on prompt optimization and draw two major insights. Contrasting with the rarity of global optimum, local optima are usually prevalent and well-performed, which can be more worthwhile for efficient prompt optimization (**Insight I**). The choice of the input domain, covering both the generation and the representation of prompts, affects the identification of well-performing local optima (**Insight II**). Inspired by these insights, we propose a novel algorithm, namely localized zeroth-order prompt optimization (ZOPO), which incorporates a Neural Tangent Kernel-based derived Gaussian process into standard zeroth-order optimization for an efficient search of well-performing local optima in prompt optimization. Remarkably, ZOPO outperforms existing baselines in terms of both the optimization performance and the query efficiency, which we demonstrate through extensive experiments.

Localized Zeroth-Order Prompt Optimization

Abstract

The efficacy of large language models (LLMs) in understanding and generating natural language has aroused a wide interest in developing prompt-based methods to harness the power of black-box LLMs. Existing methodologies usually prioritize a global optimization for finding the global optimum, which however will perform poorly in certain tasks under budget constraints. This thus motivates us to re-think the necessity of finding a global optimum in prompt optimization. To answer this, we conduct a thorough empirical study on prompt optimization and draw two major insights. Contrasting with the rarity of global optimum, local optima are usually prevalent and well-performed, which can be more worthwhile for efficient prompt optimization ( Insight I ). The choice of the input domain, including both the generation and the representation of prompts, affects the identification of well-performing local optima ( Insight II ). Inspired by these insights, we propose a novel algorithm, namely localized zeroth-order prompt optimization (ZOPO), which incorporates a Neural Tangent Kernel-based derived Gaussian process into standard zeroth-order optimization for an efficient search of well-performing local optima in prompt optimization. Remarkably, ZOPO outperforms existing baselines in terms of both the optimization performance and the query efficiency, which we demonstrate through extensive experiments. Our implementation is available at https://github.com/allen4747/ZOPO .

1 Introduction

Large language models (LLMs) have demonstrated remarkable capabilities for understanding and generating natural languages [27, 41, 45]. Thanks to the instruction-following abilities of LLMs [28], prompting-adding crafted, discrete prompts, or namely natural language text, to the input emerges as an effective and lightweight approach to direct LLMs to generate specific, desired responses [20, 21]. Such an approach is of particular interest when users interact with state-of-the-art LLMs like ChatGPT [25] and GPT-4 [24], which can only be accessed through black-box APIs (i.e., the interface of black-box LLMs only accepts discrete texts as input). So, prompt optimization becomes a critical effort in pursuing the optimal performance of black-box LLMs on downstream tasks.

Although human knowledge may subjectively guide prompt designs [21, 32], this process is commonly time-intensive and its results are not always desirable in practice. To mitigate such human efforts

and achieve better performance in optimizing crafted prompts, random sampling [48], Bayesian optimization [3, 18], and evolutionary algorithms [10] have been proposed to generate and select well-performing prompts automatically. However, most of these existing strategies prioritize global optimization , dedicating substantial portions of the query budget to explore the entire search space for the global optima and consequently making it query-inefficient in practice. Meanwhile, these strategies typically implement their prompt optimization across various input domains (i.e., natural texts [10, 48] or hidden embeddings [3, 18]), resulting in diverse performance outcomes in practice. These results consequently inspire us to re-think the questions about the necessity of finding a global optimum and the essence of the input domain for efficient and effective prompt optimization.

To answer these questions, we provide a thorough empirical study on prompt optimization. Firstly, we visualize the performances of some randomly sampled prompt candidates on various tasks to show that, in contrast to the scarcity of global optima, local optima are commonly prevalent and perform reasonably well, making them more valuable for query-efficient prompt optimization ( Insight I in our Sec. 3.1). Secondly, we visualize the estimated accuracy distributions for a number of prompt candidates and the corresponding function surfaces using various embeddings as their representation. The results demonstrate that the selection of the input domain, including both the generation and representation of prompt candidates, will influence the identification of high-performing prompts, especially those local optimal ones ( Insight II in our Sec. 3.2). These insights consequently highlight the importance of local optima and input domain for efficient and effective prompt optimization.

Inspired by these insights, we novelly propose the Localized Zeroth -Order Prompt Optimization ( ZOPO ) algorithm for a considerably improved prompt optimization as evidenced by Fig. 1. Motivated by Insight II, we first propose a general domain transformation that utilizes LLMs for prompt generation and existing embedding models for transforming these generated prompts into their corresponding hidden representations, which thereby enjoys not only the remarkable generation ability from any type of LLMs (white/black-box) but also the impressive representation ability from existing embedding models for our prompt optimization (Sec. 4.1). Inspired by Insight I, we then leverage a cutting-edge zeroth-order optimization (ZOO) method enhanced by a derived Gaussian process for efficient gradient estimation [36] to underpin our localized prompt optimization, which goes one step further by incorporating the Neural Tangent Kernel (NTK) [12] to handle the complex and highdimensional prompt optimization tasks (Sec. 4.2). Lastly, we present an uncertainty-informed local exploration method designed to improve the gradient estimation in our derived NTK-GP framework, thereby augmenting the practical performance of the ZOPO algorithm (Sec. 4.3).

To summarize, the contributions of our work include:

To the best of our knowledge, we are the first to conduct a thorough empirical study in prompt optimization to underscore the value of local optima and the essence of input domain for efficient and effective prompt optimization (Sec. 3).

Drawing on the insights gained from our empirical study, we design the ZOPO algorithm (Sec. 4) which outperforms existing baselines in optimization performance and query efficiency.

We conduct extensive studies to confirm the efficacy of our algorithmic framework and elucidate the underlying principles or insights of our ZOPO algorithm (Sec. 5).

2 Problem Setup

Given an NLP task that is characterized by a data distribution D and a black-box LLM f ( · ) , e.g., ChatGPT [25], discrete prompt optimization aims to generate a piece of human-readable text, namely the prompt v , which will then be applied to the black-box LLM f ( · ) along with a test input x such that the queried LLM output f ([ v ; x ]) is able to correctly predict the ground-truth label y for each ( x, y ) ∼ D . This problem is then commonly framed as a black-box maximization problem over the discrete language input domain Ω [3, 18]:

where R ( f ([ v ; x ]) , y ) is applied to measure the alignment between the LLM output f ([ v ; x ]) and the groundtruth y , and D V is the validation set sampled from D . Note that the performance of the optimal instruction found on D V (i.e., arg max v F ( v ) ) will be evaluated on a held-out test set D T .

0

10

20

¿

0.25

0.50

0.75

½

(

¿

)

APE

InstructZero

INSTINCT

EvoPrompt

PB

OPRO

ZOPO (ours)

taxonomy_animal

cause_and_effect

informal_to_formal

0.0

0.5

1.0

3 Empirical Study on Prompt Optimization

3.1 Local Optima vs. Global Optimum

In prompt optimization, methods like [3, 18] are generally more effective than the others [10, 48], which is usually contributed to their usage of Bayesian optimization, a popular global optimization strategy, that is able to find the global optimum in low-dimensional problems [22]. However, these methods will perform poorly in certain prompt optimization tasks, e.g., cause_and_effect and informal_to_formal from [11], indicating that they typically fail to find the global optimum in these tasks given a limited query budget. This is likely because substantial portions of the budget are wasted in these methods to explore the entire search space for the global optimum. However, is it really necessary to find the global optimum in query-efficient prompt optimization?

To answer this question, we have employed a 3-dimensional scatter plot to visualize the performance (differentiated by colors) for 300 randomly sampled prompt candidates on various tasks, whose prompt embeddings (i.e., the last token embedding as in [18]) are reduced by t-distributed stochastic neighbor embedding (t-SNE) (see more details in our Appx. D.1.1). The results are in Fig. 2 which shows that the global optimum (i.e., the points achieving the highest accuracy) is consistently rare for a range of prompt optimization tasks, making it extremely challenging to achieve this global optimum in practice. In contrast, prompt optimization often features a number of local optima (e.g., the points achieving accuracy higher than 80% in taxonomy_animal of Fig. 2). Importantly, these local optima commonly enjoy relatively good performances, suggesting that local optima shall be more worthwhile to obtain in prompt optimization, especially for the scenarios of limited query budgets, as summarized below.

Insight I

In contrast with the rarity of global optimum, local optima are usually prevalent and well-performed, which is more worthwhile for query-efficient prompt optimization.

3.2 Essence of Input Domain

Besides, existing works [3, 10, 18] typically implement their prompt optimization across various input domains, leading to a wide range of performances in practice. These results thus inspire us to ask: How essential is the input domain for finding well-performing prompts, particularly the local optimal ones? Thoroughly exploring this question is fundamental for the design of a well-performing prompt optimization algorithm.

To answer this, we first visualize the accuracy distributions of 300 prompt candidates that are randomly generated by Vicuna-13B and ChatGPT for various tasks to study the essence of prompt generation in Fig. 3 (more details in Appx. D.1.2). Fig. 3 reveals that the prompt candidates produced by ChatGPT (a black-box model) generally exhibit better performance than those produced by Vicuna-13B (a white-box model), which has been widely applied in [3, 18] for prompt optimization. Importantly, ChatGPT demonstrates a greater likelihood of generating locally optimal prompts (e.g., the ones of accuracy higher than 0.8 in taxonomy_animal of Fig. 3). These results indicate that the ability to

0.0

0.5

1.0

0.0

0.5

1.0

1.5

2.0

Probability Density

taxonomy_animal

Vicuna-13B

ChatGPT

0.0

0.5

1.0

0

2

4

cause_and_effect

Vicuna-13B

ChatGPT

0.0

0.2

0.4

0

2

4

6

8

informal_to_formal

Vicuna-13B

ChatGPT

Validation Accuracy

Last Token

taxonomy_animal

SBERT

cause_and_effect

informal_to_formal

0.0

0.2

0.4

0.6

0.8

1.0

generate well-performing local optima in prompt optimization usually varies for different models. So, the selection of the prompt generation model is crucial for finding well-performing optima.

We then investigate the function surface (i.e., accuracy landscape) using two different embeddings for prompt candidates in Fig. 4 (more details in Appx. D.1.2) where the embeddings are mapped into a 2-dimensional domain using the t-SNE for better visualization. Interestingly, Fig. 4 unveils that different embeddings will convey a varying number of well-performing local optima in practice. Particularly, the last token embedding is usually able to produce a larger number of well-performing local optima than the SBERT (i.e., a popular sentence embedding transformer [31]) embedding, making it easier to enjoy a good prompt optimization performance on this domain, as validated in Tab. 8. This therefore implies that the choice of the prompt embedding model is also essential for the finding of well-performing optima. In all, we conclude our aforementioned insights as below.

Insight II

The choice of the input domain, including both the generation and the representation of prompt candidates, affects the identification of well-performing local optima.

4 The ZOPO Algorithm

Given the insights established in our Sec. 3, we then propose our Localized Zeroth -Order Prompt Optimization ( ZOPO ) algorithm (Algo. 1) for a better-performing as well as more query-efficient prompt optimization. Specifically, following our Insight II, we first develop a more general transformation for the input domain of prompt optimization (Sec. 4.1), which can enjoy both the remarkable generation ability from any type of LLMs (white/black-box) and the impressive representation ability from many NLP models. Subsequent to this transformation, inspired by our Insight I, we propose to use zeroth-order optimization (ZOO) with a derived NTK Gaussian process inspired from [36] to find well-performing local optima (Sec. 4.2). Lastly, we introduce an uncertainty-informed local exploration technique to refine the gradient estimation in our derived NTK Gaussian process, aiming to enhance the performance of our ZOPO algorithm in practice (Sec. 4.3).

4.1 A More General Input Domain Transformation

As introduced in our Sec. 3.2, the choice of input domain (including the generation and representation of candidates) significantly influences the ultimate performance in prompt optimization: Black-box LLMs (e.g., ChatGPT) typically enjoy an advanced generation ability and different embedding models (e.g., SBERT) have varying representative capacity for prompt optimization. This naturally inspires us to develop an improved domain transformation that can utilize not only the remarkable generation ability from white/black-box LLMs but also the impressive representation ability from certain NLP models for our prompt optimization. To achieve this, we propose to make use of the prompt v ∈ Ω generated from a LLM g ( · ) and subsequently transform it into a continuous hidden representation z ∈ Z ⊂ R d by other sentence embedding model h ( · ) for the optimization, i.e., v = h -1 ( z ) , where (1) can then be re-framed as

Algorithm 1 The ZOPO Algorithm 1: Input: the prompt generation model g ( · ) , the prompt embedding model h ( · ) , size of prompt candidates m , iteration number T , prompt candidate set V = ∅ , prompt embedding set Z = ∅ 2: repeat 3: v ← g ([ D demo ]) 4: z ← h ( v ) 5: if v / ∈ V then V ← V ⋃ { v } , Z ← Z ⋃ { z } 6: until |V| = m 7: for t = 1 to T do 8: if 1 A t ( z t ) = 1 then do uncertainty-informed local exploration in Sec. 4.3 9: z t +1 = P Z ( z t + η t µ t ( z t )) 10: Query z t +1 to yield ˜ F ( z t +1 ) 11: end for 12: z ∗ ← arg max z 1: T ˜ F ( z ) -1 ∗

13: Return h ( z )

Of note, our input domain transformation and (2) enjoy a number of major advantages compared with previous works: (a) Different from the direct optimization over the discrete and complex language space v ∈ Ω in [10] where optimization algorithms in the numerical domain can hardly be applied, our transformed input domain leads to a dense numerical space of lower dimension and therefore allows the usage of query-efficient optimization algorithms for (2) (e.g., our Algo. 1). (b) Different from the potential many-to-one mapping in the previous works [3, 18], i.e., the same discrete prompt v may be generated by various continuous soft prompts s , we develop a one-to-one mapping where one prompt generally has a unique hidden representation z , which thus can help eliminate the redundant queries during optimization and ultimately lead to more query-efficient prompt optimization. (c) Our domain transformation with an independent generation and representation process is capable of enjoying the remarkable generation ability from any type of LLMs (white/black-box) and the impressive representation ability from many embedding models whereas previous works are highly restricted to the LLMs, thus leading to a wider application.

Practical Implementations. Before the start of the optimization on (2), we usually generate numerous prompt candidates V = { v } and their corresponding representations Z = { z } (line 2-6 of Algo. 1), where Z can be produced by an embedding model h ( · ) . We store ( z, v ) in key-value pairs for constructing the one-to-one inverse mapping h -1 ( · ) . Two practical methods are considered here for prompt generation: (a) Feeding randomly sampled soft prompts s ∈ R d and a few demonstrations D demo into a white-box LLM g ( · ) . (b) Sampling the output distribution of a black-box LLM g ( · ) given a generation template filled with D demo. Specifically, if we consider the generation method in (a) , z can be chosen as the last token embedding from g ( · ) [18] or the soft prompt s [3] when generating v . Here h ( · ) then represents a mapping function from v to z .

4.2 Local Optimization with Derived NTK-GP

As local optima are more prevalent than global optimum and can exhibit compelling performance for prompt optimization tasks (Sec. 3.1), we propose to apply zeroth-order optimization (ZOO), particularly gradient descent using estimated gradients, for a well-performing local prompt optimization on our transformed input domain Z in Sec. 4.1. Unfortunately, existing ZOO algorithms are typically query-inefficient as many additional queries are required for gradient estimation in every gradient descent update [9, 23]. In light of this, we resort to the most recent ZoRD algorithm [36] where a localized surrogate model will be applied for query-efficient gradient estimations.

According to [36], given a well-specified kernel function k ( · , · ) such that the function ˜ F is sampled from a Gaussian process ˜ F ∼ GP (0 , k ( · , · )) or alternatively min G ∼GP (0 ,k ( · , · )) max z ∈Z | ˜ F ( z ) -G ( z ) | = 0 and the observed value r of function ˜ F follows the Gaussian noise N (0 , σ 2 ) , then conditioned on the history of function queries D t ≜ { ( z τ , r τ ) } t τ =1 of size t , ∇ ˜ F follows a derived

Gaussian Process GP ( µ ( · ) , Σ( · , · )) , i.e.,

in which the mean function µ t ( · ) and the covariance function Σ 2 t ( · , · ) are defined as

Here, k t ( z ) ⊤ ≜ [ ∂ z k ( z, z τ )] t τ =1 is a d × t -dimensional matrix, K t ≜ [ k ( z τ , k ( z τ ′ )] t τ,τ ′ =1 is a t × t -dimensional matrix, r ⊤ t ≜ [ r τ ] t τ =1 is a t -dimensional column vector, and k ′′ ( z, z ′ ) ≜ ∂ z ∂ z ′ k ( z, z ′ ) is a d × d -dimensional matrix. As a result, µ t ( z ) can be applied to estimate the gradient of the black-box function ˜ F at input z .

Of note, the underlying black-box function ˜ F here is highly related to deep neural networks (DNN), more specifically transformers. It naturally inspires us to apply the Neural Tangent Kernel (NTK) [12] theory for a better approach to the aforementioned assumption of a well-specified kernel function k ( · , · ) . This is because it has been widely proven that NTK is capable of well characterizing the predictions of neural networks [2, 16, 34, 35] and therefore should be a better-specified kernel in the setting of prompt optimization than the simple kernel (i.e., Matérn kernel) applied in ZoRD [36]. Specifically, given a neural network ϕ ( θ, z ) parameterized by θ ∈ R p , we employ the following empirical NTK as the kernel in (3) and (4):

where θ 0 is the initialized parameter of neural network ϕ . By incorporating (5) into (4), we realize the derived NTK-GP for the gradient estimation in our prompt optimization.

Based on this derived NTK-GP, we finally apply standard first-order optimization (e.g., stochastic gradient descent) with projected gradients for our local prompt optimization. Specifically, in every iteration t of our Algo. 1, the next promising prompt candidate will be selected via:

where P Z ( z ) ≜ arg min z ′ ∈Z ∥ z -z ′ ∥ is the projection function that projects the updated z ∈ R d into domain Z and η t is learning rate.

Practical Implementations. Following the localized modeling principle, only the neighbors of z in the query history D t are used to calculate the gradient µ t ( z ) . As we do not know the exact DNN for the underlying black-box function ˜ F , we propose to approximate it using a small DNN, which can work well thanks to the theoretically guaranteed universal approximation ability of DNNs [14, 33]. Our experiments in Sec. 5.3 will further validate the effectiveness of this implementation.

4.3 Uncertainty-Informed Local Exploration

Though the derived NTK-GP allows us to estimate the gradient at any z ∈ Z according to [36], we introduce the following Prop. 1 to demonstrate that the error in gradient estimation at a specific input z ∈ Z implies considerable variability, which is strongly correlated with the number of historical queries that are effectively relevant for the gradient estimation at the specific input z ∈ Z . This insight, in turn, motivates the creation of our uncertainty-informed local exploration approach, as opposed to the adoption of the virtual update mechanism described in [36] for our prompt optimization strategy.

Proposition 1. Assume k ( z, z ′ ) ≤ α and ∥ k ′′ ( z, z ) ∥ ≤ κ for any z, z ′ ∈ Z . Let δ ∈ (0 , 1) and N z,β ≜ { z ′ ∈ { z τ } t τ =1 | ∥ ∂ z k ( z ′ , z ) ∥ 2 ≥ β } for given input z ∈ Z , the following holds with a probability of at least 1 -δ ,

where ω = d +2( √ d +1)ln(1 /δ ) and Σ 2 t ( z ) ≜ Σ 2 t ( z, z ) .

The proof is given in Appx. A. Here, N z,β denotes a set of historical input queries that are effectively relevant for the gradient estimation at z where β can be regarded as a measure of effective relevance. Prop. 1 shows that the gradient estimation error of (3) at a specific input z ∈ Z is bounded by the norm of covariance matrix Σ 2 t ( z ) , which is related to the query set N z,β of effective relevance. Specifically, the gradient estimation error at different z varies if the effective relevance β and the number of relevant queries | N z,β | varies with z . When β or | N z,β | becomes small during ZOO, the gradient estimation error is likely increased, which will lead to poor performance in practice. This likely will happen in prompt optimization especially considering the sparsity of prompt candidates w.r.t. the continuous domain R d . That is, both the effective relevance β and the number of relevant queries | N z,β | can be small due to this sparsity. As a consequence, additional input queries should be conducted to increase both β and | N z,β | for a better-performing prompt optimization.

To this end, we propose an uncertainty-informed local exploration method that utilizes additional input queries from local searches to reduce predictive uncertainty and hence the gradient estimation error in derived NTK-GP according to Prop. 1. Specifically, we propose the local exploration condition informed by the local trajectory:

where A t = { z t | σ ( z t -i ) ≥ λ, ∀ i ∈ [0 , ξ ] } is the condition that incorporates uncertainties and λ, ξ are the thresholds. If this condition is met (i.e., 1 A t ( z t ) = 1 ), we will query the neighbors of z t in the local region to update our derived NTK-GP, thus improving its gradient estimation.

Practical Implementations. If we define the set of the n nearest neighbors of z t as N t ⊆ Z s.t. |N t | = n and ∀ a ∈ Z \ N t , ∥ a -z t ∥ ≥ max b ∈N t ∥ b -z t ∥ , we propose to query each z ∈ N t in the local region, whenever 1 A t ( z t ) = 1 .

5 Experiments

In this section, we perform prompt optimization for ChatGPT (i.e., the black-box LLM f ( · ) ) with a limited query budget of 165. We evaluate the performance of ZOPO against several strong baseline methods, including APE [48], InstructZero [3], INSTINCT [18], EvoPrompt [10], PromptBreeder (PB) [8], and OPRO [46], on 30 instruction induction tasks [11], 3 arithmetic reasoning tasks [4, 19, 29], and the GLUE benchmark [42]. The task-specific prompt is optimized for each task independently. We use the performance profile [7], defined in Appx. C.1, as the overall evaluation metric that measures the frequency (i.e., ρ ( τ ) ) of a method within some distance (i.e., τ ) from the highest accuracy achieved by any method. We defer more experimental details to Appx. C.

5.1 Instruction Induction

Instruction induction tasks are commonly used to investigate the prompt optimization performance by assessing LLM's zero-shot in-context learning ability in previous works [3, 18, 48]. Although our ZOPO is a general prompt optimization method given any prompt generation strategy, here we follow the same setting of prompt generation from INSTINCT and InstructZero, only for fair comparison . We also adopt the last token embedding from Vicuna-13B as the prompt embedding (same as INSTINCT). Here Vicuna-13B is used to generate task-specific prompts by feeding random soft prompts. More experimental details are deferred to Appx. C.3.

Superior performance of ZOPO . For better distinguishability, we follow the experimental setting from Lin et al. [18] to display the results on 20 challenging tasks reported in Tab. 1, where ZOPO significantly outperforms all baseline methods. Particularly, our ZOPO performs the best in 14 out of the 20 tasks presented, while achieving the best performance profile across different τ (see Fig. 1) compared with all baseline methods. For more results on all 30 tasks, refer to Tab. 3 in Appx. D.2, where the ZOPO consistently outperforms existing methods.

Connecting ChatGPT with ZOPO . With our proposed domain transformation, we empirically demonstrate that ZOPO is capable of performing numerical optimization on ChatGPT-generated prompts. Specifically, we use the same generation method as in APE [48] to generate task-specific prompts (i.e., V ) from ChatGPT, and use a popular embedding model SBERT to provide the corresponding sentence embeddings (i.e., Z ) for V . Then we apply ZOPO to perform optimization over the given V and Z , which we name ZOPO GPT. The result of ZOPO GPT compared against other baselines is shown in Tab. 1, with the corresponding performance profile shown in Fig. 9 in App. D.2. Fig. 9 demonstrates that ZOPO GPT significantly outperforms other baselines, achieving the best performance in 10 out of the 20 tasks as shown in Tab. 1. Specifically, ZOPO GPT achieves significantly higher accuracy on some challenging tasks such as second_word_letter and sentence_similarity , which we attribute to the high-quality of prompt candidates generated by ChatGPT. This is also consistent with our discussion on the input domain in Sec. 3.2. Here we could not draw a direct comparison between ZOPO and ZOPO GPT, as the Vicuna last token embedding is specifically associated with the prompt generation process in ZOPO and cannot be applied to ZOPO GPT. However, using either ZOPO or ZOPO GPT is sufficient to outperform baseline methods, which also provides the flexibility of prompt optimization in practice. Future research may consider employing better embeddings to further improve the performance of ZOPO GPT.

ZOPO has better query efficiency. To justify that our local optimization method is more query-efficient , we compare ZOPO against baselines at different query budget scales. The result shown in Fig. 5 illustrates that ZOPO generally achieves better performance with the same number of queries compared with other baseline methods and yields superior performance upon convergence. We notice that ZOPO achieves lower validation accuracy yet higher test accuracy on the taxonomy_animal task than INSTINCT, which suggests ZOPO likely has better generalization ability. More results on other tasks in Fig. 10 in Appx. D.2 also indicate that ZOPO has consistent advantages.

40

80 120160200

0.25

0.50

0.75

Test Acc.

taxonomy_animal

40

80 120160200

0.50

0.75

cause_and_effect

40

80 120160200

0.4

0.6

informal_to_formal

40

80 120160200

0.25

0.50

0.75

Val. Acc.

40

80 120160200

0.50

0.75

40

80 120160200

0.5

0.6

# queries

INSTINCT

InstructZero

EvoPrompt

PB

OPRO

ZOPO

5.2 Improving Chain-of-Thought Prompt

Reasoning prompts, e.g., "Let's think step by step" (denoted as hand-craft), have been shown effective in improving LLMs' zero-shot multi-step reasoning performance [13, 15]. We show that ZOPO can find a better chain-of-thought prompt across different arithmetic reasoning tasks, as evidenced in Tab. 2. Particularly, ZOPO produces a better prompt "Let's find the solution by using the given information." on GSM8K [4] compared to other baselines, improving the performance from 71 . 80 (hand-craft) to 75 . 36 . Refer to Appx. C.4 for more experimental details.

5.3 Ablation Study

Verifying the essence of input domain. To fairly validate the importance of input domain on prompt generation, we compare the optimization performances with different prompts generated by Vicuna-13B and ChatGPT respectively, using the same embedding model SBERT (i.e., h ( · ) ). The result is shown in Table. 7 in Appx. D.6, with the performance profile in Fig. 11 suggesting that applying ZOPO on ChatGPT-generated prompts is better. We ascribe its better performance to ChatGPT's remarkable prompt generation ability. This confirms the importance of the input domain on prompt generation in our Insight II.

Besides, different embeddings (i.e., Z ) of the same prompt candidates can potentially affect the function landscape as shown in Fig. 4. Thus, we need to study the performance of ZOPO using different embedding representations given the same set of prompts. We consider four different embeddings here: the last token embedding from Vicuna-13B, the OpenAI embedding provided through an API [26], the SBERT embedding, and a randomly projected embedding baseline. We observe from Tab. 8 in Appx. D.6 that, although last token embedding is generally better, there are certain tasks that OpenAI and SBERT embeddings perform equally well or better. Besides, random embedding shows a distinct lesser performance. This again highlights the importance of using more structured embeddings for prompt optimization and indicates the optimal choice of embedding can be task-dependent . We discuss how we might find the best embedding model and further show the generality of ZOPO by experimenting with more embedding models in Appx. D.6.

Study of NTK-GP and uncertainty-informed local exploration. We conducted additional experiments to validate the NTK-GP (Sec. 4.2) and uncertainty-informed local exploration (Sec. 4.3) components of ZOPO . We evaluated the impact of these components by testing two variants of the ZOPO algorithm: (a) replacing the NTK component with Matérn kernel (as in ZoRD ), and (b) removing the uncertainty-informed local exploration. Comparisons of these variants against the original ZOPO on instruction induction tasks (see Tab. 11 in Appx. D.7) highlight the significant contributions of these components to ZOPO 's overall effectiveness.

Additional results. The results on the GLUE benchmark in Appx. D.3 consistently validate the superior performance of ZOPO . We also demonstrate that ZOPO can handle prompt optimization in the few-shot setting in Appx. D.4. We conduct further experiments to show ZOPO generalize to different combinations of prompt generation models and black-box LLMs in Appx. D.5. We also perform an ablation study to examine the impact of a larger size of the generated prompt candidates (i.e., |V| ) on ZOPO and ZOPO GPT in Appx. D.8, which suggests a relatively small set of strong

prompt candidates (e.g., |V| = 500 ) is sufficient (compared with size 1000 or 2000). Additionally, we provide more demonstrations of our empirical findings in Sec. 3 on other tasks in Appx. D.1.

6 Related Work

Soft Prompt Tuning. To control LLMs to perform specific downstream tasks (e.g., reasoning), soft prompt tuning [17] is usually used as a lightweight method to fine-tune the LLMs by only optimizing a continuous vector prepended to the input tokens using gradient descent, in contrast to fine-tuning millions to billions of model parameters [37, 43]. However, when the gradient information of the model is inaccessible, gradient-free prompt tuning methods [40, 39, 6] are developed to alleviate human efforts in prompt design. Still, those efforts to optimize soft prompts have conventionally relied on the white-box access to the embedding layers of LLMs, making it inapplicable to state-of-the-art LLMs like ChatGPT [25] and GPT-4 [24] that can only be accessed through black-box APIs (i.e., only accept natural language as input).

Discrete Prompt Optimization. We refer to the process of optimizing discrete prompts as "prompt optimization", which is also a more practical setting as black-box LLMs only accept discrete inputs. Reinforcement learning-based methods [5, 47] focus on discrete token optimization but rely on the output distribution of the LLMs, which is not accessible in black-box API LLMs (e.g., ChatGPT). Zhou et al. [48] instead makes use of LLMs to produce promising candidate prompts through resampling without applying specific optimizations. Some recent works, EvoPrompt [10] and PB [8], further extend this model-free approach to evolutionary algorithms and design meta-prompts that explicitly ask the LLM to perform iterative mutation and crossovers of existing prompt candidates. Besides, Pryzant et al. [30] ask the LLM to perform implicit gradient descent on existing prompts; OPRO [46] uses the LLM as an implicit optimizer to perform prompt optimization, where its designed meta-prompt takes in the optimization trajectory and instructs the LLM to output a promising prompt candidate. However, these methods rely on powerful LLMs (e.g., GPT-3.5) and typically require a large number of iterations and queries to perform well. In this regard, InstructZero [3] leverages the induction ability from other white-box LLM g ( · ) for generating the task-specific prompt v that is conditioned on a continuous soft prompt s ∈ R d . After that, the optimization on v can be transformed into an optimization on the soft prompt s , where BO algorithms are employed for a global black-box optimization. INSTINCT [18] further employs neural bandit algorithms and the last token embeddings from the white-box LLM to further improve the prompt optimization performance. However, these works prioritize a global optimization approach that emphasizes the exploration of the entire space. With an empirical understanding of the underlying target function (i.e., the black-box API LLMs), we propose a localized ZOO method that is in contrast to the global optimization approaches.

7 Conclusion

In this work, we first provide a thorough empirical study to understand the characteristics of the target function, and then propose our ZOPO algorithm for prompt optimization. ZOPO embraces a ZOO approach in pursuit of finding local optima efficiently. Extensive experiments on 30 instruction induction tasks, 3 reasoning tasks, and the GLUE benchmark demonstrate the efficacy of ZOPO , and ablation studies also validate the design principles and the generality of ZOPO . Besides, we propose a domain transformation that connects powerful LLMs with remarkable embedding models, which provides the flexibility of choices of input domains in prompt optimization. A limitation of this paper is the lack of principle to select LLMs and embedding models in our input domain transformation for better-performing prompt optimization, which we aim to explore in future work.

Acknowledgments and Disclosure of Funding

This research is supported by the National Research Foundation Singapore and the Singapore Ministry of Digital Development and Innovation, National AI Group under the AI Visiting Professorship Programme (award number AIVP2024 -001 ). This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG 2 -RP2020 -018 ).

References

Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv:2305.10403 , 2023.

Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R., and Wang, R. On exact computation with an infinitely wide neural net. In Proc. NeurIPS , 2019.

Chen, L., Chen, J., Goldstein, T., Huang, H., and Zhou, T. InstructZero: Efficient instruction optimization for black-box large language models. arXiv:2306.03082 , 2023.

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv:2110.14168 , 2021.

Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H., Shu, T., Song, M., Xing, E., and Hu, Z. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In Proc. EMNLP , 2022.

Diao, S., Huang, Z., Xu, R., Li, X., Yong, L., Zhou, X., and Zhang, T. Black-box prompt learning for pre-trained language models. Transactions on Machine Learning Research , 2023.

Dolan, E. D. and Moré, J. J. Benchmarking optimization software with performance profiles. Mathematical programming , 91:201-213, 2002.

Fernando, C., Banarse, D., Michalewski, H., Osindero, S., and Rocktäschel, T. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv:2309.16797 , 2023.

Flaxman, A., Kalai, A. T., and McMahan, H. B. Online convex optimization in the bandit setting: Gradient descent without a gradient. In Proc. SODA , 2005.

Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y . Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In Proc. ICLR , 2024.

Honovich, O., Shaham, U., Bowman, S. R., and Levy, O. Instruction induction: From few examples to natural language task descriptions. In Proc. ACL , 2023.

Jacot, A., Hongler, C., and Gabriel, F. Neural Tangent Kernel: Convergence and generalization in neural networks. In Proc. NeurIPS , 2018.

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. In Proc. NeurIPS , 2022.

Kratsios, A. and Papon, L. Universal approximation theorems for differentiable geometric deep learning. The Journal of Machine Learning Research , 23(1):8896-8968, 2022.

Lau, G. K. R., Hu, W., Liu, D., Chen, J., Ng, S.-K., and Low, B. K. H. Dipper: Diversity in prompts for producing large language model ensembles in reasoning tasks. In NeurIPS 2024 Workshop on Foundation Model Interventions , 2024.

Lee, J., Xiao, L., Schoenholz, S. S., Bahri, Y ., Novak, R., Sohl-Dickstein, J., and Pennington, J. Wide neural networks of any depth evolve as linear models under gradient descent. In Proc. NeurIPS , 2019.

Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. In Proc. ACL , 2021.

Lin, X., Wu, Z., Dai, Z., Hu, W., Shu, Y., Ng, S.-K., Jaillet, P., and Low, B. K. H. Use Your INSTINCT: INSTruction optimization usIng Neural bandits Coupled with Transformers. In Proc. ICML , 2024.

Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proc. Annual Meeting of the ACL , 2017.

Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Mishra, S., Khashabi, D., Baral, C., Choi, Y ., and Hajishirzi, H. Reframing instructional prompts to gptk's language. In Proc. ACL Findings , pp. 589-612, 2021.

Moriconi, R., Deisenroth, M. P., and Sesh Kumar, K. High-dimensional bayesian optimization using low-dimensional feature spaces. Machine Learning , 109:1925-1943, 2020.

Nesterov, Y. E. and Spokoiny, V. G. Random gradient-free minimization of convex functions. Found. Comput. Math. , 17(2):527-566, 2017.

OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

OpenAI. ChatGPT. https://openai.com/blog/chatgpt , 2024.

OpenAI. Documentation of OpenAI's text embeddings. https://platform.openai.com/docs/guides/embeddings , 2024.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Proc. NeurIPS , 2022.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. arXiv:2203.02155 , 2022.

Patel, A., Bhattamishra, S., and Goyal, N. Are nlp models really able to solve simple math word problems? In Proc. NAACL , 2021.

Pryzant, R., Iter, D., Li, J., Lee, Y ., Zhu, C., and Zeng, M. Automatic prompt optimization with 'gradient descent' and beam search. In Proc. EMNLP , 2023.

Reimers, N. and Gurevych, I. Sentence-BERT: Sentence embeddings using siamese bertnetworks. In Proc. EMNLP-IJCNLP , 2019.

Reynolds, L. and McDonell, K. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , pp. 1-7, 2021.

Shen, Z., Yang, H., and Zhang, S. Optimal approximation rate of relu networks in terms of width and depth. Journal de Mathématiques Pures et Appliquées , 157:101-135, 2022.

Shu, Y., Cai, S., Dai, Z., Ooi, B. C., and Low, B. K. H. NASI: Label- and data-agnostic neural architecture search at initialization. In Proc. ICLR , 2022.

Shu, Y., Dai, Z., Wu, Z., and Low, B. K. H. Unifying and boosting gradient-based training-free neural architecture search. In Proc. NeurIPS , 2022.

Shu, Y., Dai, Z., Sng, W., Verma, A., Jaillet, P., and Low, B. K. H. Zeroth-order optimization with trajectory-informed derivative estimation. In Proc. ICLR , 2023.

Shu, Y., Hu, W., Ng, S.-K., Low, B. K. H., and Yu, F. R. Ferret: Federated full-parameter tuning at scale for large language models. In NeurIPS 2024 Workshop on Federated Foundation Models , 2024.

Shu, Y., Lin, X., Dai, Z., and Low, B. K. H. Federated zeroth-order optimization using trajectoryinformed surrogate gradients. In ICML 2024 Workshop on Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators , 2024.

Sun, T., He, Z., Qian, H., Huang, X., and Qiu, X. Bbtv2: Pure black-box optimization can be comparable to gradient descent for few-shot learning. arXiv:2205.11200 , 2022.

Sun, T., Shao, Y., Qian, H., Huang, X., and Qiu, X. Black-box tuning for language-model-as-aservice. In Proc. ICML , 2022.

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V ., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y ., Mao, Y ., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y ., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 , 2023.

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proc. ICLR , 2019.

Wei, C., Shu, Y., He, Y. T., and Yu, F. R. Flexora: Flexible low-rank adaptation for large language models. In NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability , 2024.

Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. WizardLM: Empowering large pre-trained language models to follow complex instructions. In Proc. ICLR , 2024.

Xu, X., Wu, Z., Qiao, R., Verma, A., Shu, Y., Wang, J., Niu, X., He, Z., Chen, J., Zhou, Z., Lau, G. K. R., Dao, H., Agussurja, L., Sim, R. H. L., Lin, X., Hu, W., Dai, Z., Koh, P. W., and Low, B. K. H. Position paper: Data-centric ai in the age of large language models. In Proc. EMNLP , 2024.

Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. In Proc. ICLR , 2024.

Zhang, T., Wang, X., Zhou, D., Schuurmans, D., and Gonzalez, J. E. TEMPERA: Test-time prompt editing via reinforcement learning. In Proc. ICLR , 2023.

Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human-level prompt engineers. In Proc. ICLR , 2023.

Appendix A Proofs

A.1 Proof of Prop. 1

We follow the ideas in [36, 38] to prove our Prop. 1. To begin with, we first introduce the following lemmas adapted from [36]:

Lemma A.1 (Thm. 1 in [36]) . Let δ ∈ (0 , 1) and ω ≜ d +2( √ d +1)ln(1 /δ ) . For any z ∈ Z and any t ≥ 1 , the following holds with probability of at least 1 -δ ,

Lemma A.2 (Lemma B.4 in [36]) . For any z ∈ Z and any t ≥ 1 , the following holds

Proof of Prop. 1. Recall that the covariance function (refer to (4)) of our derived NTK-GP conditioned on the history of function queries D t ≜ { ( z τ , r τ ) } t τ =1 of size t will be

For any c ∈ R and z ∈ Z , define N z,β ≜ { z ′ ∈ { z τ } t τ =1 | ∥ ∂ z k ( z, z ′ ) ∥ 2 ≥ β } with | N z,β | = N , the following then holds on the set N z,β :

where ( a ) comes from the fact the maximum eigenvalue of a matrix is always larger or equal to its averaged eigenvalues, ( b ) is based on tr( AB ) = tr( BA ) , ( c ) is from the definition of k N ( z ) , and ( d ) results from the definition of N z,β .

Meanwhile,

where ( a ) comes from Lemma A.2, ( b ) is based on the assumption of ∥ k ′′ ( z, z ) ∥ ≤ κ and the definition of maximum eigenvalue. In addition, ( c ) comes from λ max ( K N ) ≤ N max z,z ′ ∈ N z,β k ( z, z ′ ) (i.e., the Gershgorin theorem) and the assumption that k ( z, z ′ ) ≤ α for any z, z ′ ∈ Z , and ( d ) is based on the results in (8).

Finally, by introducing the results above into Lemma A.1, we conclude the proof.

Appendix B Broader Impacts

As LLMs have recently received great popularity in human society and their various applications have significantly affected many aspects of society, it is important to make sure the technology related to LLMs is helpful and harmless. Our work focuses on improving the performance of black-box LLMs by automatically optimizing the prompts, which can significantly save human efforts in prompt engineering. However, such work can be potentially used for malicious purposes. When an adversarial user defines a harmful objective function, our work could be exploited to output harmful prompts that lead to negative societal impacts. Therefore, we urge the black-box LLM API providers to impose a security check for the prompt that prevents users from querying for malicious purposes.

Appendix C Details of Experimental Settings

C.1 Evaluation Metrics

Following previous works [48, 18], we use the F1 score for tasks including common_concept and in-formal_to_formal ; we use the exact set matching for orthography_starts_with and taxonomy_animal ; we use the set containing for synonyms ; we use the exact matching metric for the rest of instruction induction tasks; and we use the accuracy metric for the arithmetic reasoning datasets.

As the number of datasets is tremendous, we use the performance profile [7] as the evaluation metric that measures the frequency (i.e., ρ ( τ ) ) of a method within some distance (i.e., τ ) from the optimality achieved by any method, defined below

where Π is the set of all tasks, r π,m is the accuracy of method m on task π , and r ∗ π = max { r π,m : ∀ m ∈ M} is the best performance achieved by any method in M on task π . Specifically, ρ (0) represents the number of tasks where a method achieves the best performance. Accordingly, we use both ρ (0) and ρ (5) as the evaluation indicators in our tables to report the results.

C.2 Hyperparameters

For all experiments using ZOPO in this work, we set the learning rate to 0.01, the uncertainty thresholds λ, ξ to 0.1 and 5 respectively, and the number n of nearest neighbors to query in local exploration (Section 4.3) to 10. A neural network with 2 fully connected layers of size 32 and ReLU activation functions is used in NTK-GP as the kernel. We use 20 nearest neighbors to fit the NTK-GP.

C.3 Instruction induction

In this subsection, we describe the experimental details of the instruction induction tasks.

C.3.1 Experimental Specifications

The same data partition and evaluation process as in previous works [48, 3, 18] is adopted in this work, where, for each task, we optimize the generated prompt on a training set D , and report the best-performing prompt's inference accuracy on a held-out test set D T . Specifically, 5 examples are sampled from the training set as the demonstrations (i.e., D demo) for instruction induction, and another sampled 20 examples from the training set are used as the validation set D V to evaluate the objective function value as in Equation (1). The total query budget for each instruction induction task is fixed at 165 for all methods.

C.3.2 Implementation Details

To comprehensively compare with the baseline methods, we use GPT-3.5-turbo-0301 (supported by OpenAI API) as the black-box model for prompt evaluation and Vicuna-13B-v1.1 as the white-box LLM (i.e., g ( · ) ) to generate the task-specific prompts by feeding g ( · ) with randomly sampled soft prompts and D demo, which is the same as InstructZero and INSTINCT. In the experiments, we only generate 500 prompt candidates for ZOPO (i.e., |V| = 500 ). Similarly, we also use 40 out of the 165

queries for random initialization of our optimization method, which could serve as the only global exploration of the function landscape at the beginning of local optimization.

To tackle the high dimensionality of soft prompt (i.e., 5120 for one token embedding as in Vicuna13B) in optimization, InstructZero and INSTINCT use random projection to project the soft prompt into a much smaller intrinsic dimension (e.g., 100). This intrinsic dimension may empirically affect the quality of generated prompts, as shown in Lin et al. [18]. Therefore, tuning the intrinsic dimension and the soft token length could lead to better performance. Previous methods (i.e., InstructZero and INSTINCT) perform a grid search over the intrinsic dimension in {50, 100, 200} and the soft token length {3, 5, 10} on the validation set and report the accuracy on a held-out test set using the best prompt found using the validation set. We also adopt this technique in ZOPO here for fair comparison. The soft prompt will be concatenated with the tokenized embedding of the prompt generation template to generate task-specific prompt from Vicuna-13B. The prompt generation template and the prompt evaluation template are shown below in the bounding boxes.

Prompt Generation Template (Soft Prompt)

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

The prompt was to?

Evaluation Template

prompt :

⟨ prompt ( i.e., v ) ⟩

Input :

⟨ TEST INPUT ⟩

Output :

We directly use the reported results of APE, IntructZero, and INSTINCT from Lin et al. [18] for comparison, and we report the results of EvoPrompt with our re-implementation. For a fair comparison, we also use Vicuna-13B for generating the initial prompt population (of size 20) for EvoPrompt, and we use GPT-3.5 turbo to perform the genetic algorithm in EvoPrompt and generate its new prompts. Using GPT-3.5 turbo to generate new prompts will help improve EvoPrompt's performance, as compared with using the relatively smaller model Vicuna-13B.

C.3.3 Experimental Details on Query Efficiency

To facilitate a more comprehensive comparison of different prompt optimization methods at different query budget scales, we set the maximum query budget to 200, and report the test accuracy of the best prompt found on the validation set with each incremental query budget, as shown in Fig. 5 in the main text. We report the mean accuracy and standard error, using 3 runs with different random seeds. For InstructZero, INSTINCT, and ZOPO , we directly fix the intrinsic dimension for generating the soft prompt as 10 and the number of soft tokens as 5, without using the validation set to perform a grid search over the intrinsic dimension and the number of soft tokens.

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output

:

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

ChatGPT Prompt Generation Template

I gave a friend an prompt. Based on the prompt they produced the following input-output pairs:

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output

:

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

Input :

⟨ INPUT ⟩

Output :

⟨ OUTPUT ⟩

The prompt was to

C.3.4 Experimental Details on ZOPO GPT

For our experiment on ZOPO GPT in the main text, we apply ZOPO on ChatGPT (i.e., GPT-3.5 turbo) generated prompts. We follow the generation template from APE [48], as shown above, to generate task-specific prompts from ChatGPT. To generate various prompts using the APE method, we need to sample different sets of demonstrations (i.e., D demo) from the training set, and, for each D demo, we also need to randomly sample from the ChatGPT's response by setting a high temperature (e.g., 0.95). To maintain the same size of prompt candidates as in the previous experimental setting of ZOPO , we also generate 500 prompt candidates for each instruction induction task. To harness the representation power of existing embedding models, we adopt the sentence transformer model [31] 'all-mpnet-base-v2' from HuggingFace to generate the high-dimensional sentence embedding for each generated prompt from ChatGPT.

C.4 Improving Chain-of-Thought Prompt

To improve the zero-shot chain-of-thought prompt performance on arithmetic reasoning tasks, we make use of the LLM's induction ability and enable LLMs to generate different chain-of-thought prompt candidates by providing some example chain-of-thought prompts. We consider the evaluation of our method on three arithmetic reasoning datasets (i.e., GSM8K[4], AQUARAT[19], SVAMP[29]). Similar as APE [48], we use all data from the test set for GSM8K and AQUARAT, and we sample 400 data points from AQUARAT's test set to evaluate the corresponding test accuracy. For all these three datasets, we sample 200 data points from their training dataset respectively as their individual validation dataset.

We follow the experimental setting of Lin et al. [18]: use the soft prompt to generate prompts from Vicuna-13B with a fixed intrinsic dimension of 1000 and search the soft token length {3, 5, 10} on the validation set. The corresponding prompt generation template is given below.

Prompt Generation Template for Chain-of-Thought

I have some prompt examples for solving school math

problems.

prompt:

Let's figure it out!

prompt:

Let's solve the problem.

prompt:

Let's think step by step.

Write

your

new

prompt

that

is

different

from

the examples to solve the school math problems.

prompt:

C.5 Details on Compute Resources

All experiments are conducted on a server with Intel(R) Xeon(R) CPU and NVIDIA H100 GPUs. We mainly perform the prompt optimization for the GPT-3.5-Turbo model (for which OpenAI charged USD 0 . 5 per 1M tokens for input and USD 1 . 5 per 1M tokens for output). The time of execution of our algorithm on each prompt optimization task (e.g., any task in the 30 instruction induction tasks) normally takes less than 20 minutes, where the actual time would depend on OpenAI API's response speed.

Appendix D Additional Results

D.1 Extended Empirical Study on Function Landscape

In Section 3, we have empirically studied the landscape of the target function and incorporated the findings into the design of ZOPO . In the main text, we have demonstrated the results on three instruction induction datasets, including taxonomy_animal, cause_and_effect, and informal_to_formal . Here we use more datasets to validate our findings. Due to the large size of instruction induction tasks (i.e., 30 tasks in total) and the query budget limit (i.e., it incurs monetary costs when we query the objective function ChatGPT to evaluate the prompt on the given task), we only experiment with few more randomly chosen tasks here to further validate our findings.

D.1.1 Local Optima vs. Global Optimum

To validate our local optimization design, we study the local optima in the function landscape, by using a 3-dimensional (reduced by t-SNE) scatter plot to represent the prompt embeddings (last token embeddings from Vicuna-13B). Here we provide the empirical results on more instruction induction tasks, shown in Fig. 6. The heatmap color represents the validation accuracy of the corresponding prompt. This allows us to interpret the local optima visually, and we conclude that many local optima can already exhibit compelling performance.

word_sorting

sentiment

synonyms

singular_to_plural

common_concept

0.5

1.0

0.0

D.1.2 Essense of Input Domain

Prompt Generation To study the prompt quality of different prompt generation methods, we compare the prompts generated from Vicuna-13B and those generated from ChatGPT (i.e., GPT 3.5 turbo). For Vicuna-13B, we use the randomly sampled soft prompts with a fixed intrinsic dimension of 200 and a number token length of 10. For ChatGPT, we randomly sample prompts from the ChatGPT's response by using the APE generation template filled with random example demonstrations. For each generation method on each task, we generate 300 random prompts, and we query the target function with all prompts. We show the validation accuracy distribution of prompts generated by the two methods on four more (due to budget constraints) tasks here in Fig. 7. It demonstrates that ChatGPT has a larger probability of generating prompts with higher accuracy, also with a larger mean. The result shows that ChatGPT-generated prompts are generally better, further validating our finding of the importance of the input domain.

0.0

0.2

0

5

10

Probability Density

auto_categorization

0.0

0.5

1

2

negation

0

1

0

5

10

singular_to_plural

0.0

0.5

0

2

4

synonyms

Validation Accuracy

Vicuna-13B

ChatGPT

Prompt Embedding The complexity of modeling the target function depends on its function landscape defined by the embedding domain. To empirically analyze the black-box target function, we show the accuracy landscape of different tasks, where we reduce the dimension of the prompt embedding (we use the last token embedding of Vicuna-13B here) to 2 by using t-SNE. The loss landscape is visualized in the surface plot shown in Fig. 8. We observe that different optimization methods achieve similar performances on tasks like sentiment and singular_to_plural , as they have many good local optima. For other challenging tasks with complex function landscapes, the good local optima are less, but our methods can still achieve superior performance. This validates our insight that there are many good local optima in the embedding space.

word_sorting

0.00

0.00

0.00

0.00

0.05

0.05

0.05

0.05

0.05

0.05

0.05

0.05

0.10

0.10

0.10

0.10

0.15

0.15

0.15

sentiment

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.3

0.3

0.3

0.3

0.3

0.3

0.3

0.3

0.3

0.3

0.3

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.6

0.6

0.6

.6

0.6

0.6

0.6

0.6

0.6

0.6

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.8

0.8

0.8

0.8

synonyms

0.00

0.00

0.00

0.00

0.00

0.06

0.06

0.06

0.06

0.06

0.06

0.06

0.12

0.12

0.12

0.12

0.12

0.12

0.18

0.18

0.18

0.24

singular_to_plural

0.0

0.1

0.1

0.1

0.1

0.1

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.3

0.3

0.3

0.3

0.3

0.3

0.3

0.3

0.3

0.3

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0.6 0.6

0.6

0.6

0.6

0.6

0.6

0.6

0.6

0.6

0.6

0.6

0.6

0.6

0.6

0.6

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.7

0.8

0.8

0.8

0.8

0.8

0.8

0.8

0.8

0.8

0.8

0.8

0.8

0.9

0.9

0.9

common_concept

0.000

0.000

0.015

0.015

0.015

0.015

0.015

0.015

0.015

0.015

0.015

0.030

0.030

0.030

0.030

0.045

0.0

0.5

1.0

0.6

D.2 Comparison on Instruction Induction Tasks

In Section 5.1 of the main text, we compared our methods with other baselines on 20 challenging instruction induction tasks. Here we provide the full results on 30 instruction induction tasks in Section 5.1.

The performance profile of ZOPO GPT compared against other baseline methods is shown in Fig. 9. This corresponds to the result shown in Tab. 1.

0

10

20

¿

0.0

0.5

½

(

¿

)

APE

InstructZero

INSTINCT

EvoPrompt

PB

OPRO

ZOPOGPT(ours)

We also provide additional results on other instruction induction tasks to compare ZOPO against baseline methods in terms of query efficiency. The result is shown in Fig. 10.

40

80 120 160 200

0.0

0.5

Test Acc.

word_sorting

40

80 120 160 200

0.2

0.4

auto_debugging

40

80 120 160 200

0.2

0.4

synonyms

40

80 120 160 200

0.25

0.50

word_unscrambling

40

80 120 160 200

0.1

0.2

common_concept

40

80 120 160 200

0.25

0.50

0.75

Val. Acc.

40

80 120 160 200

0.4

0.6

40

80 120 160 200

0.25

0.50

40

80 120 160 200

0.5

0.6

40

80 120 160 200

0.1

0.2

0.3

# queries

INSTINCT

InstructZero

EvoPrompt

PB

OPRO

ZOPO

D.3 Results on the GLUE Benchmark

We here follow the same experimental setting as our previous experiments on the instruction induction tasks and apply the prompt optimization on the GLUE benchmark, which consists of several more traditional natural language processing tasks. The result in Tab. 4 shows that our method ZOPO is still able to achieve advanced performance when compared with other baselines.

D.4 Few-shot Setting

Our algorithm ZOPO is in fact able to handle the few-shot settings as evidenced by the results in Tab. 5 below. Interestingly, the performance of our ZOPO is even better in a few-shot setting, which is reasonable since in-context exemplars will help the black-box models better understand the context and the output format of the task, and consequently will be able to lead to a better performance than the zero-shot setting. In this few-shot experiment, we provide exemplars for prompt evaluation and also report the test accuracy of the best prompt with exemplars provided.

D.5 Results of Different Combinations of Generation and Evaluation Models

We use further experiments to show that our method can generalize to different prompt generation models and different black-box API LLMs for prompt evaluation (as f ( · ) ). Specifically, we here consider two open-sourced models: Vicuna-13B and WizardLM-13B [44] for the prompt generation and we use their corresponding last token embeddings in our algorithm. For the black-box API LLMs, we consider GPT-3.5 (the one considered in our main text), PaLM2 [1], and GPT-4. In total, we have six combinations. Tab. 6 shows the results of different combinations on the instruction induction tasks. The results show that our ZOPO performs well on all these black-box API models (with GPT-4 performing the best on most tasks), which further verifies the generality of our method when a different black-box LLM f ( · ) is considered in the objective function in Eq. 1. We also notice that the Vicuna model generally performs better than the WizardLM model, which suggests it is more suitable for prompt generation and representation when applying our method ZOPO .

D.6 Verifying the Essence of Input Domain

Prompt Generation To fairly compare the effect of prompts generated by Vicuna-13B and ChatGPT in terms of the optimization performance by using ZOPO , we adopt the same embedding representations here, that is we use the SBERT embedding model for both prompts generated by Vicuna-13B and ChatGPT. For the prompt generation process, we fix the number of prompt candidates for both methods to 500. The result of the comparison on 20 instruction induction tasks is shown in Table. 7, where the corresponding performance profile shown in Fig. 11 suggests that applying ZOPO on ChatGPT-generated prompts is better than applying it on Vicuna-generated prompts. This again confirms the importance of the choice of the input domain (i.e., the prompt generation).

0

10

20

¿

0.6

0.7

0.8

0.9

½

(

¿

)

Vicuna-13B

ChatGPT

Prompt Embedding Here we analyze how different embeddings affect the optimization of ZOPO . We first generate a fixed set of prompts of size 500 from Vicuna-13B as those in Tab. 1. For the same set of prompts, we consider four different embeddings here: (a) the Last Token embedding from Vicuna-13B (b) the OpenAI embedding obtained through its embedding model 'text-embedding-ada002" API. [26], (c) the SBERT embedding obtained through the sentence transformer ('all-mpnetbase-v2' from HuggingFace), and (d) the Random embedding obtained by randomly projecting the Vicuna embedding into the same dimension. The dimensions of the four embeddings (from (a) to (d)) are 1536, 756, and 5120 respectively. We compare the optimization performance of the four embeddings using ZOPO and the results are shown in Tab. 8. We observe although last token embedding is generally better, there are certain tasks that OpenAI and SBERT embeddings perform equally well or better, which indicates the optimal choice of embedding can be task-dependent . Intuitively, random embedding is not representative. Its lesser performance shown in Tab. 8 again confirms our Insight II in Sec. 3.2, which says the choice of embedding/input domain is important in prompt optimization.

To demonstrate the generality of ZOPO , we provide more results below to show we are not focusing on a specific combination of LLMs and embedding models. For the same optimization objective (i.e., we still perform prompt optimization for ChatGPT), we further study our ZOPO with more choices of embedding models and pair it with different black-box LLMs for prompt generation (i.e., GPT-4). We consider three more embedding models from HuggingFace and OpenAI, including "Instructor-Large", "MiniLM-L6-v2", and "text-embedding-3-small" in Tab. 9 below, as well another black-box model (i.e., GPT-4) for prompt generation in Tab. 10 below.

Note that, in our main text, we use the Vicuna-13B model as the prompt generation model mainly for the fair comparison against baselines (i.e., InstructZero and INSTINCT), and we can tell from Tab. 9 that the Vicuna embedding is a good embedding to use in prompt optimization. The result from Tab. 10 suggests choosing a better embedding model (i.e., Instructor-Large) for GPT-4 generated prompts can even further improve its performance.

Can we possibly find the best embedding model? As we have shown in our previous experiments, the best choice for the embedding model can be task-dependent . To find the suitable paired embedding model without enumerating every embedding model, we could possibly analyze the variance V e of

the eigenvalues of the covariance matrix of the embeddings (i.e., Z ). The eigenvalues represent the variances along the principal directions (eigenvectors) of the embeddings. If the embeddings are distributed with equal spacing in the high-dimensional space, we would expect the eigenvalues to be approximately equal (i.e., low variance). Intuitively, if the embeddings are in such representative space with equal spacing, it could help function modeling.

Therefore, if such variance V e is small, the optimization performance using such an embedding model is more likely to be better. Based on the test accuracies from Tab. 9 and Tab. 10, we show that there exists a sufficiently high negative Spearsman's correlation (i.e., the average correlation is -0.47) between V e and the performance of ZOPO using different embedding models on each task. Therefore, we can check every embedding prior to the experiment, and it can be done efficiently. We acknowledge that this approach is not perfect and finding the best embedding model is not the main focus of this work. By finding the best pair of generation and embedding models for prompt optimization, we believe our ZOPO algorithm can be further boosted. We would like to take it as a potential future direction.

D.7 Study of NTK-GP and Uncertainty-Informed Local Exploration

To validate the effectiveness of the components, namely NTK-GP (in Sec. 4.2) and uncertaintyinformed local exploration (in Sec. 4.3) of ZOPO , we perform controlled experiments to replace these components. Specifically, we (a) replace the NTK component with Matérn kernel (as in the recent ZOO method ZoRD), and (b) remove the uncertainty-informed local exploration feature. We evaluate the two settings on 20 instruction induction tasks. The result shown in Table 11 illustrates these two settings are both significantly worse than the original ZOPO , which validates the effectiveness of NTK-GP and uncertainty-informed local exploration.

D.8 Study of ZOPO with More Prompt Candidates

Intuitively, generating more prompt candidates offers a closer approximation to the true function landscape. As our optimization method ZOPO is operated under a given set of prompt candidates, we here conduct an ablation study to examine the impact of a larger size of the generated prompt candidates (i.e., |V| ) on the optimization performance. For ZOPO , we use random soft prompts to feed Vicuna-13B and generate prompts until V = 500 or V = 2000 . We compare the optimization results of ZOPO using the two different sizes of prompts, and the results are shown in Table 12. We also follow the APE generation template to prompt ChatGPT to generate different sizes of prompt candidates and use SBERT to produce their embeddings. For ChatGPT-generated prompts in ZOPO GPT, we also consider two settings, V = 500 or V = 1000 (due to budget constraint). The corresponding result is shown in Table 13. We observe from the two tables that a larger set of prompt candidates may not necessarily lead to strictly better performance, and generating a relatively small set of strong prompt candidates (e.g., of size 500) is already good enough when we aim to find the optimal prompt.

D.9 Best Prompts Found

We list the best prompts discovered by our method ZOPO for every instruction induction task here in Table 14, which corresponds to the results in Table 3.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have provided extensive experimental results to validate the efficacy of our proposed method. The claims are also empirically supported.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are mentioned in Sec. 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions and proofs are included in the Appx. A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have included all the necessary information to implement our algorithm, and the experimental details to conduct the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have included our code in the supplementary materials with exact commands. The code will be uploaded to GitHub upon acceptance of the work.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental details are specified in Appx. C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the average accuracy with standard error (3 runs) in our tables, and plot the error bar for the query efficiency experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computing resources needed to reproduce the experiments are stated in Appx. C.5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research adopts the ethical practices mentioned in the Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Both potential positive and negative societal impacts are discussed in Appx. B. Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No new data or models in this work will be released. All data and models involved in this work have no risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The sources of the datasets are cited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the prompt optimization task. The authors first propose two insights: (1) Instead of pursuing the global optimum, this paper claims that local optima are usually prevalent and well-performing. (2) The input domain for prompt optimization affects the choice of local optima. Inspired by these two observations, this paper proposes a zeroth-order optimization method that incorporates a Neural Tangent Kernel-based derived Gaussian process to search for local optima. This method achieves competitive results on benchmarking datasets.\n\nSTRENGTHS:\n1. I like the analysis in Section 3. The two insights are well supported by the provided studies, and the motivation and reasoning for decision-making in this paper are explained in an informative way.\n\n2. Compared to methods that aim to find global optimization, incorporating NTK-based Gaussian processes in prompt optimization should theoretically be much faster.\n\n3. The input domain transformation process leads to a dense numerical feature space for prompts, making the optimization problem easier.\n\nWEAKNESSES:\n1. This paper does not discuss the following recent prompt optimization methods:\n\n[1] Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. Guiding large language models via directional stimulus prompting. Advances in Neural Information Processing Systems, 36, 2024.\n\n[2] Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. Query-dependent prompt evaluation and optimization with offline inverse rl. In The Twelfth International Conference on Learning Representations, 2023.\n\n[3] Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric Xing, and Zhiting Hu. Promptagent: Strategic planning with language models enables expert-level prompt optimization. In The Twelfth International Conference on Learning Representations, 2024. 3\n\n2. For those compared methods, ZOPO did not consistently show advantages in Table 1 and Table 3 in Appendix D2.\n\n3. This paper emphasized efficiency. However, in Figure 5, Figure 10, and Appendix D.2, I can observe an obvious advantage in efficiency when compared with other methods. In addition, we can observe some results that are contradictory to the paper’s claim. In many scenarios in Figure 10, ZOPO does not show obvious advantages when the query number is small, which is contradictory to the 'query-efficient prompt' claim.\n\nQUESTIONS:\nI cannot see a consistent advantage of ZOPO in many figures and tables. Can you explain this part? Thanks.\n\nLIMITATIONS:\nThey listed the limitation and claimed to solve it in future work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper titled "Localized Zeroth-Order Prompt Optimization" proposes a novel algorithm called ZOPO (Localized Zeroth-Order Prompt Optimization) aimed at enhancing the efficiency of prompt optimization in large language models (LLMs). The authors argue that local optima, as opposed to global optima, are more prevalent and can be more effectively targeted for prompt optimization. They introduce a combination of Neural Tangent Kernel (NTK) and Gaussian processes within a zeroth-order optimization framework to improve query efficiency and optimization performance.

STRENGTHS:
1. The thorough empirical study conducted provides a detailed comparison between local and global optima, highlighting the potential advantages of targeting local optima.

2. The ZOPO algorithm is well-designed, incorporating NTK-based Gaussian processes to enhance the optimization process, which shows promise in improving query efficiency.

WEAKNESSES:
The proposed ZOPO algorithm is complex and might be challenging to implement for practitioners who are not deeply versed in NTK and Gaussian processes. This limits the accessibility and practical utility of the proposed method.

QUESTIONS:
How do the proposed prompt-tuning approaches compare to the fully fine-tuning ZO approaches such as MeZO? It would be better to justify the settings that require prompt optimization.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes multiple contributions:\n\n1. Establishes a new visualization technique for the objective landscapes of blackbox functions over prompts. This is done by converting the high dimensional embeddings of strings into 2D (via t-SNE), and visualizing the landscape in 3D. Using this, the paper finds several patterns:\n  * There is a correlation between the smoothness of the landscape and the strength of the prompt generator.\n  * Much of the landscape is filled with local minima.\n\n2. Proposes a new Bayesian Optimization-like algorithm, with the following setup:\n* The regressor + uncertainty estimator is a NTK-GP using an MLP\n* The acquisition maximization is a gradient descent in the embedding space, with a projection back into the original prompt space.\n\nSTRENGTHS:\n* The proposed visualization method is simple yet surprisingly very insightful. I believe this might become a very important tool for any string-based blackbox optimization to assess the landscape.\n\n* The Bayesian optimization-like algorithm makes intuitive sense (bar the weaknesses, see below). This paper is well-written and is straightforward to read.\n\n* The conducted experiments are rigorous and comprehensive over numerous tasks with multiple baselines. Ablation studies are also relevant and insightful.\n\nWEAKNESSES:\n* Section 4.2 is not well-motivated. I understand that the idea is to construct an acquisition function expressing explore/exploit tradeoffs, and the most natural regressor to use is a Gaussian process, leading to the idea of using the NTK kernel. But this may seem overly complicated. For instance, why not use a simpler regressor / uncertainty estimator, like an ensemble of MLPs?\n  * Since the NTK requires computing dot-products of gradients, this makes it tricky to use for larger models (which have much longer gradients as feature vectors).\n\n* (Small) I would tone down the statement that Bayesian Optimization (or more generally, regressor guided search) will do poorly in local-optima situations simply because it was designed to search for global optima. There are several previous works over traditional black-box optimization showing Bayesian Optimization remains competitive even with multiple local optima. Furthermore, one could argue that the paper is essentially a Bayesian Optimization technique given the gradient ascent over essentially an explore-exploit acquisition.\n\nQUESTIONS:\nPlease address the main questions in the weaknesses above. \n\nThese are more for clarification:\n1. How does $h^{-1}$ (i.e. mapping an embedding back into some text) work? L165 mentions storing $(z,v)$ for constructing this inverse mapping - does this mean there are already alot of candidate prompts pre-generated forming an embedding set $Z$, and for a new $z$, we simply perform randomized rounding or projection to the nearest legitimate prompt in $Z$? \n\n2.  Eq 6: What happens if we fully attempt to argmax the acquisition (like regular Bayesian Optimization), rather than just move by a gradient step? I understand that this gradient step may be motivated by the landscape being full of local optima, but there may be missed gains here. Is this explained in L220-L230?\n\nLIMITATIONS:\nSection 7 discusses some limitations, but it may be worth considering the raised weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses prompt optimization for a black-box API LLM. This paper empirically investigated the objective function landscape of the prompt optimization problem and derived two insights: (I) local optima are usually prevalent and well-performed, (II) choice of the input domain affects the identification of well-performing local optima. Based on these insights, a novel local prompt optimization algorithm based on NTK-GP is proposed. Empirical comparisons have been conducted to show the performance differences between the proposed and baseline approaches.\n\nSTRENGTHS:\n* a novel and efficient prompt optimization algorithm targeted for a black-box API LLM\n\n* promising performance over baseline approaches\n\n* analysis and visualization of the objective function landscape using t-SNE\n\nWEAKNESSES:\n* The clarity of the algorithm could be improved (see the question part)\n\n* The validity of the derived insights is not sufficiently high (see the question part)\n\nQUESTIONS:\nL 118. “We then investigate the function surface (i.e., accuracy landscape) using two different embeddings for prompt candidates in Fig. 4 (more details in Appx. D.1.2) where the embeddings are mapped into a 2-dimensional domain using the t-SNE for better visualization. “ It is not clear how the local optima in the 2D t-SNE space is related to the local optimality of the objective function in the original space? Because the relation is not clear, I am not sure if the insight derived in this paper is valid or not.\n \nIt is not clear how Section 4.1 is related to Insight (II). What is the novelty of this part? This question is also related to the next question.\n\nL165. “We store (z, v) … for constructing the one-to-one inverse mapping.” Because h is a mapping from a discrete space to a continuous space, there does not exist a bijection theoretically. Therefore, it is not clear what the authors mean by this sentence. It is also not clear how this goal is achieved. Please clarify this point.\n\nL197. “theta_0 is the initialized parameters …“ Is it trained? If so, how and when? \n\nTable 1. I couldn’t find the explanation of ZOPO_{GPT}. What is the difference between ZOPO and ZOPO_{GPT}?\n\nThe performance of the baselines and the proposed approaches are compared only up to 200 queries. It is interesting to see how it will change if more queries are allowed.\n\nLIMITATIONS:\nA limitation has been addressed in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}