Title: Efficient Prompt Optimization Through the Lens of Best Arm Identification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Prompt Optimization; Best-arm Identification; Limited Budget; Large Language Models
Abstract: The remarkable instruction-following capability of large language models (LLMs) has sparked a growing interest in automatically finding good prompts, i.e., prompt optimization. Most existing works follow the scheme of selecting from a pre-generated pool of candidate prompts. However, these designs mainly focus on the generation strategy, while limited attention has been paid to the selection method. Especially, the cost incurred during the selection (e.g., accessing LLM and evaluating the responses) is rarely explicitly considered. To overcome this limitation, this work provides a principled framework, TRIPLE, to efficiently perform prompt selection under an explicit budget constraint. TRIPLE is built on a novel connection established between prompt optimization and fixed-budget best arm identification (BAI-FB) in multi-armed bandits (MAB); thus, it is capable of leveraging the rich toolbox from BAI-FB systematically and also incorporating unique characteristics of prompt optimization. Extensive experiments on multiple well-adopted tasks using various LLMs demonstrate the remarkable performance improvement of TRIPLE over baselines while satisfying the limited budget constraints. As an extension, variants of TRIPLE are proposed to efficiently select examples for few-shot prompts, also achieving superior empirical performance.

Efficient Prompt Optimization Through the Lens of Best Arm Identification

Abstract

The remarkable instruction-following capability of large language models (LLMs) has sparked a growing interest in automatically finding good prompts, i.e., prompt optimization. Most existing works follow the scheme of selecting from a pregenerated pool of candidate prompts. However, these designs mainly focus on the generation strategy, while limited attention has been paid to the selection method. Especially, the cost incurred during the selection (e.g., accessing LLM and evaluating the responses) is rarely explicitly considered. To overcome this limitation, this work provides a principled framework, TRIPLE , to efficiently perform prompt selection under an explicit budget constraint. TRIPLE is built on a novel connection established between prompt optimization and fixed-budget best arm identification (BAI-FB) in multi-armed bandits (MAB); thus, it is capable of leveraging the rich toolbox from BAI-FB systematically and also incorporating unique characteristics of prompt optimization. Extensive experiments on multiple well-adopted tasks using various LLMs demonstrate the remarkable performance improvement of TRIPLE over baselines while satisfying the limited budget constraints. As an extension, variants of TRIPLE are proposed to efficiently select examples for few-shot prompts, also achieving superior empirical performance.

1 Introduction

Large language models (LLMs) have rapidly changed technology landscapes in our society [20, 72, 73, 11, 55]. Researchers continuously find effective ways to unlock their potential on various downstream tasks. Among different research directions, the remarkable ability of LLMs to follow instructions has motivated the study of searching for suitable prompts to interact with them [50]. This approach is particularly attractive as it does not require updating the inside parameters of an LLM, and is natural in the way of human conversations. Nevertheless, it has also been recognized that the performance of an LLM is sensitive to the selected prompts [87, 52], and manually designing suitable prompts can be a labor-intensive process [54]. Thus, there is a growing interest to perform automatic prompt optimization [90, 79, 21, 19, 59, 84, 28, 58, 60, 81].

While these studies have proposed different prompt optimization designs, they commonly follow the approach of generating a pool of candidate prompts and then selecting from them. With a deeper look, it can be recognized that the focus in these existing works largely leans towards how to generate the candidate pool, while limited attention has been paid towards how to select from the candidates.

For example, many works [35, 79, 28, 59] directly evaluate all the generated prompts on the entire development dataset. However, this less-emphasized selection process typically requires accesses to LLMs, which are often (1) financially costly (e.g., each OpenAI API access incurs a cost); (2) time-wise consuming (e.g., even a locally hosted LLM would typically require seconds to respond); (3) under total usage limits (e.g., OpenAI has hard per-day and per-month limits on API accesses). Furthermore, it is often overlooked that evaluating the responses of an LLM for different candidate prompts can be costly as many tasks (e.g., writing improvement, mathematical reasoning, etc.) would require human (and sometimes domain expert) opinions. As a result, the prompt optimization process can incur an unaffordable cost without a proper selection method.

To make the learning process more accessible, this work proposes to study prompt optimization under an explicitly imposed budget constraint when interacting with the targeted LLM, in addition to the previously considered requirements (e.g., discrete, interpretable, and black-box). To the best of our knowledge, budget constraints are only briefly mentioned in Zhou et al. [90], Pryzant et al. [60], and there are no systematic or principled investigations of how to address the limited budget constraint in prompt optimization. The main contributions of this work are summarized as follows.

· The constraint of a limited budget is explicitly introduced into prompt optimization, which has been largely ignored before. As most of the prompt optimization methods rely on selecting from a pre-generated candidate prompt pool, we focus our study on how to carefully allocate budgets to test each candidate prompt so that the optimal one can be learned efficiently and effectively.

· We propose a general solution framework, termed TRIPLE (bes T a R m I dentification for P rompt LE arning), by establishing a novel connection between prompt optimization and multi-armed bandits (MAB) [41]. In particular, we focus on harnessing the power of fixed-budget best arm identification (BAI-FB) [4, 37] to address prompt optimization (especially, selection) with a limited budget constraint. Two representative designs TRIPLE-SH and TRIPLE-CR , inspired by celebrated BAIFB algorithms, are presented. To improve scalability, two enhanced methods, TRIPLE-CLST and TRIPLE-GSE , are further proposed, where prompt embeddings are leveraged by exploiting the ideas of clustering and function approximation to accelerate the learning process.

· Extensive experimental results are reported using well-adopted prompt tasks and varying LLMs to demonstrate the superiority of TRIPLE over previous baselines. In particular, on GPT3.5 and Llama2, compared with baseline methods also not using prompt embeddings, the basic TRIPLE-SH and TRIPLE-CR achieves performance improvements by (on average) 3% to 16% . When leveraging prompt embeddings, the enhanced TRIPLE-CLST and TRIPLE-GSE also outperform corresponding baselines by (on average) 10% to 56% with fewer prompts than budget and (on average) 16% to 45% with more prompts than budget. The gains are further evidenced on other LLMs, i.e., Gemma and Mistral. Moreover, the proposed methods can be directly plugged into two popular prompt optimization pipelines, APE [90] and APO [60], with end-to-end performances significantly improved over their original implementations.

· This work extends broadly to providing a new perspective of prompt optimization from MAB, and also a new application scenario of MAB in prompt optimization. This established connection may spark further innovations in both fields. As one concrete example, we extend the study to optimizing the selection of examples in few-shot prompts [9], which can be recognized as a BAI-FB problem in the setup of combinatorial bandits [14, 12]. Experimental results illustrate that the extensions of TRIPLE achieve superior performance, demonstrating its rich potential.

Key Related Works. We discuss a few works that explicitly or implicitly touch upon the selection efficiency in prompt optimization, and a complete literature review can be found in Appendix A. First, Zhou et al. [90] discusses a naive filtering strategy without theoretical or empirical justifications. Chen et al. [13] leverages Bayesian optimization (BO) with expected improvement (EI) as the acquisition function to select continuous soft prompts. BO can be viewed as similar to BAI while mostly focusing on infinite-arm cases [62]. Moreover, Pryzant et al. [60], Lin et al. [48] use specific MAB methods targeting regret minimization to perform prompt selection, which, as further illustrated in Sec. 3.3, are not well-suited as they optimize the cumulative selection performance over a period instead of the final selection output. Thus, compared with this work, existing investigations either lack a comprehensive discussion of the connection between prompt optimization and MAB or choose unsuitable MAB techniques to tackle prompt optimization. Moreover, as illustrated in Sec. 5, the TRIPLE solution outperforms the previously adopted methods empirically.

2 Prompt Optimization under a Limited Budget

Following Zhou et al. [90], Chen et al. [13], we present a concrete formulation of the problem of prompt optimization. Consider that we are using an LLM f ( · ) , which provides a mapping from any input X ∈ V to a distribution ∆ V over the language space V . The answer ˆ Y ∈ V given by the LLM is assumed to be sampled from f ( X ) as ˆ Y ∼ f ( X ) . Note that instead of treating f ( · ) as a deterministic function providing a specific output answer, we generally consider the practical setting where the answers of LLM exhibit a certain level of randomness.

For prompt optimization, we aim to find a prompt p such that when concatenated with inputs X of a certain task (i.e, as [ p ; X ] ), it provides good performance in expectation with respect to the input distribution I X and the inherent randomness of LLM f ( · ) . The performance is measured as

where s ( X, ˆ Y ) denotes a score function that measures the quality of the output ˆ Y for the input X .

Motivated by the common usage scenario of LLMs, recent studies have imposed several constraints on this learning problem [90, 58, 13, 28], where the three key ones are (I) black-box : the method can be applied to black-box LLMs, i.e., only have access to an API f ( · ) and no access to the intermediate structure or parameters inside (including gradients, output likelihood, etc.); (II) discrete : the learned prompt must be discrete characters, instead of continuous values (i.e., soft prompts); and (III) interpretable : the learned prompt must be understandable by humans, instead of gibberish words.

Intuitively, the process of learning a good prompt requires interactions with the LLM (i.e., sample ˆ Y ∼ f ([ p ; X ]) and evaluating its responses (i.e., obtain score s ( X, ˆ Y ) ). However, as mentioned in Sec. 1, such interactions and evaluations are costly. Thus, besides the aforementioned constraints, we further explicitly take into account that the prompt optimization process should have (IV) a limited budget : the total number of trials with the LLM that happen during the learning is at most N . Finally, the prompt optimization problem considered in this work can be formulated as:

finding p ∗ with high performance µ ( p ∗ ) under constraints of black-box, discrete, interpretable, and a limited budget.

Directly tackling this prompt optimization problem has been widely recognized as challenging even without the constraint of a limited budget [50]. As highlighted in Pryzant et al. [60], Chen et al. [13], it essentially requires performing a black-box discrete optimization. Instead, many proposed methods rely on the pipeline of first generating a pool of candidate prompts and then selecting from it [35, 90, 79, 59, 28]. The prompt generation can either be performed manually or follow designed automatic protocols. For example, the famous APE design [90] selects from prompts generated by an LLM using demonstrations. From a unified perspective, we can simplify the problem into generating a pool of prompts P and finding the optimal prompt in it:

While many efforts have been devoted along this line, we recognize that they are largely focused on how to generate prompts, while limited attention has been paid to how to select from the already generated prompts (as mentioned in Sec. 1 and further discussed in Appendix A). Naive treatments, such as uniformly evaluating all prompts, are understandable since budget limitations are not considered previously, i.e., unlimited evaluations can be performed. With an explicit budget limitation, however, we need to carefully allocate the budgets to each prompt so that the optimal prompt (or at least a sufficiently good one) can be correctly learned, which is the main focus of this work. An overview of the

considered prompt optimization pipeline and our focus is illustrated in Fig. 1.

3 Connecting Prompt Optimization with Best Arm Identification

We provide a new perspective of prompt optimization through the lens of tools in multi-armed bandits (MAB) [41, 40]. In particular, prompt optimization under a limited budget is shown to be intrinsically aligned with the problem of fixed-budget best-arm identification (BAI-FB) [4, 37]. In the following, a brief introduction to MAB is first provided. Then, the connection between prompt optimization (especially selection) and MAB (especially BAI-FB) is established. Based on this connection, we propose to fully leverage the rich toolbox from BAI-FB to perform efficient prompt optimization.

3.1 Multi-armed Bandits

The research of multi-armed bandits (MAB) has a long and rich history; see representative surveys of Lattimore and Szepesvári [41], Bubeck et al. [10]. The most basic form of MAB, i.e., the finite-armed stochastic bandits, considers a system with a set K finite arms (i.e., actions) that provide stochastic rewards when pulled. When interacting with the system, the agent can select one arm k ∈ K to pull at each time, and she receives a stochastic reward: r k ∼ dist k ( ν k ) , where dist k ( ν k ) denotes the action k 's reward distribution with an unknown expectation ν k .

The learning objective of the agent in MAB can be roughly divided into two categories: (1) regret minimization , which maximizes the expected cumulative rewards collected by the agent [5, 3, 26]; (2) best arm identification , which targets at outputting the best arm k ∗ = arg max k ∈K ν k [4, 27, 32]. These two objectives often require different learning strategies. Regret minimization typically relies on a carefully designed balance between exploration (i.e., obtaining new information) and exploitation (i.e., collecting higher rewards based on the previous information). Best arm identification, on the other hand, is also called pure exploration as it only focuses on obtaining information to find the best arm. We here particularly note that although the designs targeting regret minimization often can converge to the optimal arm k ∗ given a sufficient period of time, they are known to be inefficient for the objective of best arm identification in the MAB studies.

3.2 A Bandit View of Prompt Optimization

Based on the above introduction, it can be intuitively understood that the prompt optimization (especially, selection) problem can be mapped into an MAB setting:

The pool of candidate prompts P is equivalent to the set of arms K ;

Using a prompt p to interact with LLM can be viewed as selecting a bandit arm k to pull in MAB;

The feedback of the score function, i.e., s ( X, ˆ Y ) , provides the reward signal r k , where dist k ( ν k ) characterizes the randomness of X ∼ I X and ˆ Y ∼ f ([ p ; X ]) . The expected performance µ ( p ) is the counterpart of the expected reward ν k in MAB.

It can be further recognized that the target of prompt optimization is more suitable to be captured as the best arm identification (BAI) problem, instead of a regret minimization one, as it only cares about finding the optimal prompt p ∗ instead of the cumulative performance of interactions performed during the learning process.

With the relationship between prompt optimization and BAI established, we further consider the constraint of learning under a limited budget. We argue that this aligns with one of the main research directions in BAI called fixed-budget best arm identification (BAI-FB) [37, 75, 22]. BAI-FB particularly considers the problem of maximizing the probability of correctly identifying the best arm k ∗ while not pulling arms more than T times. It can be observed that this formulation matches the goal of prompt optimization under a limited budget; thus BAI-FB provides

a perfect toolbox to enhance the commonly required prompt selection process. The connection between prompt optimization and MAB, in particular, BAI-FB, is further illustrated in Table 1. To avoid confusion, in the remainder of this paper, we will adopt the notation of prompt optimization as introduced in Sec. 2.

3.3 Harnessing the Power of BAI-FB

As mentioned, we recognize that prompt optimization under a limited budget is a matching application scenario for BAI-FB. In this paper, we propose a general framework called TRIPLE (bes T a R m I dentification for P rompt LE arning) to harness the power of BAI-FB in solving the prompt optimization problem. This is possible because BAI-FB has witnessed significant development over the years, with several efficient designs being proposed. As a first step, we choose two popular and successful BAI-FB schemes and implement them for prompt optimization, which are briefly described below. Their complete descriptions are provided in Algs. 2 and 3 of Appendix C.

Sequential Halving (SH). SH is one of the first provably efficient BAI-FB designs [37] and remains popular after a decade of its proposal. It follows a protocol that divides the total budget N into ⌈ log 2 ( |P| ) ⌉ equal-length phases. In each phase, SH uniformly tries all active prompts (initialized as P ) and eliminates half of them with the lower sample means for the next phase. The final active arm is output as the identified optimal prompt.

Continuously Reject (CR). CR is a recently proposed method [75], which can be viewed as an extension of the classical Successively Reject (SR) design [4]. It uniformly explores active prompts (initialized as P ) and performs potential elimination of poorly-performed prompts after each pull. The elimination is based on carefully designed criteria using the Large Deviation Principle. It can be observed that, without the phased structure, CR is more adaptive than SH (and SR), which makes it appealing both theoretically and practically.

While MAB has found broad applications in recommender systems [44], healthcare [63], wireless communications [24], and beyond [8], a systematical connection between MAB and prompt optimization has not been established before to the best of our knowledge, which may spark new research activities (see discussions in Sec. 7). In addition, although SH and CR are selected as the representatives, the connection between prompt optimization and MAB is fundamental. Any existing or forthcoming BAI-FB designs can be flexibly incorporated into TRIPLE , e.g., the Bayesian perspective provided in Komiyama et al. [38], Atsidakou et al. [2]

Remark 3.1 . As mentioned in Sec. 1, Pryzant et al. [60], Lin et al. [48] leverage specific MAB designs to perform prompt selection without a comprehensive discussion as above on their connection. Moreover, Pryzant et al. [60] argues that UCB [5] is suitable, while Lin et al. [48] also uses a UCBvariant, NeuralUCB [89], as the core method. However, both of UCB and NeuralUCB are designed for regret minimization (i.e., optimizing the cumulative interaction performance during learning). As illustrated in Sec. 3.1, designs for regret minimization cannot achieve optimal performance for the goal of identifying the optimal arm (i.e., BAI), which thus are not well-suited for prompt optimization.

4 Handling Large Candidate Pools via Prompt Embeddings

The connection built in the last section provides us with the core idea of leveraging BAI-FB designs to tackle prompt optimization. As having been theoretically established [4], solving BAI-FB without additional structures, however, will unavoidably incur an identification error that is positively related to the number of candidate prompts |P| . In other words, given a larger pool of prompts, it becomes harder to find the optimal prompt with the basic BAI-FB designs, which restricts their applicability to practical prompt optimization problems (where possibly the number of prompts exceeds the budget).

The key reason behind this is that each candidate prompt is treated independently in the basic BAI-FB. Thus, budgets need to be assigned to all the prompts and no information can be shared among them, which is often not the case in prompt optimization. For a prompt optimization problem, the underlying task is often stable, e.g., rewriting emails, constructing TLDR, etc. The candidate prompts, regardless of their generation methods, should all reflect the purpose of the underlying task and thus share similarities. For example, the candidate prompts generated via demonstrating LLMs [90] often share similar structures and differ only in a few words or word orders.

With the above observation, we target sharing information among prompts during learning. To achieve this, we propose to leverage an embedding model, denoted as embed : V → R d , to obtain the sentence embedding of the prompts: e ( p ) := embed ( p ) ∈ R d , E := { e ( p ) : p ∈ P} , where d refers to the embedding dimension. In the experiments, the OpenAI embedding API is adopted while, in general, any sufficiently expressive models can be incorporated. Also, due to this flexibility, using embedding models is fundamentally different from requiring a white-box LLM [13, 48]. With the

obtained prompt embeddings, we propose two useful enhancements to further improve the learning effectiveness when the pool of candidate prompts is large.

4.1 Leveraging Similarities via Clustering

Since the key challenge is a large pool of candidate prompts, an intuitive idea is to effectively decrease the size of the pool. We thus propose a two-phased BAI-FB scheme for prompt optimization. In Phase I, the entire pool of candidate prompts is clustered into several groups based on their embeddings, and BAI-FB is performed on the clusters with an initial target of finding the optimal cluster (or the few good clusters). Then, in Phase II, BAI-FB is performed on the prompts

Algorithm 1 TRIPLE-CLST

1: Input: the pool of candidate prompts P and their embeddings E , overall budget N , Phase I budget N 1 , number of clusters L

2: Cluster P into clusters C = {C 1 , · · · , C L } based on embeddings E (e.g., via k -means)

3: Obtain ̂ C ∗ ← BAI-FB ( C , N 1 ) { Phase I }

4: Obtain ˆ p ∗ ← BAI-FB ( ̂ C ∗ , N -N 1 ) { Phase II }

5: Output: prompt ˆ p ∗

in the optimal cluster with the target of identifying one final prompt. For both phases, different BAIFB designs can be incorporated, e.g., SH and CR. The entire procedure, referred to as TRIPLE-CLST , is described in Alg. 1.

The effectiveness of TRIPLE-CLST relies on the clustering results produced in Phase I. Ideally, prompts with similar performances should be clustered together. Then, Phase I can quickly eliminate the prompts with poor performances, leaving a small pool of good prompts for Phase II to process. In the experiments, this intuitive phenomenon is indeed observed. In particular, in Fig. 2, as expected, prompts in the same cluster share similar performances. In particular, it can be observed that the prompts in the same cluster (i.e., the same color and shape) share similar performance (i.e., similar sizes). Especially, the optimal prompt (marked by the red star) is

clustered together with a few prompts with comparably near-optimal performances.

4.2 Sharing Information via Function Approximation

Besides clustering, another idea to incorporate the prompt embeddings is to learn a common function (e.g., an MLP) to predict the prompt performances based on their embeddings. Similar ideas of function approximation have also been widely adopted in MAB literature to share information among large action spaces, with functions ranging from linear ones [1, 82] to neural networks [91, 89]. In the setting considered in this work, we adopt a recently developed BAI-FB scheme as described in the following as TRIPLE-GSE , with details provided in Alg. 4 of Appendix C.

GSE. The general phased elimination flow of SH described in 3.3 is inherited. The major difference is that SH uses sample means to perform eliminations. GSE [6], on the other hand, leverages collected samples from previous phases to train a reward function g θ ( · ) : R d → R that maps prompt embeddings to the predicted performance, which is further used to eliminate prompts.

5 Experiments

In this section, extensive experimental results are reported to evaluate the efficiency of TRIPLE across diverse prompting tasks from two standard datasets: Instruction-Induction [30] and BigBench [69]. The results reported in this section are mainly collected from GPT-3.5, Llama2, Gemma, and Mistral (see the specific model numbers listed in Appendix E.1). Full experimental details can be found in Appendix E. The complete results of 47 tasks are reported in Appendix F, while here we particularly focus on 12 representative tasks, which are not too hard (i.e., all generated prompts achieve nearzero performances) or too easy (i.e., all generated prompts achieve near-one performances). The experimental codes can be found at https://github.com/ShenGroup/TRIPLE .

5.1 Evaluating TRIPLE with Fixed Prompt Pools

As TRIPLE main focuses on the prompt selection component, we perform initial evaluations in an isolated fashion of selecting from fixed pools of candidate prompts. For this experiment, candidate pools of prompts are generated following the well-established APE design [90] with a high LLM temperature to ensure randomness. Then, under a limited budget, the performances of TRIPLE algorithms are compared with the following four baselines, where the latter two (i.e., BO and NeuralUCB) leverage prompt embeddings:

Uniform. Many previous designs choose to evaluate the entire candidate pool on all development data [28, 59] which corresponds to uniformly dividing the total budget to test all prompts.

UCB. The upper confidence bound (UCB) method is a famous design for regret minimization in MAB. We evaluate UCB using its vanilla version from Auer et al. [5], which is reported to have good performance in Pryzant et al. [60].

BO. Bayesian optimization (BO) with expected improvement (EI) acquisition function is adopted in Chen et al. [13], which assumes a Gaussian process prior specified by prompt embeddings to perform posterior updates and makes selection to maximize EI. To further examine the performance of BO, another acquisition function, i.e., probability of improvement (PI), is also adopted.

NeuralUCB. Lin et al. [48] uses NeuralUCB [89] to perform prompt selection, which extends UCB by training a reward function to predict prompt performances based on embeddings.

Cause

and

effect

Common

concept

Disambiguation

qa

Gender inc. DE

Hyperbaton

Larger

animal

Movie

recommendation

Object

counting

Starts with

Question

selection

Rhymes

Snarks

0.0

0.6

1.2

1.8

|

| = 30

N

=150

UCB

TRIPLE-SH

TRIPLE-CR

BO-EI

BO-PI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

Cause

and

Common

concept

Disambiguation

qa

Gender inc. DE

Hyperbaton

Larger

animal

Movie

recommendation

Object

counting

Starts with

Question

selection

Rhymes

Snarks

0.0

0.6

1.2

1.8

2.2

3.1

1.9

3.3

|

| = 30

N

=150

effect

Cause

and

effect

Common

concept

Disambiguation

qa

Gender inc. DE

Hyperbaton

Larger

animal

Movie

recommendation

Object

counting

Starts with

Question

selection

Rhymes

Snarks

0.0

0.6

1.2

1.8

2.3

2.2

2.1

2.6

2.0

|

| = 30

N

=150

BO-EI

BO-PI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

Cause

and

effect

Common

concept

Disambiguation

qa

Gender inc. DE

Hyperbaton

Larger

animal

Movie

recommendation

Object

counting

Starts with

Question

selection

Rhymes

Snarks

0.0

0.6

1.2

1.8

1.9

|

| = 30

N

=150

Performance with fewer prompts than budget. We first test candidate pools with 30 prompts per task. Results reported in Fig. 3(a) reflect the selection performance with an overall budget of 150 . It can be observed that TRIPLE-SH and TRIPLE-CR achieve better performance than Uniform ( 15% and 12% improvements on average for GPT-3.5; 15% and 16% for Llama2) and UCB ( 5% and 3% improvements on average for GPT-3.5; 6% and 7% for Llama2). Moreover, for methods using

prompt embeddings, the enhanced TRIPLE-CLST and TRIPLE-GSE also demonstrate remarkable improvements over BO-EI ( 11% and 10% on average for GPT-3.5; 56% and 52% for Llama2) and NeuralUCB ( 17% and 17% improvements on average for GPT-3.5; 26% and 27% for Llama2). These results empirically evidence the superiority of TRIPLE with or without prompt embeddings.

Performance with more prompts than budget. In the above test, the budget is larger than the number of candidate prompts. We further perform experiments in a more difficult setting, i.e., there are more prompts than the budget. In particular, candidate pools with 150 prompts per task are generated, and the overall budget is set as 100 . In this scenario, only the methods that can leverage embeddings (i.e., BO, NeuralUCB, TRIPLE-CLST , TRIPLE-GSE ) can be used, as otherwise the total budget is not sufficient to provide even one evaluation to initiate the performance estimation of each candidate prompt. Results are reported in Fig. 3(b). In particular, it can be observed that TRIPLE-CLST and TRIPLE-GSE significantly improve over BO-EI ( 21% and 28% on average for GPT-3.5; 31% and 42% for Llama2) and NeuralUCB ( 38% and 45% on average for GPT-3.5; 26% and 16% for Llama2).

Asummary of the averaged performance ranks of the baselines and TRIPLE is listed in Table 2, which contains results on four LLMs (i.e., GPT-3.5, Llama2, Mistral, Gemma). It can be observed that in varying setups and with different LLMs, the proposed TRIPLE methods consistently obtain better performances than the previous baselines, remarking its efficiency and broad applicability.

Impact of the total budget. For a more comprehensive understanding, using candidate pools with 30 prompts, we further examine the impact of budgets, starting with 5 evaluations per prompt on average (i.e., 150 overall as adopted in Fig. 3(a)), and then gradually increasing to 30 (i.e., 900 overall, which is the same as the experiments in Zhou et al. [90]). From the results shown in Fig. 4, we see that the improvements of TRIPLE over baselines are more pronounced with lower budgets. In particular, with a budget of 10 evaluations per prompt on average (i.e., 300 overall), TRIPLE-CR , TRIPLE-CLST and TRIPLE-GSE maintain notable 9 . 7% , 13 . 5% and 17 . 4% improvement over Uniform, respectively; when the budget escalates to 20 evaluations per prompt on average, TRIPLE-CLST and TRIPLE-GSE still achieve an approximate 8% improvement. Once the budget reaches 30 evaluations per prompt on average (i.e., 900 overall), all methods provide approximately the same performance as they can all identify the optimal prompts under this generous budget.

Impact of the prompt pool size. Moreover, we investigate the prompt selection performance under prompt pools with different sizes. First, while Figs. 3(a), 3(b) and Table 2 has demonstrated the superiority of TRIPLE with 30 and 150 prompts, we further enlarge the size of prompt pool size to 1000 and consider an overall budget of 500 . The results reported in Fig. 5 further illustrate that the improvement of TRIPLE over the baselines is consistent across the sizes of prompt pools. Also, to benefit empirical usage, we take a deep look into whether larger prompt pools are necessary to provide better candidates. From Fig. 6, it can be observed that actually the prompt performance distributions do not vary much with the pool size increased from 100 to 1000 , indicating that generating a sufficiently large prompt pool (e.g., 100 ) is enough to further perform the selection and find the final prompt candidate to use.

5.2 Integrating TRIPLE into End-to-End Pipelines

We now explore whether TRIPLE can provide performance improvements when plugged into end-toend prompt optimization pipelines that include both prompt generation and selection. To this end,

5

10

15

20

25

30

Budget

0

5

10

15

20

25

30

Relative Improvement (%)

UCB

TRIPLE-SH

TRIPLE-CR

TRIPLE-GSE

TRIPLE-CLST

Larger

Movie

0.0

0.6

1.2

NeuralUCB

BO-EI

BO-PI

TRIPLE-GSE

TRIPLE-CLST

animal recommendation

0.0

0.2

0.4

0.6

0.8

1.0

Scores

0%

4%

7%

11%

14%

18%

21%

Density

Number of Prompts

100 prompts

500 prompts

700 prompts

1000 prompts

0.0

0.2

0.4

0.6

0.8

1.0

Scores

0%

5%

10%

14%

19%

24%

Density

Number of Prompts

100 prompts

500 prompts

700 prompts

1000 prompts

two end-to-end designs are considered, aiming to assess the performance of TRIPLE in more fluid and iterative settings, which are discussed in the following with our implementation details.

APE. Proposed by Zhou et al. [90], the APE pipeline lets LLMs generate prompt candidates and then selects from them. In our experiments, for each task, following original templates, 30 prompts are generated, followed by different methods to perform selection with a budget of 5 evaluations per prompt on average (i.e., 150 LLM accesses overall). Zhou et al. [90] suggest a non-iterative version with uniform evaluations of prompts, which is taken as the baseline here.

APO. The APO pipeline [60] is an iterative one, letting LLMs criticize the previous prompts. Here, following the original templates, three iterations are performed and 10 prompts are generated per iteration. Different selection methods are then tested with a budget of 50 per iteration so that an overall budget of 150 is used, aligning with that of APE. Pryzant et al. [60] have reported UCB as the most effective prompt selection method, which is adopted as the baseline here. We note that OPRO [81] shares a similar iterative scheme as APO while using a different component to improve prompts. Due to their similarity, the experiments are mainly focused on APO here, while TRIPLE can also be flexibly integrated with OPRO.

The end-to-end experiment results are reported in Table 3, which reveal the consistently better performance of TRIPLE over the originally adopted baseline methods. This observation highlights the applicability and flexibility of the TRIPLE framework, i.e., it can benefit any prompt optimization pipelines requiring a selection component.

6 Extension: Selections of Examples for Few-shot Prompts

Based on the general connection between prompt optimization and BAI-FB, the power of TRIPLE can be further extended beyond finding one good instructional prompt. In the following, we provide discussions on how to leverage TRIPLE to efficiently select examples for few-shot prompts.

As noticed in Brown et al. [9], LLMs can perform varying tasks when prompted with several related examples, i.e., few-shot prompting. It has been widely recognized that a good choice of examples in few-shot prompts is important to obtain good downstream performances [49, 52]. Using the terminology introduced in Sec. 1, we can formulate the problem of example selection as follows. From a set of examples G , we target at selecting M examples ( g 1 , · · · , g M ) to form a few-shot prompt, whose performance is measured as µ ( g 1 , · · · , g M ) := E X ∼I X E ˆ Y ∼ f ([ g 1 , ··· ,g M ; X ]) [ s ( X, ˆ Y )] . The optimal selection of examples can be defined as ( g ∗ 1 , · · · , g ∗ M ) := arg max g 1 , ··· ,g M ∈G µ ( g 1 , · · · , g M ) .

From the MAB perspective, the learning target can be interpreted as finding the optimal combination of M arms from the overall arm set G , which is the focus of the study on combinatorial MAB (CMAB) [16, 15, 18]. Then, TRIPLE can be further extended to incorporate BAI-FB designs from CMAB to perform the desired example selection. In particular, based on some heuristics on the performance µ ( g 1 , · · · , g M ) , TRIPLE-SAR and TRIPLE-CSAR are proposed, extending Chen et al. [14], Gabillon et al. [23], which are further discussed in Appendix D. The performances of these extensions are presented in Table 4, with more details and results provided in Appendix E.6 and F.

7 Conclusions

Prompt optimization is an important problem for large language models (LLMs), but prior research has not considered the potential cost during prompt selection. We have explicitly incorporated a budget constraint to prompt optimization, and studied the problem of how to efficiently select prompts with the given budget. A systematical connection between multi-armed bandits (MAB) and prompt optimization was established. Through this lens, we proposed a general framework, termed TRIPLE , to fully harness the power of fixed-budget best arm identification (BAI-FB) to perform prompt optimization. Besides standard BAI-FB designs, two embedding-based enhancements were proposed to accelerate learning. Extensive experimental results demonstrated the superiority of TRIPLE over multiple representative tasks and various targeted LLMs. Furthermore, we showed that TRIPLE could be plugged into popular end-to-end prompt optimization pipelines, with better performance than previous implementations, demonstrating its effectiveness and flexibility.

In addition to the technical contributions, we believe that the connection between prompt optimization and MAB may be of broader interest. It not only provides a rich set of tools from MAB to advance prompt optimization but also introduces a new application scenario for MAB (especially BAI) research. In particular, the discussed extension to the selection of examples for few-shot prompts demonstrates the rich potential of TRIPLE . As future steps, the research on contextual bandits [44] may provide insights into selecting input-dependent prompts [78]. Also, the application of prompt optimization may spark new research efforts in MAB, e.g., efficient BAI methods for correlated arms.

Acknowledgments and Disclosure of Funding

The work of CSs, KY, and ZC was supported in part by the US National Science Foundation (NSF) under awards CNS-2002902, ECCS- 2029978, ECCS-2143559, and CNS-2313110, and the Bloomberg Data Science Ph.D. Fellowship. The work of JY was supported in part by the US NSF under awards CNS-1956276 and CNS-2114542.

References

Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24.

Atsidakou, A., Katariya, S., Sanghavi, S., and Kveton, B. (2022). Bayesian fixed-budget best-arm identification. arXiv preprint arXiv:2211.08572 .

Audibert, J.-Y., Bubeck, S., et al. (2009). Minimax policies for adversarial and stochastic bandits. In COLT , volume 7, pages 1-122.

Audibert, J.-Y., Bubeck, S., and Munos, R. (2010). Best arm identification in multi-armed bandits. In COLT , pages 41-53.

Auer, P., Cesa-Bianchi, N., and Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256.

Azizi, M. J., Kveton, B., and Ghavamzadeh, M. (2023). Fixed-budget best-arm identification in structured bandits. arXiv preprint arXiv:2106.04763 .

Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B. (2011). Algorithms for hyper-parameter optimization. Advances in neural information processing systems , 24.

Bouneffouf, D. and Rish, I. (2019). A survey on practical applications of multi-armed and contextual bandits. arXiv preprint arXiv:1904.10040 .

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901.

Bubeck, S., Cesa-Bianchi, N., et al. (2012). Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122.

Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y ., Lundberg, S., et al. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 .

Bubeck, S., Wang, T., and Viswanathan, N. (2013). Multiple identifications in multi-armed bandits. In International Conference on Machine Learning , pages 258-265. PMLR.

Chen, L., Chen, J., Goldstein, T., Huang, H., and Zhou, T. (2023). Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03082 .

Chen, S., Lin, T., King, I., Lyu, M. R., and Chen, W. (2014). Combinatorial pure exploration of multi-armed bandits. Advances in neural information processing systems , 27.

Chen, W., Hu, W., Li, F., Li, J., Liu, Y ., and Lu, P. (2016). Combinatorial multi-armed bandit with general reward functions. Advances in Neural Information Processing Systems , 29.

Chen, W., Wang, Y., and Yuan, Y. (2013). Combinatorial multi-armed bandit: General framework and applications. In International conference on machine learning , pages 151-159. PMLR.

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 .

Combes, R., Talebi Mazraeh Shahi, M. S., Proutiere, A., et al. (2015). Combinatorial bandits revisited. Advances in neural information processing systems , 28.

Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H., Shu, T., Song, M., Xing, E. P., and Hu, Z. (2022). Rlprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205.12548 .

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 .

Diao, S., Huang, Z., Xu, R., Li, X., Lin, Y ., Zhou, X., and Zhang, T. (2022). Black-box prompt learning for pre-trained language models. arXiv preprint arXiv:2201.08531 .

Gabillon, V., Ghavamzadeh, M., and Lazaric, A. (2012). Best arm identification: A unified approach to fixed budget and fixed confidence. Advances in Neural Information Processing Systems , 25.

Gabillon, V., Ghavamzadeh, M., Lazaric, A., and Bubeck, S. (2011). Multi-bandit best arm identification. Advances in Neural Information Processing Systems , 24.

Gai, Y., Krishnamachari, B., and Jain, R. (2012). Combinatorial network optimization with unknown variables: Multi-armed bandits with linear rewards and individual observations. IEEE/ACM Transactions on Networking , 20(5):1466-1478.

Gao, T., Fisch, A., and Chen, D. (2020). Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723 .

Garivier, A. and Cappé, O. (2011). The kl-ucb algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th annual conference on learning theory , pages 359-376. JMLR Workshop and Conference Proceedings.

Garivier, A. and Kaufmann, E. (2016). Optimal best arm identification with fixed confidence. In Conference on Learning Theory , pages 998-1027. PMLR.

Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. (2023). Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532 .

Hinton, G. E. and Roweis, S. (2002). Stochastic neighbor embedding. In Becker, S., Thrun, S., and Obermayer, K., editors, Advances in Neural Information Processing Systems , volume 15. MIT Press.

Honovich, O., Shaham, U., Bowman, S. R., and Levy, O. (2022). Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.10782 .

Hu, W., Shu, Y., Yu, Z., Wu, Z., Lin, X., Dai, Z., Ng, S.-K., and Low, B. K. H. (2024). Localized zeroth-order prompt optimization. arXiv preprint arXiv:2403.02993 .

Jamieson, K. and Nowak, R. (2014). Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. In 2014 48th Annual Conference on Information Sciences and Systems (CISS) , pages 1-6. IEEE.

Jedra, Y. and Proutiere, A. (2020). Optimal best-arm identification in linear bandits. Advances in Neural Information Processing Systems , 33:10007-10017.

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. (2023). Mistral 7b.

Jiang, Z., Xu, F. F., Araki, J., and Neubig, G. (2020). How can we know what language models know? Transactions of the Association for Computational Linguistics , 8:423-438.

Kanarios, K., Zhang, Q., and Ying, L. (2024). Cost aware best arm identification. arXiv preprint arXiv:2402.16710 .

Karnin, Z., Koren, T., and Somekh, O. (2013). Almost optimal exploration in multi-armed bandits. In International conference on machine learning , pages 1238-1246. PMLR.

Komiyama, J., Ariu, K., Kato, M., and Qin, C. (2023). Rate-optimal bayesian simple regret in best arm identification. Mathematics of Operations Research .

Kone, C., Kaufmann, E., and Richert, L. (2024). Bandit pareto set identification: the fixed budget setting. In International Conference on Artificial Intelligence and Statistics , pages 2548-2556. PMLR.

Lai, T. L. and Robbins, H. (1985). Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22.

Lattimore, T. and Szepesvári, C. (2020). Bandit algorithms . Cambridge University Press.

Lester, B., Al-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 .

Levy, I., Bogin, B., and Berant, J. (2023). Diverse demonstrations improve in-context compositional generalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1401-1422.

Li, L., Chu, W., Langford, J., and Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670.

Li, X., Lv, K., Yan, H., Lin, T., Zhu, W., Ni, Y ., Xie, G., Wang, X., and Qiu, X. (2023). Unified demonstration retriever for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 4644-4668.

Li, X. L. and Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 .

Lin, X., Dai, Z., Verma, A., Ng, S.-K., Jaillet, P., and Low, B. K. H. (2024). Prompt optimization with human feedback. arXiv preprint arXiv:2405.17346 .

Lin, X., Wu, Z., Dai, Z., Hu, W., Shu, Y., Ng, S.-K., Jaillet, P., and Low, B. K. H. (2023). Use your instinct: Instruction optimization using neural bandits coupled with transformers. arXiv preprint arXiv:2310.02905 .

Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. (2021a). What makes good in-context examples for gpt3 ? arXiv preprint arXiv:2101.06804 .

Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35.

Liu, X., Ji, K., Fu, Y., Tam, W. L., Du, Z., Yang, Z., and Tang, J. (2021b). P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 .

Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. (2021). Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786 .

Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 .

Mishra, S., Khashabi, D., Baral, C., Choi, Y., and Hajishirzi, H. (2021). Reframing instructional prompts to gptk's language. arXiv preprint arXiv:2109.07830 .

OpenAI (2023a). Gpt-3.5-turbo. https://platform.openai.com/docs/models/ gpt-3-5 . Accessed: 2024-01-29.

OpenAI (2023b). Text-embedding-ada-002. https://platform.openai.com/docs/ models/embeddings . Accessed: 2024-01-29.

Opsahl-Ong, K., Ryan, M. J., Purtell, J., Broman, D., Potts, C., Zaharia, M., and Khattab, O. (2024). Optimizing instructions and demonstrations for multi-stage language model programs. arXiv preprint arXiv:2406.11695 .

Pan, R., Xing, S., Diao, S., Liu, X., Shum, K., Zhang, J., and Zhang, T. (2023). Plum: Prompt learning using metaheuristic. arXiv preprint arXiv:2311.08364 .

Prasad, A., Hase, P., Zhou, X., and Bansal, M. (2022). Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.07281 .

Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., and Zeng, M. (2023). Automatic prompt optimization with 'gradient descent' and beam search. arXiv preprint arXiv:2305.03495 .

Rubin, O., Herzig, J., and Berant, J. (2022). Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2655-2671.

Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and De Freitas, N. (2015). Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE , 104(1):148-175.

Shen, C., Wang, Z., Villar, S., and Van Der Schaar, M. (2020). Learning for dose allocation in adaptive clinical trials with safety constraints. In International Conference on Machine Learning , pages 8730-8740. PMLR.

Shi, W., Han, X., Gonen, H., Holtzman, A., Tsvetkov, Y., and Zettlemoyer, L. (2022). Toward human readable prompt tuning: Kubrick's the shining is a good movie, and a good prompt too? arXiv preprint arXiv:2212.10539 .

Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. (2020). Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 .

Soare, M., Lazaric, A., and Munos, R. (2014). Best-arm identification in linear bandits. Advances in Neural Information Processing Systems , 27.

Su, H., Kasai, J., Wu, C. H., Shi, W., Wang, T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L., Smith, N. A., et al. (2022). Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975 .

Sun, H., Hüyük, A., and van der Schaar, M. (2023). Query-dependent prompt evaluation and optimization with offline inverse rl. arXiv e-prints , pages arXiv-2309.

Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022). Challenging big-bench tasks and whether chain-ofthought can solve them. arXiv preprint arXiv:2210.09261 .

Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. (2024). Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 .

Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3/4):285-294.

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023a). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 .

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023b). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 .

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2019). GLUE: A multitask benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR.

Wang, P.-A., Tzeng, R.-C., and Proutiere, A. (2023). Best arm identification with fixed budget: A large deviation perspective. arXiv preprint arXiv:2312.12137 .

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837.

Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., and Goldstein, T. (2023). Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302.03668 .

Wu, Z., Wang, S., Gu, J., Hou, R., Dong, Y., Vydiswaran, V., and Ma, H. (2022). Idpg: An instance-dependent prompt generation method. arXiv preprint arXiv:2204.04497 .

Xu, H., Chen, Y., Du, Y., Shao, N., Wang, Y., Li, H., and Yang, Z. (2022). Gps: Genetic prompt search for efficient few-shot learning. arXiv preprint arXiv:2210.17041 .

Xu, S. and Zhang, C. (2024). Misconfidence-based demonstration selection for llm in-context learning. arXiv preprint arXiv:2401.06301 .

Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. (2023). Large language models as optimizers. arXiv preprint arXiv:2309.03409 .

Yang, J. and Tan, V. (2022). Minimax optimal fixed-budget best arm identification in linear bandits. Advances in Neural Information Processing Systems , 35:12253-12266.

Yavas, R. C. and Tan, V. Y. (2023). Fixed-budget best-arm identification in sparse linear bandits. arXiv preprint arXiv:2311.00481 .

Zhang, T., Wang, X., Zhou, D., Schuurmans, D., and Gonzalez, J. E. (2023a). Tempera: Testtime prompt editing via reinforcement learning. In The Eleventh International Conference on Learning Representations .

Zhang, Z., Wang, S., Yu, W., Xu, Y., Iter, D., Zeng, Q., Liu, Y ., Zhu, C., and Jiang, M. (2023b). Auto-instruct: Automatic instruction generation and ranking for black-box language models. arXiv preprint arXiv:2310.13127 .

Zhang, Z., Zhang, A., Li, M., and Smola, A. (2022). Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 .

Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning , pages 12697-12706. PMLR.

Zhong, Z., Friedman, D., and Chen, D. (2021). Factual probing is [mask]: Learning vs. learning to recall. arXiv preprint arXiv:2104.05240 .

Zhou, D., Li, L., and Gu, Q. (2020). Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning , pages 11492-11502. PMLR.

Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. (2022). Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910 .

Zhu, Y., Zhou, D., Jiang, R., Gu, Q., Willett, R., and Nowak, R. (2021). Pure exploration in kernel and neural bandits. Advances in neural information processing systems , 34:11618-11630.

A Related Works

Prompt Optimization. The study of prompt optimization (also known as instruction learning) focuses on automatically learning suitable prompts, which is more scalable compared with manual prompt engineering. Many efforts have been devoted to this direction [25, 77, 68, 86, 85], which is summarized in a recent survey of Liu et al. [50]. The early studies mostly consider optimizing soft prompts (i.e., continuous vectors) [42, 46, 88, 51, 78] or discrete but not interpretable prompts [65, 64] for white-box LLMs, where different gradient-guided optimization techniques are leveraged. The recent research efforts, instead, are more focused on considering the more practical setting of learning interpretable prompts for black-box LLMs [59, 90, 60, 79, 28, 58, 13], where the generatingthen-selecting pipeline in Fig. 1 is often adopted.

Compared with previous investigations, this work additionally considers a limited budget in an explicit fashion, which we believe is a practical but under-investigated concern. Most of the previous methods perform selection based on evaluating prompts on all development data [35, 79, 28, 59], which is unavoidably costly. APE [90] briefly touched on the cost issue by proposing a naive iterative top filtering strategy (which however is suggested to have only marginal benefits). APO [60] tested a few MAB methods, including two classical BAI-FB designs, i.e., SH [37] and SR [3], and reported that the performance of UCB is the more favorable. However, it neither formally introduces the budget constraint nor provides a systematical connection to BAI-FB as in this work. Also, this work goes much deeper in incorporating the state-of-the-art BAI-FB designs (i.e., CR [75] and GSE [6]) and proposing embedding-based enhancements. It is worth noting that INSTINCT [48] also incorporates one MAB method, i.e., NeuralUCB [89], to prompt optimization. However, as mentioned in Sec. 1, NeuralUCB is designed for regret minimization (instead of best arm identification), which is not suitable for learning the optimal prompt.

There are a few interesting concurrent works on topics that are worth further exploration. With the observation that finding a local optima is sufficient for many prompting tasks, Hu et al. [31] performs zeroth order optimization with an NTK-based derived Gaussian process. Opsahl-Ong et al. [57] studies the problem of prompt optimization in multi-stage LLM pipelines, where a Tree-structured Parzen Estimator [7] is adopted for selection. Lin et al. [47] extends the prompt optimization framework to consider preference feedback.

Multi-armed Bandits. Here we briefly discuss the representative studies of MAB, with comprehensive surveys available in Bubeck et al. [10], Lattimore and Szepesvári [41]. As mentioned in Sec. 3.1, the target of learning in a MAB system can be roughly categorized as regret minimization and best arm identification . The regret minimization designs target achieving a desired balance between exploration and exploitation so that the cumulative rewards are maximized, e.g., UCB [5] and Thompson sampling [71]. The best arm identification (BAI) designs are fully focused on exploration and can be further divided into two classes: fixed-budget (BAI-FB) and fixed-confidence (BAI-FC). The BAI-FB setting maximizes the probability of finding the best arm with a limited number of pulls [4, 37, 75, 6, 2, 82]. The BAI-FC setting is a dual one which focuses on minimizing the overall number of pulls while guaranteeing that the probability of finding the best arm is higher than a threshold [27, 32, 66, 33]. Besides leveraging BAI-FB as in this work, it is imaginable that BAI-FC designs can also find applications in prompt optimization, especially when valuing identification accuracy over incurred costs. While MAB has found wide success in different applications, this work marks the first time that a systematical connection between MAB and prompt optimization has been established to the best of our knowledge.

B Discussions

B.1 Broader Impacts

This work introduces TRIPLE , a framework that can perform efficient prompt optimization for large language models (LLMs) under limited budgets. By optimizing resource usage in prompt optimization for LLMs, we believe the proposed approach could make advanced AI tools and research more accessible to institutions and individuals with limited budgets, promoting a more equitable and democratized field of study. While acknowledging the need for responsible usage of the proposed method, we do not foresee major negative societal impacts.

B.2 Limitations and Future Works

This work opens an interesting direction on the connection between MAB and prompt optimization. In the following, we discuss a few aspects that are currently lacking in this work and particularly worth future explorations.

Prompt-specific costs. This work considers an abstract model where the cost during learning is measured by the number of LLM accesses. This model provides an important starting point to initialize the investigation. To make the study more practical, more refined considerations on costs can be incorporated. For example, the OpenAI API charge the interactions based on the number of input tokens, which means longer prompts incur higher costs. The cost-aware BAI studied in Kanarios et al. [36] can provide some insights to further consider prompt-specific costs.

Other BAI designs. Based on the connection between prompt optimization and BAI, this work has incorporated several BAI designs. However, the research on MAB has a long and rich history, where many other BAI designs can also be leveraged. For example, the Bayesian perspective provided in Komiyama et al. [38], Atsidakou et al. [2] and the function approximation scheme adopted in Yavas and Tan [83], Yang and Tan [82] are all worth investigation. Moreover, the multi-objective designs developed in Kone et al. [39] can be valuable extensions. This work is important in delivering the message that (both existing and forthcoming) BAI methods can benefit prompt optimization, which may inspire future explorations.

Structured prompts. In Sec. 6, we discuss how to extend TRIPLE to select examples for fewshot prompts. Based on the insights obtained in this work, we believe this direction is work further exploration. Moreover, other forms of structured prompting methods, such as Chain-of-Thoughts [76], are also interesting topics, which may further require multi-step techniques such as reinforcement learning.

C Details of TRIPLE Designs

The details of TRIPLE-SH (inspired by Karnin et al. [37]), TRIPLE-CR (inspired by Wang et al. [75]), and TRIPLE-GSE (inspired by Azizi et al. [6]) can be found in Algs. 2, 3, and 4, respectively.

Algorithm 2 TRIPLE-SH

1: Input: the pool of candidate prompts P , budget N

2: Initialization: set ˆ µ ( p ) ← 0 for all p ∈ P ; set the active prompt set A ← P

3: for phase p = 1 , · · · , ⌈ log 2 ( |P| ) ⌉ do

4: Interact with the targeted LLM using each prompt in A for ⌈ N/ ( |A|⌈ log 2 ( |P| ) ⌉ ) ⌉ times

5: Update the sample means { ˆ µ ( p ) : p ∈ A} using the collected samples

6: Update the active prompt set A as the set of ⌈A / 2 ⌉ prompts in the original A with the highest ˆ µ ( p )

7: end for

8: Output: the remaining active prompt ˆ p ∗

D Details of TRIPLE 's Extensions to Example Selection

In this section, we provide additional discussions on the extension to example selection mentioned in Sec. 6. It is first noted that the properties of good combinations of examples for few-shot prompts are a complicated topic and an active research problem [52, 49, 53, 80], which still lacks conclusive answers. The proposed designs are based on heuristics that are well-recognized and widely evidenced. With a deeper understanding of the few-shot prompt developed in later research, the perspective provided by TRIPLE and these designs would still be beneficial to guide corresponding modifications and extensions.

CSAR. First, we incorporate the intuitive heuristic that if one example leads to better performance as a one-shot prompt, it contributes positively to the overall few-shot performance [61, 45]. Based on this heuristic, we adapt the CSAR design [14, 12] to perform example selection, which can identify M prompts from G with the highest individual performances, i.e., µ ( g m ) . The details are provided in Alg. 5.

Algorithm 3 TRIPLE-CR

Algorithm 4 TRIPLE-GSE

1: Input: the pool of candidate prompts P and their embeddings E , budget N

2: Initialization: set ˆ µ ( p ) ← 0 for all p ∈ P ; set the active prompt set A ← P

3: for phase p = 1 , · · · , ⌈ log 2 ( |P| ) ⌉ do

4: Interact with the targeted LLM using each prompt in A for ⌈ N/ ( |A|⌈ log 2 ( |P| ) ⌉ ) ⌉ times

5: Use the collected samples to train a function g θ ( · ) parameterized by θ , e.g., a linear function or an MLP

6: Compute { ˆ µ ( p ) = g θ ( e ( p )) : p ∈ A}

7: Update the active prompt set A as the set of ⌈A / 2 ⌉ prompts in the original A with the highest ˆ µ ( p )

8: end for

9: Output: the remaining active prompt ˆ p ∗

SAR. It is also noticed in previous studies that selecting a diverse set of examples is vital in achieving good few-shot performances [67, 43]. Leveraging this heuristic, we propose to first divide the example set G into M clusters, denoted as {G 1 , · · · , G M } , based on the embeddings of the examples. Then, for each cluster G m , we find one example g m in it with the highest one-shot performance µ ( g m ) . To perform such a selection process efficiently, the SAR design [12] is leveraged. In this way, the diversity and quality of the selected examples are both guaranteed. The details are provided in Alg. 6.

E Full Experimental Details

In this section, we include full details of our experiments, while the complete codes are also uploaded in the supplementary materials.

E.1 LLMModels and System Instructions

Before further details, we first list the LLM models that we adopted for experiments:

GPT-3.5: gpt-3.5-turbo-1106 [55],

Llama2: Llama2-7b [73],

Gemma: Gemma-7b [70]

Mistral: Mistral-7B-v0.2 [34]

As we use chat-based LLMs, initial system instructions are needed, where the officially recommended system instructions are adopted in experiments, as shown in Fig 7.

Algorithm 5 TRIPLE-CSAR

1: Input: the set of available examples G , the size of the combination M , budget N

2: Initialization: set ˆ µ ( g ) ← 0 for all g ∈ G ; set the active example set A ← G ; ˜ T 0 ← 0 ; G acc ←∅ ; G rej ←∅ ; ˜ log( |G| ) ← ∑ i ∈ [ |G| ] 1 /i

3: for phase p = 1 , · · · , |G| do

4: ˜ T p ←⌈ ( N -|G| ) / ( ˜ log( n )( |G| p +1)) ⌉

5: Interact with the targeted LLM using each example g ∈ A as a one-shot prompt for ˜ T p -˜ T p -1 times

6: Update the sample means { ˆ µ ( g ) : g ∈ A} using the collected samples

7: Obtain order σ such that ˆ µ ( g σ (1) ) ≥ ˆ µ ( g σ (2) ) ≥ · · · ≥ ˆ µ ( g σ ( |A| ) )

8: Compute gaps

9: Compute g ′ ← arg max g ∈A ∆ g

10: Update G acc ←G acc ∪ { g ′ } if ˆ µ ( g ′ ) ≥ ˆ µ ( g σ ( M -|G acc | +1) ) ; G rej ←G acc ∪ { g ′ } otherwise

11: Update A ← G / ( G acc ∪ G rej )

12: end for

13: Output: the set G acc

Algorithm 6 TRIPLE-SAR

1: Input: the set of available examples G and their embedding E , the size of the combination M , budget N

2: Cluster G into clusters {G 1 , · · · , G M } based on embeddings E (e.g., via k -means)

3: Initialization: set ˆ µ ( g ) ← 0 for all g ∈ G ; set the active example set for cluster m as A m ←G m ; set the overall active example set as A ← G ; set the active cluster M ← [ M ] ; ˜ T 0 ← 0 ; ˜ log( |G| ) ← ∑ i ∈ [ |G| ] 1 /i

4: for phase p = 1 , · · · , |G| do

5: ˜ T p ←⌈ ( N -|G| ) / ( ˜ log( n )( |G| p +1)) ⌉

6: Interact with the targeted LLM using each example g ∈ A as a one-shot prompt for ˜ T p -˜ T p -1 times

7: Update the sample means { ˆ µ ( g ) : g ∈ A} using the collected samples

8: if ∃ m ∈ M such that |A m | = 1 then

9: Update M←M / { m }

10: Update g ∗ m ← the remaining example in A m

11: else

12: ∀ m ∈ M , compute ∆ m ← max g m ∈A m { max ¯ g m ∈A m ˆ µ (¯ g m ) -ˆ µ ( g m ) } and g ′ m ← arg max g m ∈A m { max ¯ g m ∈A m ˆ µ (¯ g m ) -ˆ µ ( g m ) }

14: Update A m ′ ←A m ′ / { g ′ m ′ }

13: Compute m ′ ← arg max m ∈M ∆ m

15: end if

16: end for

17: Output: the set { g ∗ , · · · , g ∗

E.2 Score Functions

Different score functions s ( · , · ) , i.e., metrics for evaluation, are used for diverse tasks in the InstructionInduction and BigBench-ii datasets, namely 'Exact match', 'F1-score', 'Multiple choice within', and 'Multiple choice f1-score'. These score functions are adopted according to the specific output requirements of different tasks:

Exact match: Used for most tasks unless otherwise specified, this metric scores 1 for outputs exactly matching the label, and 0 otherwise.

1 M }

SysteminstructionLlama2,

Gemma,Mistral

<<SAS>>[1SNI]<S>

Systeminstructiongpt-3.5

You areanintelligentassistant.Pleaseanswer

finishthegiventask,answerwith theoutputonly

"role": "system",

andreplynothingelse.

"content":"You are an intelligent assistant."

{




{

"role": "user",

<prompt>

"content":<prompt>

{




f1-score: Applied to tasks with complex targets like long sentences (e.g., 'informal to formal', 'negation', 'sentence similarity'), this metric (defined in Definition E.1), evaluates the overlap between the LLM response and the label.

Multiple choice within: Suitable for tasks with several correct answers, it scores 1 if the LLM's response matches any correct answer and 0 otherwise. We utilized this metric for tasks 'rhymes', 'translation-en-de', 'translation-en-es', 'translation-en-fr' and 'word in context'.

Multiple choice f1-score: Employed for tasks with multiple, lengthy correct answers ('common concept' task), it calculates the highest f1-score across all potential correct answers.

Definition E.1 (f1-score) . Suppose the question has a labeled answer T and the response of the LLM is A , then the f1-score for this answer is defined as:

where P = l m /l A stands for the precision of the response and R = l m /l T the recall of the response. Here we use l A and l T to denote the length of the response and label while l m is adopted to represent the number of matching words between A and T .

For the specific score function adopted for the BigBench-ii dataset, we advise referring to the 'metric' label for each task therein. This label indicates the appropriate metric ('Exact match' or 'Multiple choice within') for the optimal evaluation.

E.3 Experiments with Fixed Prompt Pools and APE

The prompt generation process to obtain the fixed prompt pools largely follows the one in APE [90], i.e., demonstrating LLMs with examples. In particular, in the generation of each prompt, we sample 10 examples from the training set to demonstrate LLMs with two types of generation templates: 'forward' and 'backward', which are illustrated in Fig. 8. The same setups are also adopted in the end-to-end experiments with APE in Sec. 5.2.

A side observation is that we find that in general, GPT-3.5 can handle both templates, resulting in reasonable prompts. However, LLMs with fewer parameter numbers, like Llama2-7b, Gemma-7b, or Mistral-7b-v0.2 we use exhibit difficulties in generating useful prompts from the 'backward' template, possibly due to its more simplified structure.

E.4 Experiments with APO

The APO framework [60] iteratively refines prompts based on feedback generated by LLMs. In particular, for each iteration, the system is set to identify { num_feedback } fault reasons (i.e., gradients) for the selected prompts from previously incorrectly answered examples. Then, with the selected prompts and the identified fault reasons, the LLM is instructed to create { num_prompts } new prompts for further selection. The adopted templates in our experiments are shown in Fig. 9,

ForwardGenerationTemplate

I gave a friend an instruction.Based on the

instruction they produced thefollowing(input,

output) pairs:

Input:[Q1]

Output:[A1]

BackwardGenerationTemplate

Input: [Q2]

Output:[A2]

Input: [Q1]

Output:[A1]

Input:[Q2]

Output:[A2]

Whatwastheinstruction?Provideonlythe

instructionandNOTHINGelse.

The instruction was to?

where we set { num_feedback } to 2 and { num_prompts } to 5. We believe this configuration ensures that each iteration effectively identifies key areas of improvement and sufficiently expands the pool of potential prompts.

Newpromptgenerationtemplate

"Gradient'template

I'm trying to write a zero-shot classifier.My current prompt is:

I'm trying to write a zero-shot classifier prompt.

{prompt)

My current prompt is:

But it gets the following examples wrong:

{error_exp}

{prompt)

Based on these examples the problem with this prompt is that

But thispromptgets thefollowingexamples

wrong:{error_string}give{num_feedbacks}

{gradient)

reasonswhy theprompt could havegotten these

examples wrong.

Based on the above information,I wrote {num_prompts}

differentimproved prompts.

Wrap each reason with <START> and <END>

Each prompt is wrapped with <START>and <END>. The

{num_prompts}new prompts are:{new_prompts}

E.5 Implementions of TRIPLE-CLST and TRIPLE-GSE

Obtaining Embedding. A critical component of both TRIPLE-CLST and TRIPLE-GSE is the extraction of sentence embeddings for the candidate prompts. In our experiments, the prompts are first tokenized using the cl100k_base tokenizer. Then, the tokenized prompts are input into the text-embedding-ada-002 model [56], converting them into continuous vectors.

TRIPLE-CLST . In experiments with TRIPLE-CLST , the number of clusters is set as L = ⌈ √ |P|⌉ and a third of our total budget is allocated for the initial phase, i.e., N 1 = N/ 3 . The k -means algorithm is employed as the clustering method. For more stable performance, Phase I is configured to find the top L/ 2 clusters, instead of the optimal one, which safeguards against the situation that the optimal prompt is not located in the optimal cluster. Also, for the BAI-FB designs in both phases, the CR algorithm [75] is adopted due to its flexibility.

TRIPLE-GSE . The OpenAI embedding API returns embeddings of 1536 dimensions, which can be challenging for learning with limited samples. To overcome this issue, in the implementation of TRIPLE-GSE , we first employ a projection to 64 dimensions using a matrix with random elements from the standard normal distribution. This technique is also incorporated in Chen et al. [13] and is particularly beneficial given our limited budget constraints. Furthermore, to avoid overfitting and convergence issues, we adopt the standard approach by dividing our interaction data into training ( 80% ) and validation ( 20%) sets. The prompt elimination process on line 7 in Alg. 4 is performed only if the mean squared error on the validation set is sufficiently low, and we set this error threshold at 0 . 1 in our experiments.

E.6 Experiments of Example Selection

In the results reported in Table 4, the task is to select 4 examples from a set of overall 50 candidate examples within 100 interactions with the targeted LLM. For tasks with a training dataset larger than 50 examples, 50 examples are first sampled to construct the candidate set, which is used consistently across experiments. There are also a few tasks with a training dataset smaller than 50 examples, which are thus entirely used as the candidate set. The same prompt template, as illustrated in Fig. 10, is used for all tasks to maintain consistency.

Few-shot template

Instruction:

Complete the problem.

Examples:

Input:

[

𝑄1

]

Output:

[

𝐴1

]

Input:

[

𝑄2

]

Output:

[

𝐴2

]

Input:

[

𝑄3

]

Output:

[

𝐴3

]

Input:

[

𝑄4

]

Output:

[

𝐴4

]

Task: Input:

[

𝑄

]

Output:

{output}

Provide only one answer and NOTHING else.

Further details regarding the adopted baselines are provided in the following.

Random. To validate the benefits of interactions with the targeted LLM during the selection, one commonly adopted baseline is to randomly select the required number of examples from the candidate pool.

Uniform. Similar to the uniform baseline adopted in Sec. 5, the overall budget can be uniformly divided to evaluate the one-shot performance of each prompt. Then, the examples with the highest estimated one-shot performances are selected.

Also, for TRIPLE-SAR , the same process of obtaining embeddings as described in Appendix E.5 with also k -means as the algorithm to perform clustering.

E.7 Computing Resources and Costs

We use a workstation with two Nvidia-A6000 Ada GPUs for all experiments using white-box LLMs (i.e., Llama2, Mistral, and Gemma). To reproduce our result, any GPU with over 30 GB of memory should be sufficient. With our equipment, each interaction with the white-box LLMs typically takes around 1 . 3 -2 . 0 seconds. For experiments using GPT-3.5, the whole execution is light regarding local computational resources, while access to the OpenAI API is needed to perform learning. Under our network condition, one API call typically takes around 1 second.

F Additional Experimental Results

Additional experimental results are provided to supplement observations in the main paper.

F.1 Selection of Budgets

To further guide practical implementation, we additionally investigate how to select a reasonable budget. In particular, we focus on the efficiency of various prompt selection algorithms in identifying a 'good' prompt - either the optimal prompt in the pool or achieving 95% or better of the optimal prompt's performance. Fig. 11 illustrates that initial increases in budgets significantly improve the probability of identifying a good prompt, but this benefit tapers off with further budget expansion. This finding suggests that starting with a modest budget and incrementally increasing it is the more effective approach, stopping when additional investment no longer translates into significant returns.

5

10

15

20

25

30

Budget

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Find Good Prompt Rate/Budget

Find Good Prompt Rate/Budget for Different Prompt Selection Algorithms

Uniform

UCB

SH

CR

FuncApprox

Cluster

F.2 Performances on Gemma and Mistral

For the experiments on the selected tasks with |P| = 30 prompts and budget N = 150 , additional results with Gemma and Mistral are reported in Fig. 12(a) and 12(b). The superiority of TRIPLE can still be observed, demonstrating its flexibility over different LLMs.

F.3 Performances on Additional Datasets

The experiments are further extended to more diagnostic tasks (GLUE [74]) and math problem datasets (GSM8K [17]). More specifically, TRIPLE methods are deployed to select prompts for the task 'Cola' in GLUE (on distinguishing linguistic acceptability) and also chain-of-thought prompts for GSM8K (on mathematical reasoning). The results are presented in Table 5, where it can be observed that the superiority of TRIPLE is still prominent.

F.4 Complete Evaluations on 47 Tasks

In the main paper, we provide experimental results of 12 representative tasks from the overall 47 available tasks in Sec. 5. In the following, the complete results are discussed.

|P| = 30 , N = 150 : results on the 24 available tasks in the Instruction-Induction dataset [30] are illustrated in Fig. 13(a) (GPT-3.5), and 13(b) (Llama2);

|P| = 30 , N = 150 : results on the 23 available tasks in the BigBench-ii dataset [69] are illustrated in Fig. 14(a) (GPT-3.5), and 14(b) (Llama2).

|P| = 150 , N = 100 : results on the 24 available tasks in the Instruction-Induction dataset [30] are illustrated in Fig. 15(a) (GPT-3.5), and 15(b) (Llama2);

|P| = 150 , N = 100 : results on the 23 available tasks in the BigBench-ii dataset [69] are illustrated in Fig. 16(a) (GPT-3.5), and 16(b) (Llama2).

Also, the complete performances of example selection for few-shot prompts discussed in Sec. 6 are presented in Fig. 17 (Instruction-Induction) and 18 (BigBench-ii).

Besides the average return, another key aspect of prompt selection is the frequency of selecting a good prompt. In Table 8, we further demonstrate the best prompt identification frequency of different algorithms across 20 selected tasks from 5 independent runs.

Cause

and

Common

concept

Disambiguation

qa

Gender inc. DE

Hyperbaton

Larger

animal

Movie

recommendation

Object

counting

Starts with

Question

selection

Rhymes

Snarks

0.0

0.6

1.2

1.8

2.4

2.6

2.1

2.4

1.9

2.3

1.9

|

| = 30

N

=150

UCB

TRIPLE-SH

TRIPLE-CR

BO-EI

BO-PI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

(a) Gemma

Cause

and

effect

Common

concept

Disambiguation

qa

Gender inc. DE

Hyperbaton

Larger

animal

Movie

recommendation

Object

counting

Starts with

Question

selection

Rhymes

Snarks

0.0

0.6

1.2

1.8

2.1

|

| = 30

N

=150

(b) Mistral

effect

Cause

and

effect

Common

concept

Diff

First

word

letter

Informal

to

formal

Larger

animal

Letters

list

0

1

2

Taxonomy

animal

Negation

Number

to

verbal

Active

to

passive

Singular

to

plural

Rhymes

Second

word

letter

Sentence

similarity

0

1

2

Sentiment

Starts with

Sum

Translation

en-de

Translation

en-es

Translation

en-fr

Word

in

context

0

1

2

|

| = 30

N

=150

UCB

TRIPLE-SH

TRIPLE-CR

BO-EI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

(a) GPT-3.5

Cause

and

effect

Common

concept

Diff

First

word

letter

Informal

to

formal

Larger

animal

Letters

list

0

1

2

2.23

3.08

3.27

Taxonomy

animal

Negation

Number

to

verbal

Active

to

passive

Singular

to

plural

Rhymes

Second

word

letter

Sentence

similarity

0

1

2

Sentiment

Starts with

Sum

Translation

en-de

Translation

en-es

Translation

en-fr

Word

in

context

0

1

2

|

| = 30

N

=150

UCB

TRIPLE-SH

TRIPLE-CR

BO-EI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

(b) Llama2

Implicatures

Ruin

names

Navigate

Causal

judgment

Sports

understanding

Object

counting

Epistemic

reasoning

Winowhy

0

1

2

Timedial

Snarks

Word

sorting

Hyperbaton

Linguistics

puzzles

Question

selection

Word

unscrambling

Logical

fallacy

detection

0

1

2

Dyck

languages

Disambiguation

qa

Movie

recommendation

Tense

Presuppositions

as

Gender inc. DE

Operators

0

1

2

|

| = 30

N

=150

UCB

TRIPLE-SH

TRIPLE-CR

BO-EI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

(a) GPT-3.5

Implicatures

Ruin

names

Navigate

Causal

judgment

Sports

understanding

Object

counting

Epistemic

reasoning

Winowhy

0

1

2

Timedial

Snarks

Word

sorting

Hyperbaton

Linguistics

puzzles

Question

selection

Word

unscrambling

Logical

fallacy

detection

0

1

2

Dyck

languages

Disambiguation

qa

Movie

recommendation

Tense

Presuppositions

as

Gender inc. DE

Operators

0

1

2

|

| = 30

N

=150

1.90

UCB

TRIPLE-SH

TRIPLE-CR

BO-EI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

(b) Llama2

Cause

and

effect

Common

concept

Diff

First

word

letter

Informal

to

formal

Larger

animal

Letters

list

Taxonomy

animal

Negation

Number

to

verbal

Active

to

passive

0.0

0.5

1.0

1.5

2.0

Singular

to

plural

Rhymes

Second

word

letter

Sentence

similarity

Sentiment

Starts with

Sum

Translation

en-de

Translation

en-es

Translation

en-fr

Word

in

context

0.0

0.5

1.0

1.5

2.0

BO-EI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

(a) GPT-3.5

Cause

and

effect

Common

concept

Diff

First

word

letter

Informal

to

formal

Larger

animal

Letters

list

Taxonomy

animal

Negation

Number

to

verbal

Active

to

passive

0.0

0.5

1.0

1.5

2.0

1.91

Singular

to

plural

Rhymes

Second

word

letter

Sentence

similarity

Sentiment

Starts with

Sum

Translation

en-de

Translation

en-es

Translation

en-fr

Word

in

context

0.0

0.5

1.0

1.5

2.0

BO-EI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

(b) Llama2

Implicatures

Ruin

names

Navigate

Causal

judgment

Sports

understanding

Object

counting

Epistemic

reasoning

Winowhy

Timedial

Snarks

Word

sorting

Hyperbaton

0.0

0.5

1.0

1.5

2.0

2.14

2.62

Linguistics

puzzles

Question

selection

Word

unscrambling

Logical

fallacy

detection

Dyck

languages

Disambiguation

qa

Movie

recommendation

Tense

Presuppositions

as

Gender inc.

DE

Operators

0.0

0.5

1.0

1.5

2.0

2.32

2.03

BO-EI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

(a) GPT-3.5

Implicatures

Ruin

names

Navigate

Causal

judgment

Sports

understanding

Object

counting

Epistemic

reasoning

Winowhy

Timedial

Snarks

Word

sorting

Hyperbaton

0.0

0.5

1.0

1.5

2.0

Linguistics

puzzles

Question

selection

Word

unscrambling

Logical

fallacy

detection

Dyck

languages

Disambiguation

qa

Movie

recommendation

Tense

Presuppositions

as

Gender inc.

DE

Operators

0.0

0.5

1.0

1.5

2.0

BO-EI

NeuralUCB

TRIPLE-GSE

TRIPLE-CLST

(b) Llama2

Cause

and

effect

Common

concept

Diff

First

word

letter

Informal

to

formal

Larger

animal

Letters

list

Taxonomy

animal

Negation

Number

to

verbal

Active

to

passive

0.0

0.5

1.0

1.5

Singular

to

plural

Rhymes

Second

word

letter

Sentence

similarity

Sentiment

Starts with

Sum

Translation

en-de

Translation

en-es

Translation

en-fr

Word

in

context

0.0

0.5

1.0

1.5

2.0

2.00

2.12

Random

Uniform

TRIPLE-SAR

TRIPLE-CSAR

Implicatures

Ruin

names

Navigate

Causal

judgment

Sports

understanding

Object

counting

Epistemic

reasoning

Winowhy

Timedial

Snarks

Word

sorting

Hyperbaton

0.0

0.5

1.0

1.5

Linguistics

puzzles

Question

selection

Word

unscrambling

Logical

fallacy

Dyck

languages

Disambiguation

qa

Movie

recommendation

Tense

Presuppositions

as

Gender inc. DE

Operators

0.0

0.5

1.0

1.5

2.0

2.06

Random

Uniform

TRIPLE-SAR

TRIPLE-CSAR

detection nli

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims in the abstract and introduction faithfully point out the main idea we proposed, the major contribution we made, and the technique challenges we have solved.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We included a discussion of our paper's limitations and potential future directions in Sec. B.2; please see the referred section for more information.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper's work is mainly practical, we did not claim any theoretical results and thus did not include any proof accordingly.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The methods proposed in this work have been described with full details (see the pseudocodes in Algs. 1, 2, 3, 4, 5 and 6). The complete experimental setups are presented in Appendix E, and the experiment codes can be found at https://github. com/ShenGroup/TRIPLE . With these efforts, we are confident that the main results of the paper are reproducible.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The experiment codes (together with detailed instructions) are available at https://github.com/ShenGroup/TRIPLE . The adopted datasets are all open-source ones. This should be sufficient to make anyone who wants to reproduce our experiment results capable of easily executing the experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Appendix E, we have included all the experimental details used for our experiment execution and how the hyper-parameters are chosen.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We did include the error bars and the standard deviation in all the experimental results that need this information.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have included the necessary computing/other resources, as well as the experimental times we have used to finish all our experiments. Please refer to Appendix E.7 for this information.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conducted in the paper follows the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impact of our work in Appendix B.1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly credited all the code sources, data sources, and open-source models in our paper following their license and terms.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: In this current version, we do not plan to release any new assets to the public. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not include any crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study focuses on the efficient prompt design under budget constraints, where effective prompts may have to be designed without excessive evaluation of a very large number of candidate prompts.\nThe paper aims to present a principled framework - called  TRIPLE - for tackling this problem, which is achieved by making connections between the prompt optimization problem and the fixed-budget best arm identification problem (BAI-FB), thereby drawing from the rich literature on the multi-armed bandits (MAB) problems and methods. Based on prompt design experiments for various design tasks using various LLMs, the paper shows that TRIPLE can meaningfully improve the prompt design results when there are budget constraints.\n\nSTRENGTHS:\n- It is a good and meaningful attempt to formulate the prompt optimization problem as a multi-armed bandit (MAB) problem. As shown in the paper, making connections between prompt optimization and best-arm identification in MAB allows one to draw from the rich MAB literature to devise effective prompt design methods. This is clearly demonstrated in the budget constrained case, which is the main focus of the current study, as one can leverage fixed-budget best-arm identification methods to improve prompt design under a limited budget for evaluating the (possibly large) prompt candidate pool. However, the benefits of the proposed framework are likely to go beyond the current problem setting, as MAB is an actively investigated field with rich outcomes and existing tools in MAB may be utilized for enhancing prompt optimization methods in the future.\n\n- The results (e.g., in Figure 2 and Tables 2-4) clearly demonstrate the efficacy of the proposed TRIPLE framework for prompt evaluation & selection and how it may also improve end-to-end prompt optimization by integrating the framework with popular prompt generation schemes. According to the results shown, the gains turn out to be fairly significant and also consistent across different tasks/settings.\n\nWEAKNESSES:\n- As the main contribution of this paper seems to lie in making systematic connections between prompt optimization and best arm identification in MAB, after establishing these connections, the improvements achieved in prompt optimization are direct outcomes of adopting existing FB-BAI algorithms in the MAB literature. As a result, novel methodological contributions that go beyond the utilization of existing FB-BAI methods proposed and well-studied in MAB for prompt optimization purpose are somewhat limited.\n\n- While the paper discusses the issues that arise when the prompt pool is huge (hence a large number of arms) and also proposes practical schemes to address these issues, there is no in-depth discussion of how the pool size affects the overall prompt optimization performance. There should be further discussion on this issue and the paper should include at least some empirical results (e.g., similar to Figure 3) that shows how the gain may be affected by the prompt pool size.\n\n- In Figure 2, it is unclear when the \"normalized evaluation score\" exceeds 1 (i.e., outperforming the uniform scheme\"). Please add a horizontal line to show which bars are above/below 1.\n\n- The comparison with BO-EI is interesting but as BO performance tends to vary widely depending on the acquisition function used, it would be meaningful to provide additional comparison based on at least one or two additional acquisition functions. Especially, using BO acquisition functions aimed at different aspects (e.g., uncertainty vs diversity) may be helpful.\n\nQUESTIONS:\n- Please provide some discussion on how TRIPLE may be used when the prompt optimization task has multiple objectives. How can connections to best-arm identification be leveraged in case of multi-objective prompt optimization?\n\nLIMITATIONS:\nThe paper discusses the limitations of the proposed approach in the Appendix (section B.2) as well as some future research directions to address these limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper studies prompt optimization using BAI-FB. There are several contributions:
1. Draw a connection between prompt optimization with the BAI-FB
2. Benchmark different acquisition function for prompt optimization
3. Extensive experiments are done to show the effectiveness of the proposed prompt optimization and extend the setting to example selection.

STRENGTHS:
1. The presentation of this work is clear
2. The formulation of prompt optimization with limited budget as BAI-FB is clean with strong justification
3. The experiments are extensive with a good coverage of baselines, LLMs, and datasets

WEAKNESSES:
1. The novelty of this work is limited, since there are already a lot of prompt optimization work using bandits as the authors have discussed in the paper
2. A larger domain of prompts need to be considered where exploration vs exploitation is more important. For example in [43], the domain of prompt is 10k and the experiments in [43] shows that NeuralUCB is a SOTA selection strategy in this case. A comparison of MAB-FB methods and regret minimization methods in a larger domain of prompt optimization can be more helpful.

QUESTIONS:
What's the hyper-parameter used to run the regret minimization algorithm? e.g., UCB and NeuralUCB? Since these algorithm are highly sensitive to the selection of hyper-parameters.

LIMITATIONS:
See my weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an algorithm that adopts the fixed-budget best arm (BAI-FB) identification to search for the best prompt. The authors have considered two variants of BAI-FB including sequential halving (SH) and continuously reject (CR). They then utilize prompt embeddings to enhance the BAI-FB methods via clustering and function approximation.\n\nSTRENGTHS:\n- The idea of using BAI-FB other than naive online MAB is interesting in prompt optimization.\n\n- The empirical results suggest TRIPLE is an effective method to identify good prompts.\n\nWEAKNESSES:\n- The presentation could be substantially improved. For example, Section 3 seems redundant as most of its content is not critical and can be covered by Section 2. Besides, the empirical results show TRIPLE-CLST/GSE are generally better but their justifications and descriptions (including the clustering property which is critical for TRIPLE-CLST and the embedding in function approximation) are not enough in the main text. I strongly suggest authors to reduce Section 3 and extend Section 4 by adding explanations and moving some materials from the appendix into the main text.\n\n- The evaluation setting is not described in detail. For example, are the evaluation budgets per prompt the same for all considered baselines? If so, the exact amount of evaluation budgets per prompt only means how accurate this estimation is.\n\n- The considered prompt candidates for search are very limited only containing 150 at the maximum. However, in the literature (e.g., Lin et al. [43]), they have considered over 10k candidates. Generally, it is reasonable to consider a large prompt space for the search method to really find the optimal prompt. Otherwise, the result could be biased.\n\n- Although the paper emphasizes efficiency, it is very hard to see if the method is efficient in the empirical section, where only the performances under the specific budget setting are considered.\n\n- Recent baselines with/without embeddings like ZOPO and, OPRO should also be considered.\n\n- The tasks evaluated in this paper are mostly instruction induction tasks. It would be great to show that the method can work on GLUE tasks and reasoning tasks (e.g., GSM8K, MATH).\n\nQUESTIONS:\n- Some literature adopts the deterministic function $f(\\cdot)$ for prompt optimization and this is practical as some LLMs use greedy sampling. Will your method still work under the deterministic function $f(\\cdot)$?\n\nSee other weaknesses above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study prompt optimization, with a focus on finding the best prompt from a pool of proposed prompts under highly limited budgets. They establish a connection to fixed-budget best arm identification in multi-armed bandits (MAB) and explore several algorithms from that problem, applied to prompt selection for evaluation. They then make additional improvements by observing that prompts are not independent and that information, through embedding the prompts, can be shared to make the exploration more effective. They test methods based on clustering and DNN reward functions, based off of off-the-shelf embeddings of the prompts.\n\nSTRENGTHS:\n1. The paper is well-argued and well-presented. The authors argue that existing work on exploring prompt candidates under limited budgets apply a regret minimization framework, which is less well-suited than best arm identification, and then go on to explore highly-effective simple extensions that share information across prompts.\n\n2. The results are compelling: the methods are compared against a convincing set of baselines, from UCB to Bayesian optimization (BO) with expected improvement. The outcomes consistently show the value of the author's selection of framework (BAI) and their information sharing.\n\nWEAKNESSES:\n1. The authors evaluate on tasks that to my understanding are extremely limited in scope. Most are narrow reasoning puzzles and may not reflect the type or complexity of typical prompts people use in the increasingly elaborate open-ended LM systems out there. How would such open-endedness interact with the findings?\n\n2. The results are somewhat hard to parse from the tables and especially so from the figures, though it is clear that they are overall quite positive. The budgets considered for evaluation are perhaps very strict, which is worth calling out, e.g. <1 evaluation per prompt, 5 evaluations per prompt. The method does show gains over baselines up to 30 evalutions per prompt, so it is very compelling nonetheless.\n\n3. The claims at the start of the paper that existing work just rarely considers budget of optimization is a bit overblown and overlL unnecessary for the claims of the paper, which are compelling on their own. The authors compare with several baselines that *do* consider budget, just not as effectively or thoughtfully as the authors do.\n\nQUESTIONS:\nDo TRIPLE-SAR and CSAR take ordering into account?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}