Bandit-Based Prompt Design Strategy Selection Improves Prompt Optimizers
Authors: Rin Ashizawa, Yoichi Hirose, Nozomu Yoshinari, Kento Uchida, Shinichi Shirakawa
Accepted at: acl25

Bandit-Based Prompt Design Strategy Selection Improves Prompt Optimizers

Rin Ashizawa  Yoichi Hirose  Nozomu Yoshinari
Kento Uchida  Shinichi Shirakawa
Yokohama National University
{ashizawa-rin-gs,hirose-youichi-kc,yoshinari-nozomu-ry}@ynu.jp
{uchida-kento-fz,shirakawa-shinichi-bg}@ynu.ac.jp

Abstract

Prompt optimization aims to search for effective prompts that enhance the performance of large language models (LLMs). Although existing prompt optimization methods have discovered effective prompts, they often differ from sophisticated prompts carefully designed by human experts. Prompt design strategies, representing best practices for improving prompt performance, can be key to improving prompt optimization. Recently, a method termed the Autonomous Prompt Engineering Toolbox (APET) has incorporated various prompt design strategies into the prompt optimization process. In APET, the LLM is needed to implicitly select and apply the appropriate strategies because prompt design strategies can have negative effects. This implicit selection may be suboptimal due to the limited optimization capabilities of LLMs. This paper introduces Optimizing Prompts with sTrategy Selection (OPTS), which implements explicit selection mechanisms for prompt design. We propose three mechanisms, including a Thompson sampling-based approach, and integrate them into EvoPrompt, a well-known prompt optimizer. Experiments optimizing prompts for two LLMs, Llama-3-8B-Instruct and GPT-4o mini, were conducted using BIG-Bench Hard. Our results show that the selection of prompt design strategies improves the performance of EvoPrompt, and the Thompson sampling-based mechanism achieves the best overall results. Our experimental code is provided at https://github.com/shiralab/OPTS.

1 Introduction

Large language models (LLMs) such as GPT-4 (OpenAI et al., 2024), Gemini (Team et al., 2024), and Llama 3 (Grattafiori et al., 2024) have demonstrated superior abilities in a variety of domains, including medicine (Nori et al., 2023), law (Katz et al., 2024), and code generation (Rozière et al., 2024). Since well-crafted prompts improve the performance of LLMs, prompt engineering (i.e., designing better prompts) plays a key role in the area (Bsharat et al., 2024; Schulhoff et al., 2024). Despite its importance, prompt engineering is laborious as it requires a lot of time for refinement and sufficient knowledge of the tasks. To design effective prompts with less effort, research on prompt optimization has been actively conducted. In particular, discrete prompt optimization, which optimizes prompts within the natural language space, has attracted attention. This approach is valuable as it typically allows for the optimization of prompts for black-box LLMs, such as GPT-4, while also providing interpretable results (Chang et al., 2024). To explore effective prompts within the large natural language space, several methods have been proposed, which include emulating evolutionary algorithms using LLMs (Guo et al., 2024; Fernando et al., 2023; Cui et al., 2024). These methods have discovered effective prompts, but they often differ from sophisticated prompts carefully designed by human experts.

Prompt design strategies, which provide guidelines for creating effective prompts, can be key to boosting the performance of prompt optimizers. In fact, Chain-of-Thought (CoT; Wei et al., 2022) and Role Prompting (Wang et al., 2024a) have been employed in prompt optimization, leading to better prompts (Agarwal et al., 2024). Recently, Kepel and Valogianni (2024) proposed a method termed the Autonomous Prompt Engineering Toolbox (APET), which incorporated various prompt design strategies into the optimization process. APET fed all prepared strategies into an LLM to generate a new prompt that incorporates the strategies. However, not all strategies should be incorporated because prompt design strategies can have negative effects depending on both the LLM and the task (Zheng et al., 2024; Deng et al., 2024). In APET, an LLM that generates prompts
<table>
  <tr>
    <th colspan="2">Thompson Sampling: Select one arm</th>
  </tr>
  <tr>
    <th>K descriptions of prompt design strategies</th>
    <th>Option not using prompt design strategies</th>
  </tr>
  <tr>
    <td>Arm 1 Arm 2 ...... Arm K</td>
    <td>Arm \( K + 1 \)</td>
  </tr>
</table>

![Overview of OPTS(TS), which shows how the prompt generated from the prompt optimizer is modified. If one of the first K arms is selected, the description of the prompt design strategy corresponding to the selected arm is passed to the prompt-designing LLM. If the \((K + 1)\)-th arm is selected, the prompt is not modified in any way.](page_184_153_1080_496.png)

Figure 1: Overview of OPTS(TS), which shows how the prompt generated from the prompt optimizer is modified. If one of the first \( K \) arms is selected, the description of the prompt design strategy corresponding to the selected arm is passed to the prompt-designing LLM. If the \((K + 1)\)-th arm is selected, the prompt is not modified in any way.

is required to implicitly select appropriate strategies, which may lead to suboptimal performance because LLMs cannot perform optimization effectively (Huang et al., 2024).

In this paper, we introduce explicit selection mechanisms for prompt design strategies for the first time. We also propose three selection methods, including one based on Thompson sampling (TS; Thompson, 1933; Russo et al., 2018). By integrating them with the existing prompt optimizer, EvoPrompt (Guo et al., 2024), we show that explicit strategy selection effectively leverages existing knowledge of prompt design and enhances the performance of prompt optimizers. Moreover, the optimizer with the TS-based selection mechanism outperforms other existing methods.

In summary, our contributions are as follows:

• We propose explicit prompt design strategy selection mechanisms, including a method based on Thompson sampling, for prompt optimizers.

• We experimentally show that the proposed selection mechanism enhances EvoPrompt. TS-based selection improves EvoPrompt’s performance by up to 50% when using GPT-4o mini for both generating prompts and solving downstream tasks.

• We also compare the TS-based selection with APET-based selection and uniform sampling-based selection. The results demonstrate that TS-based selection is overall superior.

2 Related Work

Prompt design strategy. The term prompt design strategy refers to a well-established guideline for designing prompts that have been empirically known to be effective. Chain-of-Thought (CoT; Wei et al., 2022) and Role Prompting (Wang et al., 2024a) are notable examples. CoT asks the LLMs to generate not only the answer, but also the reasoning process that leads to the answer. Role Prompting is a strategy that includes phrases in the prompt that give the LLM a role. Various prompt design strategies have been proposed so far (Schulhoff et al., 2024; Xu et al., 2023; Li et al., 2023; Xu et al., 2024; Deng et al., 2024; Lu et al., 2023; Bsharat et al., 2024), yet they are not always useful. Indeed, CoT and Role Prompting can lead to worse results (Deng et al., 2024; Zheng et al., 2024). As their efficacy depends on the LLM and task, users need to make a non-obvious decision on whether to use them.

Discrete prompt optimization. Discrete prompt optimization optimizes prompts in natural language space. To effectively deal with natural language space, several prompt optimizers emulate the process of black-box optimization algorithms by using LLMs. These methods are useful because optimized prompts have high interpretability, while
they can be applied to LLMs that can be accessed through black-box APIs such as GPT-4 (OpenAI et al., 2024). GrIPS (Prasad et al., 2023) repeatedly edits the phrases in the prompt, and APE (Zhou et al., 2023) repeatedly generates prompts using LLMs based on Monte Carlo search. Unlike APE and GrIPS, ProTeGi (Pryzant et al., 2023) uses a mechanism in which incorrect answers made by an LLM and the corresponding prompt are fed into another LLM to generate a proposal to improve the prompt, and another LLM responsible for designing prompts then modifies the prompt according to the proposal. In addition to this mechanism, PromptAgent (Wang et al., 2024b) also uses Monte Carlo Tree Search to efficiently optimize prompts. Besides these, adv-ICL (Long et al., 2024), which applies adversarial learning, has been proposed. In OPRO (Yang et al., 2024), instead of using an LLM to suggest a proposal to improve the prompts, an LLM directly generates new prompts using three items: previously generated prompts, their scores, and a description of the downstream task.

Recently, several methods combining LLMs with evolutionary algorithms have been proposed (Guo et al., 2024; Jin et al., 2024; Fernando et al., 2023; Cui et al., 2024). EvoPrompt (Guo et al., 2024), a representative method among them, emulates the optimization process of Genetic Algorithm (GA) or Differential Evolution (DE). In contrast to EvoPrompt (Guo et al., 2024), Prompt-Breeder (Fernando et al., 2023) also optimizes the prompt that is used for generating new prompts. PhaseEvo (Cui et al., 2024) optimizes both task instruction and examples and achieves highly effective optimization by dividing optimization into multiple stages.

In addition to dividing optimization into multiple stages, PromptWizard (Agarwal et al., 2024) utilizes prompt design strategies such as CoT and Role Prompting, but it lacks a strategy selection mechanism and applies them in all cases. EoT prompting (Jin et al., 2024) optimizes zero-shot CoT (Kojima et al., 2022) using evolutionary algorithms. APET (Kepel and Valogianni, 2024) is the most relevant to our study. In APET, a prompt and descriptions of prompt design strategies are input to an LLM. The LLM then implicitly selects strategies and generates a new prompt. In contrast, we propose explicit strategy selection mechanisms that assist prompt optimizers in exploiting appropriate strategies.

3 Proposed Methods

In this section, we describe our proposed methods for selecting prompt design strategies. We then introduce prompt optimization algorithms that combine EvoPrompt (Guo et al., 2024) with the strategy selection methods. We term our methods Optimizing Prompts with sTrategy Selection (OPTS).

Terminology Task-solving LLM is an LLM that is applied to and solves downstream tasks, while Prompt-designing LLM is another LLM that produces helpful prompts for task-solving LLMs. In contrast to prompt, which represents an instruction for a task-solving LLM, meta-prompt refers to an instruction for a prompt-designing LLM.

3.1 Selection of Prompt Design Strategies

In the following, we discuss three different methods: the TS-based selection called OPTS(TS), the uniform sampling-based selection called OPTS(US), and the APET-based selection called OPTS(APET).

OPTS(TS) OPTS(TS) selects prompt design strategies using Thompson sampling (TS; Thompson, 1933; Russo et al., 2018), which is a multi-armed bandit algorithm and empirically shows superior performance (Chapelle and Li, 2011).

The overview of OPTS(TS) is shown in Figure 1. There are \( K \) arms, each corresponding to one of the \( K \) descriptions of the prompt design strategies. Also, we append a special arm called the inaction arm, which corresponds to the option of not using the prompt design strategy, making a total of \( K + 1 \) arms. The inaction arm is needed because none of the predefined strategies may improve the prompts at all. To instantiate TS, we use the beta distributions as priors for the expected reward. Once one of the first \( K \) arms is sampled by TS, we feed the description of the corresponding prompt design strategy into the prompt-designing LLM along with the meta-prompt and the prompt to be modified. The LLM then modifies the prompt based on the input. The meta-prompt is the same as that used in APET, whose details are explained in Appendix A. After evaluating the generated prompt with the task-solving LLM and calculating its score, a reward is calculated using the score, and the distributions in TS are updated based on the reward. Throughout this paper, we compute the reward \( r \) for a prompt with the score \( s \) as

\[
r = 1 \left[ s > \max \mathcal{S} \right] \in \{0, 1\} ,
\]

(1)
Algorithm 1 EvoPrompt(DE)-OPTS
Require: Initial prompts \( \mathcal{P}_0 = \{p_1, p_2, \ldots, p_N\} \), population size \( N \), number of iterations \( T \), development set \( D_{dev} \) consisting of input and correct output pairs \( (x, y) \), scoring function \( g \), task-solving LLM \( f_T \)
1: **Evaluation** of initial prompts: \( S_0 \leftarrow \left\{ s_i = \frac{1}{|D_{dev}|} \sum_{(x,y) \in D_{dev}} g\left(y, f_T(p_i, x)\right) : p_i \in \mathcal{P}_0 \right\} \)
2: **for** \( t = 1 \) **to** \( T \) **do**
3:     **for** \( p_i \) in \( \mathcal{P}_{t-1} \) do \( \triangleright p_i \): the \( i \)-th parent prompt
4:         **Sample donors**: \( p_{r_1}, p_{r_2} \in \mathcal{P}_{t-1} \), where \( p_{r_1}, p_{r_2} \), and \( p_i \) differ from each other.
5:         **Crossover and Mutation**: \( p'_i \leftarrow f_D(m_{de}, (p_i; p_{r_1}; p_{r_2}; p_{best})) \)
6:         where \( p_{best} \) is the current best prompt. \( \triangleright f_D \): Prompt-Designing LLM
7:         \( \triangleright m_{de} \): Meta-prompt for DE-based crossover and mutation
8:         **OPTS**: Generate \( p''_i \) from \( p'_i \) by incorporating prompt design strategies (Refer to Section 3.1)
9:         **Selection**: \( p^*_i = \underset{p \in \{p_i, p''_i\}}{\operatorname{argmax}} \frac{1}{|D_{dev}|} \sum_{(x,y) \in D_{dev}} g\left(y, f_T(p, x)\right) \)
10:        \( \triangleright \) Keep the better one in the population
11:        **Update probability distribution** if the TS-based selection is used (Refer to Section 3.1)
12:    **end for**
13:    **Update**: \( \mathcal{P}_t \leftarrow \{p^*_i : 1 \leq i \leq N\} \)
14: **end for**
15: **Return** the best prompt \( p^* = \underset{p \in \mathcal{P}_T}{\operatorname{argmax}} \frac{1}{|D_{dev}|} \sum_{(x,y) \in D_{dev}} g\left(y, f_T(p, x)\right) \)

where \( \mathcal{S} \) is the set of scores of the parent prompts, which come from EvoPrompt (Guo et al., 2024) described in Section 3.2, and \( 1[\cdot] \) is the indicator function.

OPTS(US) In OPTS(US), each arm is selected according to a uniform distribution. OPTS(US) is similar to OPTS(TS), except that the probability of selecting each arm is equal and is not updated.

OPTS(APET) OPTS(APET) is the selection method based on APET (Kepel and Valogianni, 2024). It is slightly different from APET in that it has an additional option equivalent to the inaction arm. OPTS(APET) first randomly decides with a probability of 0.5 whether to modify a prompt based on the prompt design strategies. If it decides to modify the prompt, the prompt-designing LLM is applied to the prompt to incorporate the prompt design strategies. The prompt-designing LLM receives the meta-prompt, the description of all prompt design strategies, and the prompt to be modified. Then, it implicitly selects the prompt design strategies and modifies the prompt according to them.

3.2 EvoPrompt with OPTS
We combine the proposed selection methods with EvoPrompt (Guo et al., 2024). We adopt EvoPrompt because it is effective yet sufficiently simple, allowing us to focus solely on evaluating the strategy selection methods. Also, it has variants depending on whether GA or DE is employed. This feature allows us to assess the impact of prompt design strategy selection on different optimization algorithms. The algorithm integrated OPTS into EvoPrompt(DE) is shown in Algorithm 1, while that based on EvoPrompt(GA) is shown in Appendix B. Note that a response generation by LLM \( f \) is denoted by \( f(p, x) \). We insert OPTS after the crossover and mutation process of EvoPrompt. After evaluating the newly generated prompts with the task-solving LLM, the scores are used to determine the next generation’s population and, if necessary, to update the distribution of the arms in TS. See Appendix B for the details of the algorithm.

4 Experiments
We experimentally evaluate three strategy selection methods we introduce: OPTS(TS), OPTS(US), and OPTS(APET). Combined with EvoPrompt, these methods are applied to various tasks and compared with the existing baseline methods.

4.1 Dataset
We evaluate the proposed method using BIG-Bench Hard (BBH; Suzgun et al., 2022). BBH is a collection of the tasks that are challenging for LLMs. Details of each task can be found in the original BBH paper. For each task, we randomly sample 50 examples from the test set and use them as the development set, as done in (Guo et al., 2024). The
<table>
  <tr>
    <th>Prompt Design Strategy</th>
    <th>Remarks</th>
  </tr>
  <tr>
    <td>ExpertPrompting</td>
    <td>Assign expert roles to task-solving LLMs (Xu et al., 2023).</td>
  </tr>
  <tr>
    <td>Chain-of-Thought</td>
    <td>Let task-solving LLMs also generate a reasoning process (Wei et al., 2022).</td>
  </tr>
  <tr>
    <td>Tree-of-Thought</td>
    <td>Let task-solving LLMs iteratively choose the best of multiple reasoning paths, backtracking as necessary (Yao et al., 2023).</td>
  </tr>
  <tr>
    <td>Emotion Prompting</td>
    <td>Incorporate phrases that appeal to human emotions (Li et al., 2023).</td>
  </tr>
  <tr>
    <td>Re-Reading</td>
    <td>Instruct task-solving LLMs to reread the question (Xu et al., 2024).</td>
  </tr>
  <tr>
    <td>Style Prompting</td>
    <td>Specifies the desired output style (Lu et al., 2023).</td>
  </tr>
  <tr>
    <td>Rephrase and Respond</td>
    <td>Let task-solving LLMs rephrase the question before responding (Deng et al., 2024).</td>
  </tr>
  <tr>
    <td>Avoiding bias</td>
    <td>A more generalized version of the 13th principle of the 26 principles of prompting (Bsharat et al., 2024).</td>
  </tr>
  <tr>
    <td>Making prompt specific</td>
    <td>Based on Best practices for prompt engineering published by OpenAI.<sup>1</sup></td>
  </tr>
  <tr>
    <td>Shortening the prompt</td>
    <td>Based on the experimental result that accuracy can decrease as prompts become longer (Levy et al., 2024).</td>
  </tr>
  <tr>
    <td>Adding necessary information</td>
    <td>One of the strategies used in APET (Kepel and Valogianni, 2024).</td>
  </tr>
</table>

Table 1: Prompt design strategies used in the experiment. The concrete descriptions of each strategy are provided in Appendix C.

development set is used for evaluating prompts in the optimization process. At the end of the optimization, the prompt with the highest score on the development set is tested on the test set excluding the development set.

4.2 Metrics

We use accuracy as the scoring function. When evaluating a prompt using task-solving LLMs, answer parts are first extracted from the responses generated by the LLMs. The regular expression used in lm-evaluation-harness (Gao et al., 2024) is used to extract the answer parts. In the regular expression, the parts following “the answer is ” are extracted. Then, the scoring function gives a value of 1 if they exactly match the desired responses and 0 otherwise.

4.3 Implementation Details

We evaluate the strategy selection methods with DE-based EvoPrompt. We also evaluate GA-based algorithm, which is presented in Appendix D. We use the 3-shot prompts from the original BBH paper (Suzgun et al., 2022), but we optimize only the task description and leave the examples unchanged. We conduct two experiments in which Llama-3-8B-Instruct (Grattafiori et al., 2024) and GPT-4o mini are used as the task-solving LLMs. In the experiments using Llama-3-8B-Instruct, we use all 27 tasks of the BBH. In the experiments using GPT-4o mini, due to the high API cost, we sample only 3 out of the 27 tasks. We run three trials with different random seeds in both our experiments with Llama-3-8B-Instruct and GPT-4o mini. We use GPT-4o mini for the prompt-designing LLMs, set the population size to 10, and perform optimization for 50 iterations. The prompt design strategies we use are listed in Table 1.

4.4 Initial Task Descriptions

The initial task descriptions given at the beginning of the optimization are prepared by the following procedure. First, we select five task descriptions from the 20 prepared descriptions based on their evaluation scores on the development set. The 20 task descriptions consist of those used in the original BBH paper (Suzgun et al., 2022) and their 19 paraphrases generated by GPT-4o mini. For paraphrasing, we use the instruction “Generate 19 variations of the following instruction while keeping the semantic meaning,” which is a slightly modified version of the meta-prompt created by Zhou et al. (2023). Then, we use the 10 task descriptions, consisting of the five selected task descriptions and their respective paraphrases by GPT-4o mini, as the initial task descriptions. When paraphrasing the selected descriptions, in the same way as Guo et al. (2024), we use the meta-prompt for resampling with the instruction “Generate a variation of the following instruction while keeping the semantic meaning,” which is created by Zhou et al. (2023).

4.5 Baseline Methods

We use the manual prompts that use those introduced in the BBH paper, APET, and EvoPrompt as
<table>
  <tr>
    <th>Task ID</th>
    <th>Task Name</th>
    <th>Manual Prompt</th>
    <th>APET</th>
    <th>EvoPrompt(DE)</th>
    <th>EvoPrompt(DE)-OPTS(APET)</th>
    <th>EvoPrompt(DE)-OPTS(US)</th>
    <th>EvoPrompt(DE)-OPTS(TS)</th>
  </tr>
  <tr>
    <td>0</td>
    <td>boolean expressions</td>
    <td>54.00</td>
    <td>67.50</td>
    <td>74.50 (1.08)</td>
    <td>79.83 (2.05)</td>
    <td><b>84.00</b> (0.41)</td>
    <td>82.50 (4.95)</td>
  </tr>
  <tr>
    <td>1</td>
    <td>causal judgement</td>
    <td>2.19</td>
    <td>0.00</td>
    <td>40.39 (3.00)</td>
    <td>42.34 (2.98)</td>
    <td>40.15 (4.88)</td>
    <td><b>45.50</b> (3.00)</td>
  </tr>
  <tr>
    <td>2</td>
    <td>date understanding</td>
    <td>14.00</td>
    <td>3.00</td>
    <td>17.17 (1.03)</td>
    <td>19.17 (4.29)</td>
    <td>18.67 (5.57)</td>
    <td><b>20.17</b> (6.28)</td>
  </tr>
  <tr>
    <td>3</td>
    <td>disambiguation qa</td>
    <td>19.50</td>
    <td>22.50</td>
    <td>30.00 (1.87)</td>
    <td>38.33 (4.11)</td>
    <td><b>47.33</b> (6.54)</td>
    <td>42.50 (5.31)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>dyck languages</td>
    <td>6.50</td>
    <td>0.00</td>
    <td>6.67 (0.85)</td>
    <td>6.50 (0.71)</td>
    <td>6.17 (0.85)</td>
    <td><b>7.50</b> (0.00)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>formal fallacies</td>
    <td>29.50</td>
    <td>0.00</td>
    <td>40.67 (1.31)</td>
    <td><b>44.83</b> (2.66)</td>
    <td>42.50 (1.22)</td>
    <td>43.50 (3.56)</td>
  </tr>
  <tr>
    <td>6</td>
    <td>geometric shapes</td>
    <td>16.50</td>
    <td>24.50</td>
    <td><b>36.00</b> (0.41)</td>
    <td>33.00 (0.41)</td>
    <td>33.00 (4.32)</td>
    <td>35.83 (2.62)</td>
  </tr>
  <tr>
    <td>7</td>
    <td>hyperbaton</td>
    <td>53.00</td>
    <td>3.50</td>
    <td>54.67 (0.85)</td>
    <td><b>70.00</b> (0.82)</td>
    <td>59.50 (5.02)</td>
    <td>60.50 (4.42)</td>
  </tr>
  <tr>
    <td>8</td>
    <td>logical deduction five objects</td>
    <td>12.00</td>
    <td>3.50</td>
    <td>14.67 (1.03)</td>
    <td>29.00 (6.48)</td>
    <td>24.67 (7.15)</td>
    <td><b>37.17</b> (13.21)</td>
  </tr>
  <tr>
    <td>9</td>
    <td>logical deduction seven objects</td>
    <td>5.50</td>
    <td>3.00</td>
    <td>5.83 (0.24)</td>
    <td>10.17 (1.25)</td>
    <td><b>13.17</b> (0.85)</td>
    <td>13.00 (1.87)</td>
  </tr>
  <tr>
    <td>10</td>
    <td>logical deduction three objects</td>
    <td>44.00</td>
    <td>20.50</td>
    <td>45.83 (3.42)</td>
    <td>70.17 (5.10)</td>
    <td>71.83 (2.09)</td>
    <td><b>78.83</b> (5.98)</td>
  </tr>
  <tr>
    <td>11</td>
    <td>movie recommendation</td>
    <td>40.50</td>
    <td>0.00</td>
    <td>49.50 (5.12)</td>
    <td>49.50 (2.83)</td>
    <td>48.00 (2.48)</td>
    <td><b>54.33</b> (1.89)</td>
  </tr>
  <tr>
    <td>12</td>
    <td>multistep arithmetic two</td>
    <td><b>53.50</b></td>
    <td>52.00</td>
    <td><b>53.50</b> (1.78)</td>
    <td>52.17 (2.46)</td>
    <td>50.33 (0.85)</td>
    <td>52.50 (1.47)</td>
  </tr>
  <tr>
    <td>13</td>
    <td>navigate</td>
    <td>84.50</td>
    <td>38.50</td>
    <td>83.17 (0.47)</td>
    <td>80.17 (2.66)</td>
    <td>81.67 (3.40)</td>
    <td><b>84.67</b> (1.03)</td>
  </tr>
  <tr>
    <td>14</td>
    <td>object counting</td>
    <td><b>87.50</b></td>
    <td>27.50</td>
    <td>85.83 (0.62)</td>
    <td>85.67 (0.62)</td>
    <td>85.00 (0.82)</td>
    <td>85.83 (0.24)</td>
  </tr>
  <tr>
    <td>15</td>
    <td>penguins in a table</td>
    <td>26.04</td>
    <td>8.33</td>
    <td>22.57 (3.93)</td>
    <td>28.47 (7.23)</td>
    <td><b>41.67</b> (5.17)</td>
    <td>40.97 (7.90)</td>
  </tr>
  <tr>
    <td>16</td>
    <td>reasoning about colored objects</td>
    <td>21.00</td>
    <td>6.50</td>
    <td><b>53.00</b> (3.54)</td>
    <td>50.00 (4.64)</td>
    <td>45.67 (5.27)</td>
    <td>49.50 (3.27)</td>
  </tr>
  <tr>
    <td>17</td>
    <td>ruin names</td>
    <td>18.50</td>
    <td>65.50</td>
    <td>27.17 (2.90)</td>
    <td><b>68.67</b> (7.26)</td>
    <td>64.83 (2.39)</td>
    <td>67.67 (3.66)</td>
  </tr>
  <tr>
    <td>18</td>
    <td>salient translation error detection</td>
    <td>12.50</td>
    <td>6.50</td>
    <td>46.17 (3.52)</td>
    <td>46.50 (4.71)</td>
    <td><b>51.17</b> (1.55)</td>
    <td><b>51.17</b> (3.66)</td>
  </tr>
  <tr>
    <td>19</td>
    <td>snarks</td>
    <td>24.22</td>
    <td>0.00</td>
    <td>55.47 (6.72)</td>
    <td>58.59 (3.31)</td>
    <td><b>65.10</b> (4.25)</td>
    <td>64.06 (1.10)</td>
  </tr>
  <tr>
    <td>20</td>
    <td>sports understanding</td>
    <td>51.52</td>
    <td>0.00</td>
    <td>61.45 (1.67)</td>
    <td>61.62 (1.43)</td>
    <td>76.94 (7.76)</td>
    <td><b>78.45</b> (4.54)</td>
  </tr>
  <tr>
    <td>21</td>
    <td>temporal sequences</td>
    <td>57.00</td>
    <td>6.00</td>
    <td>65.83 (3.06)</td>
    <td>65.83 (1.18)</td>
    <td>60.33 (0.62)</td>
    <td><b>66.00</b> (1.47)</td>
  </tr>
  <tr>
    <td>22</td>
    <td>tracking shuffled objects five objects</td>
    <td>67.50</td>
    <td>18.00</td>
    <td>66.50 (2.16)</td>
    <td>66.17 (0.94)</td>
    <td>68.17 (1.25)</td>
    <td><b>69.33</b> (1.25)</td>
  </tr>
  <tr>
    <td>23</td>
    <td>tracking shuffled objects seven objects</td>
    <td>47.50</td>
    <td>19.50</td>
    <td>52.00 (1.22)</td>
    <td>52.50 (1.87)</td>
    <td><b>54.33</b> (1.65)</td>
    <td>49.50 (0.71)</td>
  </tr>
  <tr>
    <td>24</td>
    <td>tracking shuffled objects three objects</td>
    <td>80.50</td>
    <td>80.50</td>
    <td>78.67 (1.55)</td>
    <td>77.50 (2.83)</td>
    <td>79.17 (1.65)</td>
    <td><b>81.67</b> (2.01)</td>
  </tr>
  <tr>
    <td>25</td>
    <td>web of lies</td>
    <td>93.50</td>
    <td>42.00</td>
    <td><b>95.00</b> (0.00)</td>
    <td><b>95.00</b> (0.00)</td>
    <td><b>95.00</b> (0.00)</td>
    <td><b>95.00</b> (0.00)</td>
  </tr>
  <tr>
    <td>26</td>
    <td>word sorting</td>
    <td>44.50</td>
    <td>41.00</td>
    <td>45.50 (0.71)</td>
    <td>45.83 (4.11)</td>
    <td><b>46.17</b> (1.70)</td>
    <td>45.33 (2.25)</td>
  </tr>
  <tr>
    <td colspan="2">AVG</td>
    <td>39.52</td>
    <td>20.73</td>
    <td>48.43</td>
    <td>52.87</td>
    <td>53.87</td>
    <td><b>55.67</b></td>
  </tr>
</table>

Table 2: Accuracy on the test set for 27 tasks from BBH, evaluated with Llama-3-8B-Instruct as the task-solving LLM. The scores are averaged over three trials with different seeds. The values in parentheses represent the standard deviation. The bold scores indicate that the prompt optimized by the method achieved the highest average score.

<table>
  <tr>
    <th>Task ID</th>
    <th>EvoPrompt(DE)</th>
    <th>EvoPrompt(DE)-OPTS(TS)</th>
  </tr>
  <tr>
    <td>8</td>
    <td>2.67 (1.43)</td>
    <td><b>52.33</b> (11.45)</td>
  </tr>
  <tr>
    <td>19</td>
    <td>79.17 (0.37)</td>
    <td><b>79.43</b> (1.33)</td>
  </tr>
  <tr>
    <td>23</td>
    <td>80.67 (4.37)</td>
    <td><b>81.83</b> (3.47)</td>
  </tr>
</table>

Table 3: Accuracy on the test set for three tasks from BBH, evaluated with GPT-40 mini. The task IDs are the same as in Table 2. The scores in this table are the average scores over three trials. The values in parentheses represent the standard deviation.

the baseline methods. In APET, we use the APET procedure to incorporate prompt design strategies into task description in manual prompts.

4.6 Main Result

Table 2 shows the test scores using Llama-3-8B-Instruct as the task-solving LLM.

Effects of OPTS. First, we focus on the effect of the explicit selection mechanism in OPTS. Table 2 shows that all three variants of OPTS increase the average scores of EvoPrompt(DE) by approximately 4.5% to 7.5% compared with the naive EvoPrompt(DE). In particular, for the task “ruin names” (task ID 17), all three variants of EvoPrompt(DE)-OPTS outperform EvoPrompt(DE) by about 40%. The results indicate that OPTS can improve the prompt optimizer.

Comparing OPTS Variants. We compare three selection methods: OPTS(TS), OPTS(US), and OPTS(APET). Table 2 shows that, with EvoPrompt(DE), OPTS(TS) outperforms OPTS(US) by about 1.7% and OPTS(APET) by about 2.7% on average. This result indicates that OPTS(TS) can select more suitable strategies for task-solving LLMs and tasks. In addition, OPTS(APET) is inferior to OPTS(US), implying that LLMs have an incorrect bias in selecting prompt design strategies.

Differences between Tasks. Table 2 also shows that the improvement achieved by strategy selection methods varies depending on the task. For example, in the tasks of “ruin names” (task ID 17) and “logical deduction three objects” (task ID
<table>
  <tr>
    <th>Method</th>
    <th>Description</th>
    <th>Score</th>
  </tr>
  <tr>
    <td>Manual Prompt</td>
    <td>Select the humorous edit that 'ruins' the input movie or musical artist name.</td>
    <td>18.50</td>
  </tr>
  <tr>
    <td>EvoPrompt(DE)</td>
    <td>Decide on the eccentric twist that 'spoils' the name of the movie or music artist.</td>
    <td>31.00</td>
  </tr>
  <tr>
    <td>APET</td>
    <td><u>Imagine you are a creative expert in humor and wordplay, skilled at crafting amusing edits that playfully distort movie or musical artist names. Your task is to select the humorous edit that 'ruins' the given input name in a funny and clever way.</u><br>
      <span>(1) Let's think step-by-step: First, identify the original name provided.<br>
      (2) Next, brainstorm potential humorous edits that could transform the name into something amusing while maintaining a connection to the original.<br>
      (2) Finally, choose the edit that best exemplifies the concept of 'ruining' the name in a lighthearted manner.<br>
      Read the question again to ensure clarity before proceeding.<br>
      Remember, your goal is to evoke laughter and joy through your selection!</span></td>
    <td>65.50</td>
  </tr>
  <tr>
    <td>EvoPrompt(DE)-OPTS(TS)</td>
    <td><u>Let's think step-by-step! First, carefully read the question again and identify a movie title or musician whose name lends itself to a humorous spoof.<br>
      (2) Next, creatively reimagine that title or name with an absurdly funny twist that maintains the essence of the original while injecting a comedic element.<br>
      (2) Finally, present your funniest version clearly, ensuring it is both memorable and entertaining.<br>
      Let your creativity shine through in this process!</u></td>
    <td>70.50</td>
  </tr>
</table>

Table 4: Examples of the discovered descriptions (optimized parts within the prompts). These achieved the highest score in each method with Llama-3-8B-Instruct as the task-solving LLM and "ruin names" as the task. Underlined texts represent (1) ExpertPrompting, (2) Chain-of-Thought, (3) Making prompt specific, (4) Style Prompting, (5) Re-Reading, and (6) Emotion Prompting.

10), EvoPrompt(DE)-OPTS(TS) outperforms EvoPrompt(DE) by approximately 40% and 30%, respectively. On the other hand, OPTS degrades the performance of EvoPrompt(DE) in several tasks. A possible reason is that effective prompt design strategies differ for each task. In tasks with significant improvement, the eleven candidate strategies used in the experiment likely include effective options. In contrast, it may be difficult for tasks with performance degradation to achieve better performance using only the strategies available among the eleven candidates. Owing to the inaction arm, we note that the performance degradation is insignificant compared with the degree of performance improvement.

4.7 Results Using Another Task-Solving LLM

We also evaluated EvoPrompt(DE)-OPTS(TS) using another task-solving LLM, GPT-4o mini, which is one of the most widely used LLMs. We conducted experiments on three randomly chosen tasks from BBH. The experimental setup was the same as the experiment using Llama-3-8B-Instruct. Table 3 shows the accuracy of the test set. We observe that EvoPrompt(DE)-OPTS(TS) outperforms EvoPrompt(DE) in all three tasks. In particular, for "logical deduction five objects" (task ID 8), OPTS(TS) increases the accuracy by approximately 50%. This result suggests that OPTS is likely to be effective regardless of the task-solving LLM.

5 Analysis

In this section, we analyze OPTS(TS) from two perspectives: the analyses of the discovered task descriptions and the case where a strategy was applied more than once. Further analyses, including comparison under the same number of API calls, analysis of prompts generated during and after optimization, and analysis of score variance, can be found in the Appendices E to G.
<table>
  <tr>
    <th>Before or After OPTS</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Before</td>
    <td>Let’s think step-by-step. Identify a humorous variation that spoofs the title of a film or music artist, inventing a funny alteration while preserving its original essence. Before providing your answer, ensure full clarity and understanding.</td>
  </tr>
  <tr>
    <td>After</td>
    <td>Let’s think step-by-step. <b>First,</b> identify a film or music artist whose title can be humorously spoofed. <b>Next,</b> brainstorm a funny alteration that captures the essence of the original title while adding a comedic twist. <b>Finally,</b> clearly articulate your humorous variation, ensuring that it maintains the original’s core meaning.</td>
  </tr>
</table>

Table 5: Example of a prompt where Chain-of-Thought prompting is already used, and Chain-of-Thought prompting is selected again and modified accordingly. The task, task-solving LLM, and seed settings are the same as in Table 4.

Analysis of Discovered Descriptions. Table 4 shows the discovered task descriptions when using Llama-3-8B-Instruct as the task-solving LLMs and “ruin names” as the task to be solved. We can see that the task description discovered by EvoPrompt(DE) does not use any prompt design strategies and has a structure similar to that of a manual prompt, which is used to obtain the initial prompts for EvoPrompt. We consider that, although the crossover and mutation performed by the prompt-designing LLM in EvoPrompt can change the phrases in the prompt, it is difficult to change the structure significantly. In addition, unlike EvoPrompt(DE), EvoPrompt(DE)-OPTS(TS) discovered the task description with various prompt design strategies, including CoT, Re-Reading, Making prompt specific, and Emotion Prompting. This result means that OPTS(TS) significantly improves the task description using prompt design strategies. When comparing EvoPrompt(DE)-OPTS(TS) with APET, APET incorporated more strategies while its score was lower than EvoPrompt(DE)-OPTS(TS). Implicit selection by APET has the advantage of selecting and incorporating multiple prompt design strategies at once, but it may also include unnecessary strategies. Furthermore, we observe that the task description of EvoPrompt(DE)-OPTS(TS) uses several prompt design strategies in combination, although OPTS(TS) selects only one prompt design strategy at a time. Indeed, Re-Reading and Making prompt specific are used within the CoT. This shows that EvoPrompt(DE)-OPTS(TS) possesses the ability to combine multiple prompt design strategies as well as APET.

Analysis of Repeated Selection of a Strategy. During the optimization process of EvoPrompt-OPTS(TS), we observed that OPTS(TS) sampled a prompt design strategy that had already been incorporated into the prompt, thereby further modifying it. Table 5 illustrates a case in which CoT was applied again to a prompt within the same trial as Table 4. The example shows that reapplying CoT further aligned the prompt with the strategy. Also, the feature introduced in the step can be observed in the best prompt in Table 4. This example suggests that repeatedly applying the same strategy helps incorporate it more effectively.

6 Conclusion and Discussion

We introduced explicit selection mechanisms into prompt optimization to effectively leverage existing knowledge of prompt design. Experiments have demonstrated that the three methods we introduced improve the performance of the prompt optimizers. In particular, the method based on Thompson sampling is the best among those we compared. The prompts discovered by our methods effectively incorporate several prompt design strategies, which EvoPrompt alone was unable to discover. Our results highlight the importance of leveraging existing knowledge and selecting it explicitly.

Limitations

There are four limitations that remain for future research: (1) We formulated OPTS(TS) as the Bernoulli bandit, but alternative reward formulation may improve optimization performance. (2) Although OPTS can be easily integrated into various prompt optimizers, we have not attempted to introduce it to other optimizers than EvoPrompt. (3) We used Thompson sampling to select the prompt design strategy but did not evaluate other sophisticated methods, such as contextual bandit algorithms. (4) The performance of OPTS mechanism
can vary depending on the prompt design strategy prepared, but this is not clarified in this paper. These points remain topics for future research.

Ethical Considerations

Our method may involve the risk of being used to optimize prompts that generate malicious content, such as malware or fake news, even though this is not the intention of our method. At present, it is extremely difficult to reduce this risk. Although our method may be beneficial to some malicious users, we expect that our method can be more beneficial to many other benevolent users.

Acknowledgments

This work was partially supported by JSPS KAKENHI Grant Number JP23H00491, JST PRESTO Grant Number JPMJPR2133, JST Grant Number JPMJPF2203, NEDO JPNP20006, and a grant from the Kanagawa Prefectural Government of Japan.

References

Eshaan Agarwal, Joykirat Singh, Vivek Dani, Raghav Magazine, Tanuja Ganu, and Akshay Nambi. 2024. PromptWizard: Task-aware prompt optimization framework. Preprint, arXiv:2405.18369.

Sondos Mahmoud Bsharat, Aidar Myrzakhan, and Zhiqiang Shen. 2024. Principled instructions are all you need for questioning LLaMA-1/2, GPT-3.5/4. Preprint, arXiv:2312.16171.

Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Xiaoqian Liu, Tong Xiao, and Jingbo Zhu. 2024. Efficient prompting methods for large language models: A survey. Preprint, arXiv:2404.01077.

Olivier Chapelle and Lihong Li. 2011. An empirical evaluation of Thompson sampling. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc.

Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A. Malin, and Sricharan Kumar. 2024. PhaseEvo: Towards unified long-context prompt optimization for large language models. In First Workshop on Long-Context Foundation Models @ ICML 2024.

Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. 2024. Rephrase and respond: Let large language models ask better questions for themselves. Preprint, arXiv:2311.04205.

Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023. Promptbreeder: Self-referential self-improvement via prompt evolution. Preprint, arXiv:2309.16797.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. A framework for few-shot language model evaluation.

Aaron Grattafiore, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schellen, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783.

Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2024. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations.

Beichen Huang, Xingyu Wu, Yu Zhou, Jibin Wu, Liang Feng, Ran Cheng, and Kay Chen Tan. 2024. Exploring the true potential: Evaluating the black-box optimization capability of large language models. Preprint, arXiv:2404.06290.

Feihu Jin, Yifan Liu, and Ying Tan. 2024. Zero-shot chain-of-thought reasoning guided by evolutionary algorithms in large language models. Preprint, arXiv:2402.05376.

Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270):20230254.

Daan Kepel and Konstantina Valogianni. 2024. Autonomous prompt engineering in large language models. Preprint, arXiv:2407.11000.

Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yu-taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15339–15353, Bangkok, Thailand. Association for Computational Linguistics.

Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang,
and Xing Xie. 2023. Large language models understand and can be enhanced by emotional stimuli. Preprint, arXiv:2307.11760.

Do Long, Yiran Zhao, Hannah Brown, Yuxi Xie, James Zhao, Nancy Chen, Kenji Kawaguchi, Michael Shieh, and Junxian He. 2024. Prompt optimization via adversarial in-context learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7308–7327, Bangkok, Thailand. Association for Computational Linguistics.

Albert Lu, Hongxin Zhang, Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. 2023. Bounding the capabilities of large language models in open text generation with prompt constraints. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1982–2008, Dubrovnik, Croatia. Association for Computational Linguistics.

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of GPT-4 on medical challenge problems. Preprint, arXiv:2303.13375.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. GPT-4 technical report. Preprint, arXiv:2303.08774.

Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023. GRIPS: Gradient-free, edit-based instruction search for prompting large language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3845–3864, Dubrovnik, Croatia. Association for Computational Linguistics.

Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7957–7968, Singapore. Association for Computational Linguistics.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Cantor Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, and 7 others. 2024. Code Llama: Open foundation models for code. Preprint, arXiv:2308.12950.

Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, and 1 others. 2018. A tutorial on thompson sampling. Foundations and Trends® in Machine Learning, 11(1):1–96.

Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, Hyojung Han, Sevin Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, and 12 others. 2024. The prompt report: A systematic survey of prompting techniques. Preprint, arXiv:2406.06608.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. Preprint, arXiv:2210.09261.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Haught, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jinlin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, and 1331 others. 2024. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805.

William R Thompson. 1933. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285–294.

Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhuan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, and Junran Peng. 2024a. RoleLLM: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14743–14777, Bangkok, Thailand. Association for Computational Linguistics.

Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haitian Luo, Jiayou Zhang, Nebojsa Jojic, Eric Xing, and Zhiting Hu. 2024b. PromptAgent: Strategic planning with language models enables expert-level prompt optimization. In The Twelfth International Conference on Learning Representations.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brianichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.

Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2023. ExpertPrompting: Instructing large language models to be distinguished experts. Preprint, arXiv:2305.14688.

Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian-Guang Lou, and
Shuai Ma. 2024. Re-reading improves reasoning in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15549–15575, Miami, Florida, USA. Association for Computational Linguistics.

Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. In The Twelfth International Conference on Learning Representations.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, volume 36, pages 11809–11822. Curran Associates, Inc.

Mingqian Zheng, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. 2024. “When “a helpful assistant” is not really helpful: Personas in system prompts do not improve performances of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15126–15154, Miami, Florida, USA. Association for Computational Linguistics.

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations.

A Meta-Prompt for OPTS

The meta-prompt we used was based on APET (Kepel and Valogianni, 2024) as shown in Table 6. We fed this meta-prompt into the prompt-designing LLMs after replacing the <strategy> tag with descriptions of the prompt design strategies and the <input> tag with the prompt to be modified. For APET and OPTS(APET), the <strategy> tag was replaced with the list of K tags from <strategy 1> to <strategy K>, where the description of each prompt design strategy is inserted.

B Details of EvoPrompt-OPTS

The algorithm that integrates OPTS into EvoPrompt(GA) is shown in Algorithm 2. In the following, we provide supplementary explanations regarding EvoPrompt processes.

Crossover and Mutation. The prompt-designing LLM performs crossover and mutation based on the meta-prompts and prompts fed into them. The meta-prompts for EvoPrompt(GA) and EvoPrompt(DE) are described in Tables 7 and 8, respectively.

In EvoPrompt(GA), two prompts are selected as parents using roulette wheel selection. They are fed into the prompt-designing LLM along with the meta-prompt by replacing <prompt1> and <prompt2> in the meta-prompt provided in Table 7 with them. The LLM then generates an offspring prompt according to the meta-prompt.

In EvoPrompt(DE), four prompts in the current population are used to generate an offspring prompt: two randomly selected prompts \( p_{r_1} \) and \( p_{r_2} \), the current best prompt \( p_{best} \), and a parent prompt. Each prompt in the current population is selected once as the parent prompt. Those four prompts are fed into the prompt-designing LLM by replacing <prompt0>, <prompt1>, <prompt2>, and <prompt3> in Table 8 with the parent prompt, \( p_{r_1} \), \( p_{r_2} \), and \( p_{best} \), respectively. The LLM then performs crossover and mutation to generate an offspring prompt according to the meta-prompt.

C Descriptions of Prompt Design Strategies

Table 9 provides the descriptions for each of the 11 prompt design strategies used in our experiment.

D GA-Based Experimental Results

In this section, we present the GA-based experimental results. Table 10 shows the result of the experiment using Llama-3-8B-Instruct as a task-solving LLM, and Table 11 shows the result of the experiment using GPT-4o mini as a task-solving LLM. Tables 10 and 11 show that, as in the DE-based experiments, the performance of EvoPrompt (GA) is enhanced by selecting suitable prompt design strategy using OPTS. In addition, when comparing the variations of OPTS, we observe a tendency that OPTS(TS) is overall the best, followed by OPTS(US) and OPTS(APET), as in the case of the EvoPrompt(DE). In terms of the extent of improvement, combining the mechanism for selecting a prompt design strategy with EvoPrompt(DE) has a greater impact than that with EvoPrompt(GA). These results suggest that, although the extent of improvement varies slightly depending on the evolutionary algorithm used in the prompt optimizer, the mechanism to select the prompt design strategy can enhance the prompt optimizer regardless of the evolutionary algorithm used.
<table>
  <tr>
    <th>System prompt</th>
    <td>Imagine yourself as an expert in the realm of prompting techniques for LLMs. Your expertise is not just broad, encompassing the entire spectrum of current knowledge on the subject, but also deep, delving into the nuances and intricacies that many overlook. Your job is to reformulate prompts with surgical precision, optimizing them for the most accurate response possible. The reformulated prompt should enable the LLM to always give the correct answer to the question.</td>
  </tr>
  <tr>
    <th>User prompt</th>
    <td>Your available prompting techniques include, but are not limited to the following:

- <strategy>

Your approach is methodical and analytical, yet creative. You use a mixture of the prompting techniques, making sure you pick the right combination for each instruction. You see beyond the surface of a prompt, identifying the core objectives and the best ways to articulate them to achieve the desired outcomes.

Output instructions:""""
You should ONLY return the reformulated prompt. Make sure to include ALL information from the given prompt to reformulate.

""""

Given above information and instructions, reformulate below prompt using the techniques provided: """"
<input>
""""
</td>
  </tr>
</table>

Table 6: Meta-prompt for OPTS and APET (Kepel and Valogianni, 2024). When APET or OPTS(APET) is used, \( K \) <strategy> tags (i.e., <strategy 1>,<strategy 2>,…,<strategy \( K \)> ) are provided, and each of them is replaced with one prompt design strategy description.

E  Cost Analysis

We compare the performances of EvoPrompt-OPTS(TS) and EvoPrompt under the same number of API calls for GPT-4o mini. Table 12 shows the scores on the test set for prompts with the highest performance on the development set under the constraint of 500 API calls. We use GPT-4o mini as the prompt-designing LLM and Llama-3-8B-Instruct as the task-solving LLM. Table 12 explains that EvoPrompt-OPTS(TS) outperforms EvoPrompt under the same number of API calls. This result means that the proposed method is more cost-effective.

F  Analysis of Prompts Generated During and After Optimization

We analyze the generated task descriptions during and after prompt optimization. We confirm that different strategies are used in different tasks, and not all prompt design strategies are used. We also observe that the order and way in which the strategies are incorporated are different, even if the same combination of strategies is chosen. For instance, we pick up the prompts obtained during the optimization for the task “boolean expressions.” The generated prompts are listed in Table 13. In the third and sixth iterations, the prompts incorporated the strategies called “Re-Reading” and “Rephrase and Respond” were generated, respectively. However, the scores of these prompts on the validation dataset were worse than that of the final best prompt. In addition, both prompt design strategies have not remained in the final best prompt. In the final iteration, the Tree-of-Thought strategy was adopted, contrasting with the result observed for another task in Table 4. From these observations, we conclude that suitable strategies for the task were selected during the optimization process.

G  Analysis of Score Variance

We analyze the score variance of EvoPrompt-OPTS and EvoPrompt. Tables 2 and 10 show that EvoPrompt-OPTS has a higher variance compared to EvoPrompt. The cause is that OPTS sometimes finds significantly superior prompts in some tasks. Tables 14 and 15 show the average (in 27 tasks) of the minimum, median, and maximum scores in three trials. We can see that the averages of the minimum scores of EvoPrompt-OPTS(TS) match the averages of the maximum scores of EvoPrompt. This implies that EvoPrompt-OPTS(TS) usually finds better or comparative solutions than
User prompt    Please follow the instruction step-by-step to generate a better prompt.
1. Crossover the following prompts to generate a new prompt:
Prompt 1: Your task is to classify the comment as one of the following categories: terrible, bad, okay, good, great.
Prompt 2: In this task, you are given sentences from movie reviews. The task is to classify a sentence as one of the following categories: terrible, bad, okay, good, great.
2. Mutate the prompt generated in Step 1 and generate a final prompt bracketed with <prompt> and </prompt>.

1. Crossover Prompt: In this task, you are given comments from movie reviews. Your task is to classify each comment as one of the following categories: terrible, bad, okay, good, great.
2. <prompt>Given a sentence from a movie review, classify it into one of the following categories: terrible, bad, okay, good, or great.</prompt>

Please follow the instruction step-by-step to generate a better prompt.
1. Crossover the following prompts and generate a new prompt:
Prompt 1: <prompt1>
Prompt 2: <prompt2>
2. Mutate the prompt generated in Step 1 and generate a final prompt bracketed with <prompt> and </prompt>.

1.

Table 7: Meta-prompt for crossover and mutation in EvoPrompt(GA) (Guo et al., 2024).

EvoPrompt.

H Discussion of Applicability to Other Tasks

We discuss the applicability of EvoPrompt-OPTS to tasks other than BBH. Basically, similar to BBH, the proposed methods can be applied to other tasks like sentiment classification, MedQA, etc., while several modifications will be needed regarding the initial prompts and regular expressions used to extract the answer part. For instance, in initial prompts for sentiment classification, the user should provide descriptions of the task, such as “Classify the given sentence as either positive or negative.” Also, in order to successfully extract answers, providing few-shot examples and modifying regular expressions will be needed. As long as we can extract the answer part from the response of the task-solving LM and compare it with the correct answer, we believe that it is not difficult to apply our method to other tasks.

I License for Artifacts

BIG-Bench Hard and lm-evaluation-harness are licensed under the MIT License. Llama-3-8B-Instruct is licensed under META LLAMA 3 COMMUNITY LICENSE AGREEMENT. Our code is also released under the MIT license at https://github.com/shiralab/OPTS.

J Artifact Use Consistent with Intended Use

We declare that we have used the BIG-Bench Hard dataset and Llama-3-8B-Instruct in accordance with their original intended use. Additionally, we prohibit the use of the code we release for optimizing prompts to generate malicious content, except for research purposes.

K Experimental Environment

Our experiments were conducted on a computer running Ubuntu 22.04 with an AMD EPYC 7502P CPU and an NVIDIA A100 GPU, and on another computer running Ubuntu 22.04 with an AMD EPYC 7702P CPU and an NVIDIA A100 GPU. We used openai 1.40.8 as the python library to access GPT-4o mini, and vllm 0.6.3.post1 as the python library to access Llama-3-8B-Instruct.
Algorithm 2 EvoPrompt(GA)-OPTS

Require: Initial prompts \( \mathcal{P}_0 = \{p_1, p_2, \ldots, p_N\} \), population size \( N \), number of iterations \( T \), development set \( D_{dev} \) consisting of input and correct output pairs \( (x, y) \), scoring function \( g \), task-solving LLM \( f_T \)

1: **Evaluation** of initial prompts: \( \mathcal{S}_0 \leftarrow \left\{ s_i = \frac{1}{|D_{dev}|} \sum_{(x,y) \in D_{dev}} g\left(y, f_T(p_i, x)\right) : p_i \in \mathcal{P}_0 \right\} \)
2: **for** \( t = 1 \) **to** \( T \) **do**
3:   **for** \( i = 1 \) **to** \( N \) **do**
4:     Sampling parents by roulette wheel: \( p_{r1}, p_{r2} \in \mathcal{P}_{t-1} \)
5:     **Crossover and Mutation**: \( p'_i \leftarrow f_D(m_{ga}, (p_{r1}, p_{r2})) \)
6:       \( m_{ga} \): Meta-prompt for GA-based crossover and mutation
7:       \( f_D \): prompt-designing LLM
8:     OPTS: Generate \( p''_i \) from \( p'_i \) by incorporating prompt design strategies (Refer to Section 3.1)
9:     **Evaluation**: \( s''_i \leftarrow \frac{1}{|D_{dev}|} \sum_{(x,y) \in D_{dev}} g\left(y, f_T(p''_i, x)\right) \)
10:    Update probability distribution if the TS-based selection is used (Refer to Section 3.1)
11:   **end for**
12:  \( \hat{\mathcal{S}}_t \leftarrow \{s''_i : 1 \leq i \leq N\}, \hat{\mathcal{P}}_t \leftarrow \{p''_i : 1 \leq i \leq N\} \)
13:  **Update score**: \( \mathcal{S}_t \leftarrow \) select the best \( N \) scores in \( \mathcal{S}_{t-1} \cup \hat{\mathcal{S}}_t \)
14:  **Update**: \( \mathcal{P}_t \leftarrow \) select the best \( N \) prompts in \( \mathcal{P}_{t-1} \cup \hat{\mathcal{P}}_t \) according to \( \mathcal{S}_{t-1} \cup \hat{\mathcal{S}}_t \),
15: **end for**
16: **Return** the best prompt \( p^* = \argmax_{p \in \mathcal{P}_T} \frac{1}{|D_{dev}|} \sum_{(x,y) \in D_{dev}} g\left(y, f_T(p, x)\right) \)
User prompt
Please follow the instruction step-by-step to generate a better prompt.
1. Identify the different parts between the Prompt 1 and Prompt 2:
Prompt 1: Rewrite the input text into simpler text.
Prompt 2: Rewrite my complex sentence in simpler terms, but keep the meaning.
2. Randomly mutate the different parts
3. Combine the different parts with Prompt 3, selectively replace it with the different parts in step 2 and generate a new prompt.
Prompt 3: Rewrite the given input text into simpler English sentences while preserving the same meaning, so it can be understood by non-native English speakers.
4. Crossover the prompt in the step3 with the following basic prompt and generate a final prompt bracketed with <prompt> and </prompt>:
Basic Prompt: Make the sentence easier for people who do not speak English fluently to comprehend.

1. Identifying the different parts between Prompt 1 and Prompt 2:
Prompt 1: Rewrite the input text into simpler text.
Prompt 2: Rewrite my complex sentence in simpler terms, but keep the meaning.
Different parts:
"input text" vs "my complex sentence"
"simpler text" vs "simpler terms, but keep the meaning"

2. Randomly mutate the different parts:
"input text" -> "provided text"
"my complex sentence" -> "the difficult sentence"
"simpler text" -> "easier language"
"simpler terms, but keep the meaning" -> "simpler words while maintaining the meaning"

3. Combine the different parts with Prompt 3, selectively replace it with the different parts in step 2 and generate a new prompt:
Prompt 3: Rewrite the given input text into simpler English sentences while preserving the same meaning, so it can be understood by non-native English speakers.
New Prompt: Transform the provided text into easier language while maintaining the meaning, making it accessible for non-native English speakers.

4. Crossover the prompt in step 3 with the following basic prompt and generate a final prompt bracketed with <prompt> and </prompt>:
Basic Prompt: Make the sentence easier for people who do not speak English fluently to comprehend.
Final Prompt: <prompt>Convert the difficult sentence into simpler words while preserving the meaning, so it's easier for non-native English speakers to understand.</prompt>

Please follow the instruction step-by-step to generate a better prompt.
1. Identify the different parts between the Prompt 1 and Prompt 2:
Prompt 1: <prompt1>
Prompt 2: <prompt2>
2. Randomly mutate the different parts
3. Combine the different parts with Prompt 3, selectively replace it with the different parts in step2 and generate a new prompt.
Prompt 3: <prompt3>
4. Crossover the prompt in the step3 with the following basic prompt and generate a final prompt bracketed with <prompt> and </prompt>:
Basic Prompt: <prompt0>

1. "":

Table 8: Meta-prompt for crossover and mutation in EvoPrompt(DE) (Guo et al., 2024).
<table>
  <tr>
    <th>ExpertPrompting</th>
    <td>Crafting an expert who is an expert at the given task, by writing a high-quality description about the most capable and suitable agent to answer the instruction in second person perspective.</td>
  </tr>
  <tr>
    <th>Chain-of-Thought</th>
    <td>Explaining step-by-step how the problem should be tackled, and making sure the model explains step-by-step how it came to the answer. You can do this by adding "Let’s think step-by-step".</td>
  </tr>
  <tr>
    <th>Tree-of-Thought</th>
    <td>Imagining three different experts who are discussing the problem at hand. All experts will write down 1 step of their thinking, then share it with the group. Then all experts will go on to the next step, etc. If any expert realises they’re wrong at any point then they leave.</td>
  </tr>
  <tr>
    <th>Adding necessary information</th>
    <td>Making sure all information needed is in the prompt, adding where necessary but making sure the question remains having the same objective.</td>
  </tr>
  <tr>
    <th>Re-Reading</th>
    <td>For a given prompt, add a phrase such as “Read the question again” that instructs the Large Language Models to reread the question before generating an answer. This strategy is particularly effective for complex tasks and helps enhance the quality and reliability of the model’s outputs.</td>
  </tr>
  <tr>
    <th>Style Prompting</th>
    <td>Clearly define the desired style in the given prompt. For example, you might say, “Write a formal letter about...” or “Create a casual conversation discussing...”. This guidance helps the model produce text that matches the requested stylistic elements, whether it’s formal, informal, technical, or poetic.</td>
  </tr>
  <tr>
    <th>Rephrase and Respond</th>
    <td>For a given prompt, add a phrase that instructs the Large Language Models to rephrase the question before responding, such as “Rephrase and expand the question, and respond.</td>
  </tr>
  <tr>
    <th>Making prompt specific</th>
    <td>Make the description of the given prompt more specific. This makes it easier for Large Language Models to correctly execute prompt instructions.</td>
  </tr>
  <tr>
    <th>Avoiding bias</th>
    <td>To allow Large Language Models to make logical and unbiased inferences, add phrases to a given prompt that instruct it to remove opinionated content. This helps the model concentrate on providing responses based on careful analysis and logical reasoning, minimizing biases.</td>
  </tr>
  <tr>
    <th>Shortening the prompt</th>
    <td>If a given prompt has long instructions, make it shorter by condensing it to only the essential parts.</td>
  </tr>
  <tr>
    <th>Emotion Prompting</th>
    <td>At the end of the prompt, add a phrase that evokes a strong emotion. When doing so, keep the following four points in mind:
      <ol type="1">
        <li>Define emotional goals: Identify the emotional response you want to evoke, such as encouragement, motivation, or reassurance.</li>
        <li>Use positive language: Incorporate words and phrases that are positive and supportive. Examples include “believe in your abilities,” “excellent,” “success,” and “outstanding achievements”.</li>
        <li>Emphasize key words: Use techniques like exclamation marks and capitalized words to highlight important aspects and to enhance the emotional impact.</li>
        <li>Incorporate social and self-esteem cues: Design stimuli that leverage social influence (e.g., group membership, others’ opinions) and boost self-esteem and motivation. This can help regulate the emotional response of the Large Language Models and tap into intrinsic motivation.</li>
      </ol>
    </td>
  </tr>
</table>

Table 9: Descriptions of each prompt design strategy. Descriptions of ExpertPrompting, Chain-of-Thought, Tree-of-Thought, and Adding necessary information are quoted from APET (Kepel and Valogianni, 2024). The descriptions Re-Reading, Style Prompting, Rephrase and Respond, and Emotion Prompting are modified versions of descriptions created using GPT-4o from Xu et al. (2024), Lu et al. (2023), Deng et al. (2024), and Li et al. (2023), respectively. Description of Making prompt specific is created by us and is inspired by OpenAI’s prompt engineering best practice. Description of Avoiding bias is created by us by referring to and generalizing the 13th principle of the 26 prompting principles proposed in Bsharat et al. (2024). Description of Shortening the prompt was created by us.
<table>
  <tr>
    <th>Task ID</th>
    <th>Task name</th>
    <th>Manual prompt</th>
    <th>APET</th>
    <th>EvoPrompt(GA)</th>
    <th>EvoPrompt(GA)-OPTS(APET)</th>
    <th>EvoPrompt(GA)-OPTS(US)</th>
    <th>EvoPrompt(GA)-OPTS(TS)</th>
  </tr>
  <tr><td>0</td><td>boolean expressions</td><td>54.00</td><td>67.50</td><td>72.83 (3.30)</td><td>84.33 (3.66)</td><td>83.00 (2.94)</td><td><b>85.67</b> (1.65)</td></tr>
  <tr><td>1</td><td>causal judgement</td><td>2.19</td><td>0.00</td><td>37.71 (1.82)</td><td>40.88 (2.60)</td><td><b>44.53</b> (3.10)</td><td><b>44.53</b> (0.60)</td></tr>
  <tr><td>2</td><td>date understanding</td><td>14.00</td><td>3.00</td><td>13.67 (0.85)</td><td>14.50 (5.10)</td><td><b>21.17</b> (10.38)</td><td>16.00 (0.82)</td></tr>
  <tr><td>3</td><td>disambiguation qa</td><td>19.50</td><td>22.50</td><td>32.17 (4.33)</td><td>37.00 (2.86)</td><td>42.67 (3.27)</td><td><b>49.67</b> (6.91)</td></tr>
  <tr><td>4</td><td>dyck languages</td><td>6.50</td><td>0.00</td><td><b>6.67</b> (0.94)</td><td>6.50 (0.71)</td><td>5.83 (0.24)</td><td>6.50 (0.00)</td></tr>
  <tr><td>5</td><td>formal fallacies</td><td>29.50</td><td>0.00</td><td>42.50 (0.71)</td><td><b>45.17</b> (2.05)</td><td>43.83 (0.85)</td><td>44.50 (0.82)</td></tr>
  <tr><td>6</td><td>geometric shapes</td><td>16.50</td><td>24.50</td><td>29.83 (7.85)</td><td><b>37.33</b> (6.14)</td><td>33.00 (4.26)</td><td>32.67 (2.95)</td></tr>
  <tr><td>7</td><td>hyperbaton</td><td>53.00</td><td>3.50</td><td>54.83 (0.94)</td><td>61.67 (1.55)</td><td>57.67 (3.57)</td><td><b>63.67</b> (6.51)</td></tr>
  <tr><td>8</td><td>logical deduction five objects</td><td>12.00</td><td>3.50</td><td>12.33 (1.31)</td><td>20.00 (1.22)</td><td>24.67 (4.03)</td><td><b>25.83</b> (7.96)</td></tr>
  <tr><td>9</td><td>logical deduction seven objects</td><td>5.50</td><td>3.00</td><td>6.83 (3.01)</td><td>7.83 (3.09)</td><td><b>12.17</b> (1.55)</td><td>11.33 (1.70)</td></tr>
  <tr><td>10</td><td>logical deduction three objects</td><td>44.00</td><td>20.50</td><td>58.17 (5.72)</td><td>64.00 (3.89)</td><td>59.67 (6.02)</td><td><b>66.67</b> (3.30)</td></tr>
  <tr><td>11</td><td>movie recommendation</td><td>40.50</td><td>0.00</td><td>48.67 (1.18)</td><td>49.67 (4.17)</td><td><b>53.50</b> (4.90)</td><td>52.50 (1.63)</td></tr>
  <tr><td>12</td><td>multistep arithmetic two</td><td><b>53.50</b></td><td>52.00</td><td>50.33 (2.25)</td><td>49.83 (1.25)</td><td>46.83 (2.72)</td><td>50.17 (1.03)</td></tr>
  <tr><td>13</td><td>navigate</td><td><b>84.50</b></td><td>38.50</td><td>79.67 (4.25)</td><td>80.17 (2.32)</td><td>83.83 (3.01)</td><td>82.33 (2.25)</td></tr>
  <tr><td>14</td><td>object counting</td><td><b>87.50</b></td><td>27.50</td><td>85.50 (0.71)</td><td>82.67 (3.70)</td><td>86.67 (0.62)</td><td>84.33 (0.62)</td></tr>
  <tr><td>15</td><td>penguins in a table</td><td>26.04</td><td>8.33</td><td>25.00 (1.47)</td><td>29.86 (8.26)</td><td>30.90 (2.99)</td><td><b>32.29</b> (1.47)</td></tr>
  <tr><td>16</td><td>reasoning about colored objects</td><td>21.00</td><td>6.50</td><td>50.17 (9.33)</td><td>47.50 (1.08)</td><td>47.00 (6.38)</td><td><b>52.33</b> (2.72)</td></tr>
  <tr><td>17</td><td>ruin names</td><td>18.50</td><td>65.50</td><td>51.00 (13.83)</td><td>63.17 (3.52)</td><td><b>67.33</b> (0.24)</td><td>66.83 (0.47)</td></tr>
  <tr><td>18</td><td>salient translation error detection</td><td>12.50</td><td>6.50</td><td>31.67 (9.74)</td><td>49.00 (4.42)</td><td>48.17 (4.48)</td><td><b>52.67</b> (1.03)</td></tr>
  <tr><td>19</td><td>snarks</td><td>24.22</td><td>0.00</td><td>45.83 (6.03)</td><td>52.60 (15.94)</td><td><b>64.84</b> (2.92)</td><td>60.16 (4.42)</td></tr>
  <tr><td>20</td><td>sports understanding</td><td>51.52</td><td>0.00</td><td>62.63 (0.00)</td><td>62.63 (0.00)</td><td><b>65.15</b> (2.58)</td><td>61.95 (6.20)</td></tr>
  <tr><td>21</td><td>temporal sequences</td><td>57.00</td><td>6.00</td><td><b>63.67</b> (3.70)</td><td>59.33 (2.49)</td><td>61.83 (1.43)</td><td>61.33 (2.78)</td></tr>
  <tr><td>22</td><td>tracking shuffled objects five objects</td><td>67.50</td><td>18.00</td><td>67.67 (0.47)</td><td>67.83 (0.24)</td><td><b>68.67</b> (2.66)</td><td>67.33 (1.03)</td></tr>
  <tr><td>23</td><td>tracking shuffled objects seven objects</td><td>47.50</td><td>19.50</td><td><b>53.33</b> (2.49)</td><td>51.50 (0.82)</td><td>52.00 (1.08)</td><td>50.33 (1.84)</td></tr>
  <tr><td>24</td><td>tracking shuffled objects three objects</td><td>80.50</td><td>80.50</td><td>81.00 (0.71)</td><td>81.67 (0.62)</td><td>80.67 (0.24)</td><td><b>82.33</b> (0.62)</td></tr>
  <tr><td>25</td><td>web of lies</td><td>93.50</td><td>42.00</td><td><b>96.33</b> (1.03)</td><td>95.83 (0.62)</td><td><b>96.33</b> (1.43)</td><td>96.00 (1.47)</td></tr>
  <tr><td>26</td><td>word sorting</td><td>44.50</td><td>41.00</td><td><b>46.83</b> (1.65)</td><td>43.67 (4.40)</td><td>46.00 (7.08)</td><td>44.17 (1.03)</td></tr>
  <tr><th colspan="2">AVG</th><th>39.52</th><th>20.73</th><th>48.40</th><th>51.34</th><th>52.89</th><th><b>53.48</b></th></tr>
</table>

Table 10: Accuracy on the test set for 27 tasks from BBH, evaluated with Llama-3-8B-Instruct as the task-solving LLM. The scores are averaged over three trials with different seeds. The values in parentheses represent the standard deviation. The bold scores indicate that the prompt optimized by the method achieved the highest average score.

<table>
  <tr>
    <th>Task ID</th>
    <th>EvoPrompt(GA)</th>
    <th>EvoPrompt(GA)-OPTS(TS)</th>
  </tr>
  <tr>
    <td>8</td>
    <td>1.67 (0.85)</td>
    <td><b>48.17</b> (5.14)</td>
  </tr>
  <tr>
    <td>19</td>
    <td><b>79.43</b> (1.33)</td>
    <td>78.65 (1.84)</td>
  </tr>
  <tr>
    <td>23</td>
    <td>75.67 (1.25)</td>
    <td><b>88.33</b> (1.55)</td>
  </tr>
</table>

Table 11: Accuracy on the test set for three tasks from BBH, evaluated with GPT-4o mini. The task IDs are the same as in Table 2. The scores in the table are the average scores over three trials. The values in parentheses represent the standard deviation.
<table>
  <tr>
    <th>Task ID</th>
    <th>Task Name</th>
    <th>EvoPrompt(GA)-OPTS(TS)</th>
    <th>EvoPrompt(GA)-OPTS(TS)</th>
    <th>EvoPrompt(DE)-OPTS(TS)</th>
    <th>EvoPrompt(DE)-OPTS(TS)</th>
  </tr>
  <tr>
    <td>0</td>
    <td>boolean expressions</td>
    <td>72.83 (3.30)</td>
    <td><b>80.50</b> (2.48)</td>
    <td>74.50 (1.08)</td>
    <td><b>82.67</b> (2.39)</td>
  </tr>
  <tr>
    <td>1</td>
    <td>causal judgement</td>
    <td>37.71 (1.82)</td>
    <td><b>44.28</b> (0.69)</td>
    <td>40.39 (3.00)</td>
    <td><b>42.82</b> (2.26)</td>
  </tr>
  <tr>
    <td>2</td>
    <td>date understanding</td>
    <td>13.67 (0.85)</td>
    <td><b>15.50</b> (0.41)</td>
    <td><b>17.17</b> (1.03)</td>
    <td>15.17 (1.03)</td>
  </tr>
  <tr>
    <td>3</td>
    <td>disambiguation qa</td>
    <td>32.17 (4.33)</td>
    <td><b>41.67</b> (6.36)</td>
    <td>30.00 (1.87)</td>
    <td><b>40.83</b> (4.52)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>dyck languages</td>
    <td><b>6.67</b> (0.94)</td>
    <td>6.50 (0.00)</td>
    <td>6.67 (0.85)</td>
    <td><b>7.33</b> (0.24)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>formal fallacies</td>
    <td>42.50 (0.71)</td>
    <td><b>43.67</b> (0.62)</td>
    <td>40.67 (1.31)</td>
    <td><b>43.50</b> (3.56)</td>
  </tr>
  <tr>
    <td>6</td>
    <td>geometric shapes</td>
    <td>29.83 (7.85)</td>
    <td><b>32.17</b> (3.17)</td>
    <td>36.00 (0.41)</td>
    <td><b>36.17</b> (2.49)</td>
  </tr>
  <tr>
    <td>7</td>
    <td>hyperbaton</td>
    <td>54.83 (0.94)</td>
    <td><b>57.67</b> (2.90)</td>
    <td>54.67 (0.85)</td>
    <td><b>57.33</b> (0.24)</td>
  </tr>
  <tr>
    <td>8</td>
    <td>logical deduction five objects</td>
    <td>12.33 (1.31)</td>
    <td><b>23.83</b> (2.62)</td>
    <td>14.67 (1.03)</td>
    <td><b>32.17</b> (10.60)</td>
  </tr>
  <tr>
    <td>9</td>
    <td>logical deduction seven objects</td>
    <td>6.83 (3.01)</td>
    <td><b>11.17</b> (1.55)</td>
    <td>5.83 (0.24)</td>
    <td><b>10.50</b> (3.19)</td>
  </tr>
  <tr>
    <td>10</td>
    <td>logical deduction three objects</td>
    <td>58.17 (5.72)</td>
    <td><b>67.33</b> (9.78)</td>
    <td>45.83 (3.42)</td>
    <td><b>77.17</b> (5.95)</td>
  </tr>
  <tr>
    <td>11</td>
    <td>movie recommendation</td>
    <td>48.67 (1.18)</td>
    <td><b>52.50</b> (1.63)</td>
    <td>49.50 (5.12)</td>
    <td><b>51.50</b> (2.86)</td>
  </tr>
  <tr>
    <td>12</td>
    <td>multistep arithmetic two</td>
    <td><b>50.33</b> (2.25)</td>
    <td>48.50 (3.89)</td>
    <td><b>53.50</b> (1.78)</td>
    <td>52.50 (1.47)</td>
  </tr>
  <tr>
    <td>13</td>
    <td>navigate</td>
    <td>79.67 (4.25)</td>
    <td><b>81.67</b> (1.31)</td>
    <td><b>83.17</b> (0.47)</td>
    <td>82.50 (2.83)</td>
  </tr>
  <tr>
    <td>14</td>
    <td>object counting</td>
    <td><b>85.50</b> (0.71)</td>
    <td>85.50 (1.08)</td>
    <td>85.83 (0.62)</td>
    <td><b>86.17</b> (0.24)</td>
  </tr>
  <tr>
    <td>15</td>
    <td>penguins in a table</td>
    <td>25.00 (1.47)</td>
    <td><b>30.21</b> (1.47)</td>
    <td>22.57 (3.93)</td>
    <td><b>30.90</b> (6.44)</td>
  </tr>
  <tr>
    <td>16</td>
    <td>reasoning about colored objects</td>
    <td>50.17 (9.33)</td>
    <td><b>51.33</b> (3.66)</td>
    <td><b>53.00</b> (3.54)</td>
    <td>48.00 (1.47)</td>
  </tr>
  <tr>
    <td>17</td>
    <td>ruin names</td>
    <td>51.00 (13.83)</td>
    <td><b>65.17</b> (2.62)</td>
    <td>27.17 (2.90)</td>
    <td><b>68.17</b> (2.25)</td>
  </tr>
  <tr>
    <td>18</td>
    <td>salient translation error detection</td>
    <td>31.67 (9.74)</td>
    <td><b>52.17</b> (0.24)</td>
    <td>46.17 (3.52)</td>
    <td><b>51.83</b> (2.72)</td>
  </tr>
  <tr>
    <td>19</td>
    <td>snarks</td>
    <td>45.83 (6.03)</td>
    <td><b>60.16</b> (7.99)</td>
    <td><b>55.47</b> (6.72)</td>
    <td>55.21 (3.51)</td>
  </tr>
  <tr>
    <td>20</td>
    <td>sports understanding</td>
    <td>62.63 (0.00)</td>
    <td><b>64.81</b> (3.10)</td>
    <td>61.45 (1.67)</td>
    <td><b>80.81</b> (7.56)</td>
  </tr>
  <tr>
    <td>21</td>
    <td>temporal sequences</td>
    <td><b>63.67</b> (3.70)</td>
    <td>60.67 (2.66)</td>
    <td><b>65.83</b> (3.06)</td>
    <td>64.67 (1.84)</td>
  </tr>
  <tr>
    <td>22</td>
    <td>tracking shuffled objects five objects</td>
    <td><b>67.67</b> (0.47)</td>
    <td>66.33 (0.47)</td>
    <td>66.50 (2.16)</td>
    <td><b>69.33</b> (1.25)</td>
  </tr>
  <tr>
    <td>23</td>
    <td>tracking shuffled objects seven objects</td>
    <td><b>53.33</b> (2.49)</td>
    <td>51.00 (0.41)</td>
    <td><b>52.00</b> (1.22)</td>
    <td>51.17 (2.49)</td>
  </tr>
  <tr>
    <td>24</td>
    <td>tracking shuffled objects three objects</td>
    <td>81.00 (0.71)</td>
    <td><b>81.50</b> (0.41)</td>
    <td>78.67 (1.55)</td>
    <td><b>81.67</b> (2.01)</td>
  </tr>
  <tr>
    <td>25</td>
    <td>web of lies</td>
    <td><b>96.33</b> (1.03)</td>
    <td>95.00 (0.00)</td>
    <td><b>95.00</b> (0.00)</td>
    <td><b>95.00</b> (0.00)</td>
  </tr>
  <tr>
    <td>26</td>
    <td>word sorting</td>
    <td>46.83 (1.65)</td>
    <td><b>47.83</b> (1.93)</td>
    <td>45.50 (0.71)</td>
    <td><b>49.83</b> (0.94)</td>
  </tr>
  <tr>
    <td colspan="2">AVG</td>
    <td>48.40</td>
    <td><b>52.54</b></td>
    <td>48.43</td>
    <td><b>54.26</b></td>
  </tr>
</table>

Table 12: Accuracy on the test set for 27 tasks from BBH, evaluated with Llama-3-8B-Instruct as the task-solving LLM. The prompts with the highest score on the development set are evaluated on the test set, after 500 API calls for GPT-4o mini. The scores are averaged over three trials with different seeds. The values in parentheses represent the standard deviation. The bold scores indicate that the prompt optimized by the method achieved the highest average score.
<table>
  <tr>
    <th>Iteration</th>
    <th>Generated description</th>
    <th>Validation Score</th>
  </tr>
  <tr>
    <td>3</td>
    <td>"Read the question again: Evaluate the answer of an arbitrary Boolean expression carefully, ensuring all logical operations are considered."</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>6</td>
    <td>Rephrase and expand the question, and respond: "Assume the role of a specialist in Boolean algebra with a strong background in assessing Boolean expressions. Determine the value of a randomly generated Boolean expression by analyzing the components step-by-step. Please explain your reasoning clearly while considering all relevant variables and logical operations to arrive at the final outcome."</td>
    <td>0.12</td>
  </tr>
  <tr>
    <td>50 (final)</td>
    <td>Imagine three specialists in Boolean analysis engaged in a structured discussion to analyze the outcome of a randomly generated Boolean expression step-by-step. Each specialist will first note and record the distinct elements of the expression individually. After this initial step, they will share their insights with one another and collaboratively analyze each element in sequence, providing detailed explanations of their rationale at each stage. If any specialist realizes they have made an error at any point, they will withdraw from the discussion. Ultimately, they will integrate their insights to arrive at the final result of the Boolean expression.</td>
    <td>0.88</td>
  </tr>
</table>

Table 13: Examples of the generated descriptions (optimized parts within the prompts) during prompt optimization. These descriptions are generated by EvoPrompt(DE)-OPTS(TS) with Llama-3-8B-Instruct as the task-solving LLM and “boolean expressions” as the task.

<table>
  <tr>
    <th>Method</th>
    <th>Avg Min</th>
    <th>Avg Median</th>
    <th>Avg Max</th>
  </tr>
  <tr>
    <td>EvoPrompt(DE)</td>
    <td>45.95</td>
    <td>48.72</td>
    <td>50.62</td>
  </tr>
  <tr>
    <td>EvoPrompt(DE)-OPTS(APET)</td>
    <td>49.09</td>
    <td>53.59</td>
    <td>55.93</td>
  </tr>
  <tr>
    <td>EvoPrompt(DE)-OPTS(US)</td>
    <td>50.45</td>
    <td>53.77</td>
    <td>57.39</td>
  </tr>
  <tr>
    <td>EvoPrompt(DE)-OPTS(TS)</td>
    <td>51.83</td>
    <td>55.93</td>
    <td>59.24</td>
  </tr>
</table>

Table 14: Averages of the minimum, median, and maximum scores in three trials with DE-based EvoPrompt.

<table>
  <tr>
    <th>Method</th>
    <th>Avg Min</th>
    <th>Avg Median</th>
    <th>Avg Max</th>
  </tr>
  <tr>
    <td>EvoPrompt(GA)</td>
    <td>44.37</td>
    <td>48.86</td>
    <td>51.98</td>
  </tr>
  <tr>
    <td>EvoPrompt(GA)-OPTS(APET)</td>
    <td>47.33</td>
    <td>51.83</td>
    <td>54.86</td>
  </tr>
  <tr>
    <td>EvoPrompt(GA)-OPTS(US)</td>
    <td>49.12</td>
    <td>53.08</td>
    <td>56.45</td>
  </tr>
  <tr>
    <td>EvoPrompt(GA)-OPTS(TS)</td>
    <td>50.80</td>
    <td>53.32</td>
    <td>56.34</td>
  </tr>
</table>

Table 15: Averages of the minimum, median, and maximum scores in three trials with GA-based EvoPrompt.

Reviewer paper summaries: This paper introduces a new method for prompt optimization called Optimizing Prompts with sTrategy Selection (OPTS).  The method explicitly selects prompt design strategies, which are best practices for improving prompt performance.  The authors propose three selection mechanisms, including one based on Thompson sampling.  They integrate these mechanisms into EvoPrompt, a well-known prompt optimizer.  Experiments show that the selection of prompt design strategies improves the performance of EvoPrompt, and the Thompson sampling-based mechanism achieves the best overall results.
Summary of strengths: * Important area of prompt optimization
* Interesting idea of bandit with strategies
Summary of weaknesses: Additional points that can enhance the paper are:
* The bandit algorithm assumes the reward distribution of the arms is stationary. However, during the optimization process, each strategy’s effectiveness might change. For example, shortening the prompt might have low value at the beginning but be more beneficial at the end. Thus, the application of the bandit algorithms intrinsically conflict with the prompt optimization with strategies problem.
* The overall baseline is weak as there are quite a few another prompt topimzation framework,  some using bandit algorithms as well such as EPO [1], EASE [2], APOHF[3], and others as listed in the related work section using evolution algorithms such as PromptBreeder [4], Phaseevo [5] that would form a better baseline.
* The framework has a large variance compared to Evoprompt, Can the author provide more insight on that?
* The cost analysis should be discussed with other methods.
* The influence of hyperparemters in the framework should be discssed.
* Can the author share more about how the strategies are selected?
* Is there any learning from the overall effectiveness of different strategies across different tasks?

[1] Shi et al, 2024, Efficient Prompt Optimization Through the Lens of Best Arm Identification

[2] Wu et al, 2024, Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars

[3] Lin et al, 2024, Prompt Optimization with Human Feedback

[4] Fernando et al, 2023, Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution

[5] Cui et al, 2024, PhaseEvo: Towards Unified In-Context Prompt Optimization for Large Language Models
Comments, suggestions and typos: NA
Overall assessment scores: 2.5