Large Language Models as Optimizers
Authors: Chengrun Yang
Xuezhi Wang
Yifeng Lu
Hanxiao Liu
Quoc V Le
Denny Zhou
Xinyun Chen
Venue: ICLR 2024 poster (2024)
Keywords: large language model
optimizer
prompting
Primary area: general machine learning (i.e., none of the above)
TL;DR: We propose a simple and effective approach to use large language models as optimizers, and demonstrated its capability on math and prompt optimization problems.

Abstract: Optimization is ubiquitous. While derivative-based algorithms have been powerful tools for various problems, the absence of gradient imposes challenges on many real-world applications. In this work, we propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as optimizers, where the optimization task is described in natural language. In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step. We first showcase OPRO on linear regression and traveling salesman problems, then move on to our main application in prompt optimization, where the goal is to find instructions that maximize the task accuracy. With a variety of LLMs, we demonstrate that the best prompts optimized by OPRO outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks. Code at https://github.com/google-deepmind/opro.

Published as a conference paper at ICLR 2024
LARGE LANGUAGE MODELS AS OPTIMIZERS
Yifeng Lu Quoe V. Le Denny Zhou Xinyun Chen
Chengrun Yang"
Xuezhi Wang
Hanxiao Liu
(chengrun, xuezhiw, yifenglu, hanxiaol] @google.com com (qvl, dennyzhou, xinyunchen| egoogle.com Google DeepMind
Equal contribution
ABSTRACT
Optimization ubiquitous. While derivative based algorithms have been powerful tools for various problems, the absence of gradient imposes challenges on many real-world applications. In this work, we propose Optimization by PROmpting (OPRO), simple and effective approach to leverage large language models (LLMs) as optimizers, where the optimization task described in natural language. each optimization step, the LLM generates contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step. We first showcase OPRO on linear regression and traveling salesman problems, then move on our main application in prompt optimization, where the goal is to find instructions that maximize the task accuracy. With a variety of L.LMs, we demonstrate that the best prompts optimized by OPRO outperform ousanndesigned prompts by up to 8% on GSM8K, and by to 50% on Big-Bench Hard tasks. Code at /ttps::/ggitiii com geeppieddepmmnddd...... opro
solutions from the prompt that
accuracy
training accuracy
70.0
80.0
60.0
BBH
50.0
movie recommendaiiioo
60.0
50
100
150
50
100
150
200
steps
(a) GSM8K
(b) BBH movic rocommendation
Figure 1: Prompt optimization GSM8K (Cobbe et al. 2021 and BBH (Suzgun al. 2022 movie_ree recommendation. The optimization on GSM8K has pre-trained PaIM 2-L as the scorer and the instruction-tuner PAIM 2-L (denoted PaIM 2-L-IT) the optimizer; the optimization BBH mwviie recommendation has text sext-bbise bosson as the scorer and PaLM 2 2-L-IT as the optimizer. Each dot the average accuracy across all (up 8) generated instructions in the single step, and the shaded region represents standard deviation. See Section 5 for more details on experimental setup.
Table 1: Top instructions with the highest GSM8K zero-shot test accuracies from prompt optimization with different optimizer LLMs. All results the pre-trained PaLM 2-L as the scorer.
Source
Instruction
Acc
Baselines
Kojima et al. 2022
Let's think step by step. Let's work this out in step by step to he sure we have the right answer.
71.8
Zhou et ar 20226
58.8
(empty string)
34.0
Oury
Palk 2 2-I-IT
Take deep breath and work on this problem stop-by-step.
Break this down.
79.9
1..33.5tuzil
little bit of urithmetic and logical approach will help quickly arrive
78.5
the solution to this problem.
gpt-4
Let's combine our numerical command and clear thinking to quickly and accurately decipher the answer.
74.5

---

Published as a conference paper at ICLR 2024
1 INTRODUCTION
Optimization is critical for all areas. Many optimization techniques are iterative: the optimization starts from an initial solution, then iteratively updates the solution to optimize the objective func-
tion AAmari 1993; Qian 1999| Kingma& & Ba 2015; Bäck & Schwefel, 1993; Rios & Sahinidis
2013; Reeves |993). The optimization algorithm typically needs to be customized for an individual task to deal with the specific challenges posed by the decision space and the performance landscape, especially for derivative-free optimization.
In this work, we propose Optimization by PROmpting (OPRO), simple and effective approach to utilize large language models (LLMs) as optimizers. With the advancement of prompting techniques,
LLMs have achieved impressive performance in various domains (Wei et al. 2022 Kojima et al..
2022 Wang et al.] 2022] Zhou et al. [202aa] Madaan et al. [003 Bai et al., 2022 Chen et a (023e)) Their ability understand natural language lays out new possibility for optimization: instead of formally defining the optimization problem and deriving the update step with programmed solver, we describe the optimization problem in natural language, then instruct the LLM iteratively generate new solutions based on the problem description and the previously found solutions. Optimization with LLMs enables quick adaptation different tasks by changing the problem description in the prompt, and the optimization process can be customized by adding instructions to specify the desired properties of the solutions.
To demonstrate the potential of LLMs for optimization, we first present case studies on linear regression and the traveling salesman problem, which are two classic optimization problems that underpin many others in mathematical optimization, computer science, and operations research. On small-scale optimization problems, we show that LLMs are able to find good-quality solutions simply through prompting, and sometimes match surpass hand-designed heuristic algorithms.
Next, we demonstrate the ability of LLMS to optimize prompts: the goal is to find a prompt that maximizes the task accuracy. Specifically, we focus natural language tasks where both the task input and output are texts. LLMS are shown to be sensitive to the prompt format (Zhao et al. 2021
Lu et al. 2 Wei et al. 2023 Madaan & Yazdanbakhsh 2 in particular, semantically similar
prompts may have drastically different performance (Kojima et al., [022]] Zhou et al. 2022b Zhang
al. 2023), and the optimal prompt formats can be model-specific and task-specific (Ma et al..
Chen et al. 2023c). Therefore. prompt engineering is often important for LLMs to achieve good performance (Reynolds & McDonell 2021). However. the large and discrete prompt space makes challenging for optimization, especially when only API access to the L.M is available. Following prior work on continuous and discrete prompt optimization (Lester et al. 2021 Li & Liang 2021 Zhou et al, 2022b, Pryzant et al. 2023). we assume training set is available to compute the training accuracy as the objective value for optimization, and we show in experiments that optimizing the prompt for accuracy on a small training set is sufficient to reach high performance the test set.
The prompt to the LLM serves as a call to the optimizer, and we name it the meta-prompt. Figure shows an example. The meta-prompt contains two core pieces of information. The first piece is previously generated prompts with their corresponding training accuracies. The second piece is the optimization problem description, which includes several exemplars randomly selected from the training set to exemplify the task of interest. We also provide instructions for the LLM to understand the relationships among different parts and the desired output format. Different from recent work on using LLMs for automatic prompt generation (Zhou et al.) 2022b Pryzant et al., 2023), each optimization step in our work generates new prompts that aim to increase the test accuracy based on a trajectory of previously generated prompts, instead of editing one input prompt according to natural language feedback (Pryzant et al.) 2023) or requiring the new prompt to follow the same semantic meaning (Zhou et al. (022b). Making use of the full optimization trajectory, OPRO enables the I.LM to gradually generate new prompts that improve the task accuracy throughout the optimization process, where the initial prompts have low task accuracies.
We conduct comprehensive evaluation on several LLMs, including bext-bison and Palm 2-L in the PaLM-2 model family (Anil et al. 2023), as well as gpt 3. S-turbo and gpt in the GPT model family. We optimize prompts on GSM8K (Cobbe et al. [021] and
Hard (Suzgun et al. 2022). which are reasoning benchmarks where prompting techniques have achieved remarkable performance breakthrough (Wei et al.) 2022 Kojima et al. 2022 Suzgun et al., 2022). Starting from initial prompts with low task accuracies, we show that all L.Ms in evaluation are able to
2

---

Published as a conference paper at ICLR 2024
objective function evaluator
scores
generated solutions
return top solutions
finish
meta-prompt
L.MM optimizer
solution-score pairs
task description
Figure 2: An overview of the OPRO framework. Given the meta-prompt as the input, the L.LM generates new solutions to the objective function, then the new solutions and their scores are added into the meta-prompt for the next optimization step. The meta-prompt contains the solution-sca score pairs obtained throughout optimization, a natural language description of the task, and (in prompt optimization) a few task exemplars. Figure shows a sample meta meta-prompt for prompt optimization.
serve as optimizers, which consistently improve the performance of the generated prompts through iterative optimization until convergence (see Figure TT. In particular, while these LLMS generally produce instructions of different styles (see Table D with zero-shot prompting, their best generated instructions match the few-shot shain-of-thought prompting performance when applied 2-L, outperforming the zero-shot performance with human-designed prompts by up to 8% on GSM8K. Additionally, we observe that the OPRO-optimized prompts transfer other benchmarks of the same domain and also deliver notable performance gain.
PaLM
OPRO: LLM AS THE OPTIMIZER
Figure illustrates the overall framework of OPRO. In each optimization step, the LLM generates candidate solutions to the optimization task based on the optimization problem description and previously evaluated solutions the meta-prompt. Then the new solutions are evaluated and added to the meta-prompt for the subsequent optimization process. The optimization process terminates when the LLM is unable to propose new solutions with better optimization scores, or a maximum number of optimization steps has reached. We first outline the desired features of LLMs for optimization, then describe the key design choices based on these desirables.
2.1 DESIRABLES OPTIMIZATION BY L.MSS
Making use of natural language descriptions. The main advantage of LLMs for optimization is their ability of understanding natural language, which allows people to describe their optimization tasks without formal specifications. For instance, in prompt optimization where the goal is to find prompt that optimizes the task accuracy, the task can be described with a high-level text summary along with input-output examples.
Trading off exploration and exploitation. The exploration-expllitttio trade-off is a fundamental challenge in optimization, and it important for L.MMs serving as optimizers to balance these two competing goals. This means that the L.M should be able to exploit promising areas of the search space where good solutions are already found, while also exploring new regions the search space so as to not miss potentially better solutions.
2.2 META-PROMPT DESIGN
As the input to the optimizer LLM, the meta-prompt contains the following essential parts.
Optimization problem description. The first part is the text description the optimization problem, including the objective function and solution constraints. For example, for prompt optimization, the LLM can be instructed to "generate a new instruction that achieves a higher accuracy". and we denote such instructions in the meta-prompt inetainnttuuctions. We also provide customized
3

---

Published as a conference paper at ICLR 2024
instainnttuuctios an informal regularization of the generated solutions, such as "the instruction should be concise and generally applicable".
Optimization trajectory. Besides understanding natural language instructions, LLMs are also shown to be able to recognize patterns from in-context demonstrations (Wei et al. 2023 Madaan & Yazdanbakhsh 2022 Mirchandani al. 2023). Our meta- reta-prompt makes of this property and in- structs the LLM to leverage the optimization trajectory for generating new solutions. Specifically, the optimization trajectory includes past solutions and their optimization scores, sorted in the ascending order. Including optimization trajectory in the meta-prompi allows the LLM to identify similarities of solutions with high scores, encouraging the LLM to build upon existing good solutions construct potentially better ones without the need of explicitly defining how the solution should be updated.
2.3 SOLUTION GENERATION
At the solution generation step, the LLM generates new solutions with the meta-prompt as input. The following are the key optimization challenges we address in this stage.
Optimization stability. In the optimization process, not all solutions achieve high scores and monotonically improve over prior ones. Due the sensitivity of in-context learning the prompt. LLM output can be drastically affected by low-quality solutions in the input optimization trajectory. especially at the beginning when the solution space has not been adequately explored. This sometimes results in optimization instability and large variance. To improve stability, we prompt the LLM to generate multiple solutions at each optimization step, allowing the LLM to simultancously explore multiple possibilities and quickly discover promising directions to move forward.
pnploraatio-epploaattiin trade-off. We tune the LLM sampling temperature to balance between exploration and exploitation. A lower temperature encourages the LLM to exploit the solution space around the previously found solutions and make small adaptations, while a high temperature allows the LLM to more aggressively explore solutions that can be notably different.
3 MOTIVATING EXAMPLE: MATHEMATICAL OPTIMIZATION
We first demonstrate the potential of LLMs serving as optimizers for mathematical optimization. We present a case study linear regression
example of continuous optimization, and another case study on the Traveling Salesman Problem (TSP) (Jünger et al. 1995; Gutin Punnen (0066) as an example of discrete optimization. We present the numerical results in Table and 3 and deter their experiment setups to Appendix B On both tasks, we see LLMs properly capture the optimization
directions small-scale problems merely based the past optimization trajectory provided in the meta-prompt. We also see limitations and failure cases of using LLMs as optimizers, and we defer the discussion Appendix C
Table 2: Linear regression by optimizer LLMs: the mean = standard deviation of the number of steps and the number of unique (w, b) pairs explored before reaching the global optima. Both w and start from 5 random starting points in [10, 20). We use temperature 1.0 for all models. We each setting times. The starting points are the same across optimizer LLMs but are different across 5 runs, and grouped by: within the starting region, outside and close to the starting region, and outside and farther from the starting region. Bold numbers indicate the best among three L.LMs in each setting. See Appendix B. for experiment setup.
number of steps
number of unique (w, 6) pairs explored
byne
text-bison
qpt-3. 5-turbe
gpt-4
text-bison cpt-3.5-tur
qpt
15
5.8 1266
7.6 145
4.0 1.
40.0 124
152
17.2 5.1.1
17
4.0
12.6
6.0
33.4
$3.8 169
26.0 10.6
16
10
3.8
10.4
6.2
30.2 1::4
42.8
24.2 1:22
5
-
9.8 28
10.8 1:27
12.2
35.8 16.1
39.6
33.0 1:4.0
25
19.6 11.4
26.4
12.2=27
104.0
78.6
44.2 +.3
2
30
31.4 263
4..
38.0
126.4
125.6
99.0
36
-1
35.8 6.6
45.4 = 169
50.4 + 18.8
174.0 + 282
142.2
116.4 = 12.7
4

---

Published as a conference paper at ICLR 2024
Table 3: Results of the Traveling Salesman Problem (TSP) with different number of nodes n. where each n contains 5 problems. "# steps" calculates the mean + standard eтrоr of optimization steps for successful runs that find the optimal solution. "# successes" counts the number of problems that OPRO results in the optimal solution. When no optimal solution is found for any evaluated problem, Hapendix[B.2feee
the corresponding number steps is N/A. See Appendix B.2 for experiment setup.
optimality gap (%)
steps
NN
F1
text -bainblson
qpc S-tcbo
qpt
Lext -blaon
qpt 3.5 S.rbb
qpt-4
0.0
(5)
15
9.4
12
1.2 L.6
4,4.1 1:3
1.2 111
0.2 L02
N/A (0)
202.0 411 (4)
58.5 2.0 (4)
195.5
30
+31
+31
219.8 137
133.0 #68
11.0 #6
N/A (0)
N/A (0)
N/A (0)
have some texts along with their corresponding scores. The texts are arranged ascending order based on their scores, where higher scores indicate better quality.
text:
Let's solve the problem.
score:
63
(... more instructions and scores ...)
The following exemplars show how to apply your text: you replace <INS> in each input with your text, then read the input and give an output. We say your output wrong if your output is different from the given output, and we say your output is correct if they are the same.
input:
Q: Alannah, Beatrix, and Queen are preparing the new school year and have been given books by their parents. Alannah has 20 more books than Beatrix. Queen has 1/5 times more books than Alannah. If Beatrix has 30 books, how many books do the three have together?
A: <INS>
output:
140
(... more exemplars...
Write your new text that is different from the old ones and has score as high as possible. Write the text in square brackets.
Figure 3: An example of the meta- meta-prompt for prompt optimization with instruction-tume PaLM 2-L (PaLM 2-I- IT) on GSM8K, where the generated instruction will be prepended to the beginning of "A:" in the scorer L.LM output (A M__begin in Section <INS> denotes the position where the generated instruction will be added. The blue text contains solution-scom pairs; the purple text describes the optimization task and output format; the orange text are meta-instructions. Appendix E.2 presents the full meta-prompts for other optimizer LLMs.
APPLICATION: PROMPT OPTIMIZATION
Next, we demonstrate the effectiveness of OPRO on prompt optimization, where the objective is to find the prompt that maximizes task accuracy.
We focus on prompt optimization for natural language tasks, where both the input and output are in the text format. The task is represented as dataset with training and test splits, where the training set is used to calculate the training accuracy the objective value during the optimization process, and we compute the test accuracy on the test set after the optimization finishes. While traditional optimization often requires decently large training set, experiment shows that small number fraction of training samples (e.g.. 3.5% of the training set for GSMSK, 20% for Big-B Big-Bench Hard) is sufficient. The objective function evaluator LLM to which the optimized prompt will be applied,
5

---

Published as a conference paper at ICLR 2024
and it can be the same different from the LLM for optimization. We denote the LLM for objective function evaluation as the scorer LLM, and the LLM for optimization as the optimizer LLM.
The output of the optimizer LLM is instruction, which is concatenated to the question part of every exemplar and prompts the scorer LLM. We consider the following positions to insert the instruction: Q b_begin (the instruction is added before the original question), Q_end (the instruction is added after the original question), and A __egin (the instruction is added to the beginning of the scorer LLM output. This is applicable to pretrained LLMs without instruction tuning, where the prompt is formatted as a sequence of QA pairs). We show examples of these prompting formats in Appendix D
Figure 3 shows example of the meta-prompt for prompt optimization on GSM8K (Cobbe et al. 2021p. The meta-prompt includes three parts: task examples few exemplars from the training set to show how the task looks like and where the generated instructions will be added to: optimization trajectory the instructions generated in previous optimization steps, along with their scores (only showing the top ones ascending order); meta-instructions the instructions to the optimizer LLM that explain the optimization goal and instruct the model how to use the above information.
5 PROMPT OPTIMIZATION EXPERIMENTS
We present prompt optimization results in this section. Our experiments demonstrate that OPRO brings a significant performance gain across the board, with different optimizer and scorer LLMs.
5.1 EVALUATION SETUP
Models. The LLMs we use optimizer are pre-trained PaLM 2-L, instruction-tuned PaLM 2-L (denoted PaLM 2-I-IT), text-bis bion, gpt-3. 3.turbb,,, and gpt-4; the LLMs we use as scorer are pre-trained PaLM 2-L and text bisson We search for A_ begin instructions on pre-trained PaLM 2-L, and for Q_beiin and e_d instructions on mexx-bisoon.
Benchmarks. Our primary evaluation benchmarks for prompt optimization are GSM8K (Cobbe ct al. 202I) and Big-Bench Hard (BBH) (Suzgun et al., 2022). To examine the transferability of the optimized instructions, also evaluate the instructions optimized for GSM8K on two other mathematical reasoning datasets, i... MultiArith (Roy & Roth 2016) and AQuA (Ling et al. 2017).
Implementation details. We use temperature 0 (greedy decoding) to evaluate generated instructions. We set the default temperature to 1.0 for optimizer LLMs to generate diverse instructions. each step, prompt the optimizer LLM with the peea--pompt 8 times to generate 8 instructions. Our meta- prompt contains the best 20 instructions so far and 3 randomly picked exemplars from the training set. The exemplars are independently sampled at each step to reduce the risk of overfitting on the given exemplars. We study the effect of different hyperparameters in ablation studies (Section 5.3).
5.2
MAIN RESULTS
Here we show prompt optimization curves on GSM8K and two BBH tasks. The curves on other BBH tasks are deferred to Appendix and the tables containing all accuracy numbers in Appendix J Appendix D analyzes overfitting in prompt optimization. Appendix M compares the prompt optimization performance of meta meta-prompi prompts in OPRO and EvoPrompt (Guo et al. 2023).
5.2.1 GSM8K
For GSM8K, we randomly sample 3.5% examples from the training set, and the same subset used throughout optimization. This balances evaluation cost with generalization performance. After the optimization finishes, we evaluate the found instructions on the entire GSM8K test set.
Figure ((a) in Section shows the prompt optimization curve with pre-trained PaLM 2-L scorer and PALM 2-L-IT as optimizer, and the initial instruction is "Let's solve the problem" " with a (approximated, and same below) training accuracy 60.5. The curve has overall upward trend with several leaps throughout the optimization process. The curve also shows a decrease of the variance among the accuracies of instructions generated at each step, indicating the optimizer LLM generates

---

Published as a conference paper at ICLR 2024
accuracy difference
accuracy difference
60
40
40
20
0-
20
-20
(a) minus "Let's think step by step."
(b) minus empty starting point
Figure 4: On 23 BBH tasks, the accuracy differences among instructions found by prompt optimiza- tion (with the PaLM 2-L scorer and the PaLM 2-L-IT optimizer), "Let's think step by step.", and the empty string (optimization starting point). The bar charts with task names and those with the text-bison scorer are deferred to Figure T9 in Appendix J.1
distributionally better instructions throughout optimization. Appendix shows similar trends when using text -bison scorer and when using pre-trained PaLM L as optimizer.
Table D summarizes top instructions. We see the styles of instructions found by different optimizer LLMs vary lot: PaLM 2-L-IT and text-bise ones concise; GPT ones are long and detailed. And although some top instructions contain the "step-by-step" phrase, most others achieve comparable accuracy with different semantic meanings.
5.2.2 BBH
On BBH, the optimization starts from empty string the initial instruction by default. For each task, we utilize a subset of 20% examples for prompt optimization, and the rest examples are for testing. As shown in Figure [(b) we also observe upward trend in the optimization
BBH. See Figure 17 and T8 in Appendix|] for curves on all BBH tasks from the text-bi son scorer and the PaLM 2-L optimizer. We show experimental results on different scorer and optimizer LLMs, instruction positions, and starting points in Appendix !T]
Figure visualizes the per-task accuracy difference on all 23 BBH tasks compared to the instruction "Let's 'ss think step by step." (Kojima et al. 2022) and the empty instruction, and we detail the accuracy values To Appendix[] JJ We show that the instructions found by OPRO outperform "Let's think step by step." on almost all tasks by large margin: with the PaLM 2-L scorer, our instructions outperform by over on 19/23 tasks. Our prompt optimization algorithm also improves instructions from the empty starting point by over 5% on most tasks: 20/23 with the Pa LM 2-I scorer.
5.2.3 TRANSFERABILITY OF FOUND INSTRUCTIONS
We assess the transferability of found prompts to different datasets in the same domain, where evaluate the top instructions found for GSMSK on MultiArith and AQuA. Table our optimized prompts also outperform baseline prompts with different scorer LLMs these two benchmarks.
5.3 ABLATION STUDIES
We use text-bise -oxt-bbie son as scorer and PaLM 2-L as optimizer for all ablation studies. The tasks we evaluate are GSM8K (math reasoning) and BBH sports_ uooorssundest (non-math reasoning).
Meta-prompt design: instruction ordering. The meta-prompt design is crucial in achieving good prompt optimization performance. Here we investigate the order of the previous instructions. We compare the following: (1) lowest to highest (default setting); (2) highest lowest; (3) random. Figures ((a) and ((b) show that the default setting achieves better final accuracies and converges faster. One hypothesis is that the optimizer LLM output is affected more by the past instructions closer to the end of the meta-prompt This is consistent with the recency bias observed in Zhao et al.) (2021). which states that LLMs more likely to generate tokens similar to the end of the prompt.
The number of generated instructions per step. Computing a mini-batch of gradients reduces the variance of stochastic gradient descent procedure. Similarly, generating multiple instructions in each
7

---

Published as a conference paper at ICLR 2024
Table 4: Transferability across datasets: accuracies of top instructions found for GSM8K on Multi- Arith and AQuA.
Instruction position
Accuracy
Scorer
Instruction
MultiArith
AQUA
Baselines
PaLM 2-L
Kojima et al.
A_ A_giii
Let's think step by step.
85.7
44.9
2022)
PaLM 2-1
Ziou et al.
A begin
Let's work this out in step by step to be
72.8
48.4
X022b
sure we have the right answer.
PaLM 2-b
A_ begin
Let's solve the problem.
87.5
44.1
PaLM 2-L
A_ begin
(empty string)
69.3
37.8
text-biscn
Kojima et al. 2022
Q begin
Let's think step by step.
92.5
31.9
LexL-bisen
Ziou
begiin
Let's work this in step by step way be
32.3
2022b
sure we have the right answer.
text-bisen
Q begin
Let's solve problem. (emply string)
85.5
29.9
text-bison
Q Q_begin
82.2
33.5
Ours
PalMM 2-1.
PalM 2-1-TT
A begin
Take a deep hreath and work on this problem
95.3
54.3
on GSM8K
step-by-step.
text-bisen
PaLM 2-L-IT on GSM8K
begin
Let's work together to solve math word prohlems! First, we will read and discuss the problem together make
96.8
37,8
undcrstand Then, will work together find the solution. will give you hints and help you work through the problem you get stuck.
accuracy
70
accuracy
100
accuracy
70
50 accurad
100
60
60
50
100 # steps
200
100 # steps
200
50 800
1600
0 0 800
1600
evaluated instructions
evaluated instructions
ascending (default)
ascending (default)
1
(default)
8 (default)
descending
descending
random
random
(a) instruction ordering (GSM8K)
(b) instruction ordering sgports_understanding)
(c) # single-step instruc- (d) # single-step instruc- tions (GSM8K)
tions (sports_ understanding)
Figure 5: Ablation studies. The dots are the average values across 3 optimization repetitions, and shaded regions represent standard deviations. In Figure and (d) the x-axis represents the total number of evaluated instructions through the optimization; e.g., we run 200 optimization steps when sampling 8 instructions each step, run 400 steps when sampling 4 instructions in each step, etc.
step improves the optimization stability with LLMs. On the other hand, to achieve better performance with fixed budget for the number of instructions to evaluate, the number of per-step instructions should not be too large, so as to allow more optimization steps to incorporate richer information of past instructions with their accuracies. Taking both aspects into consideration, Figure [(c] and (5d) compares the optimization performance of sampling 1 2 / 4 8 (default) 16 instructions in each step, showing that sampling 8 instructions each step overall achieves the best performance.
We show ablation studies of other algorithmic aspects in Appendix K
6 RELATED WORK
Prompt optimization. Prior works have developed soft prompt-tuning methods that optimize the prompt represented task-specific continuous vectors (Lester et al. 2021 [i Liang 2021
Liu et al.
8

---

Published as a conference paper at ICLR 2024
2021 Qin & Eisner 2021), ), as well as performing discrete prompt optimization by cradient-guidee
search (Shin et al.] 2020 Wen et al. [023] Gao et al., 2020 Chen et al. (023d) and reinforcement
learning (Deng al. [022 Zhang et al. 2023). These approaches become inapplicable when there is only API access to the LLM. Other works designed edit-based approaches for gradient-free prompt optimization (Xu al. 2022 Prasad et al. 2022), where the editing can be done with human- defined operations (c.g., swapping two phrases) (Prasad et al.) (022)) or language models (c.g. back translation) (Xu et al. 2022). Some recent works investigate LLMs for prompt optimization (Zhou et al. 2022b; Pryzant et al. 2023) Xu et al. (023). Specifically, APE (Zhou et al. (022b) first uses the LLM to generate initial instructions. Afterwards, APE selects top instructions with the highest accuracies, then prompts the LLM with each individual instruction to generate semantically similar variant of the initial instruction. APO (Pryzant et al. 2023) in each step instructs the LLM to produce text feedback on how to update an old instruction. Different from edit-based approaches, the optimizer LLM in our work directly generates new instructions at each optimization step, and the optimizer LLM is merely asked to improve the task accuracy without being required to imitate past instructions. Compared to Zhou et al. (202bb and Pryzant et al. (2023), our optimization process incorporates the past generated instructions with their scores in the meta-prompt, enabling the optimizer LLM to discover common patterns of high-quality instructions.
Tuning language models for optimization. Some previous works tune or prompt language models to behave as mutation and crossover operators in evolutionary algorithms. |Meyerson et al. (2023) uses language models with few-shot exemplars to propose evolutionary cross-overs on tasks like image and code generation. In Lehman et al. (2022), the LLM trained on code diff' generation is used the mutation operator, and they further design fine-tuning method to improve performance in the Sodarace domain for robot simulation. EvoPrompting (Chen et al. 2023a) uses LLMs to evolve neural network architectures, where they combine evolution with soft prompt tuning. With respect to taking the trajectory as the input for optimization, OptFormer (Chen et al. (022) trains transformer model on large collections of hyperparameter optimization data. On the other hand, our work performs optimization solely by prompting without additional training.
7 CONCLUSION
We embark on employing LLMs as optimizers, where the LLM progressively generates new solutions to optimize an objective function. We first motivate OPRO with linear regression and traveling salesman problems. then proceed to prompt optimization concrete application. Our evaluation demonstrates that LLMs have the capacity of gradually improving the generated solutions based on the past optimization trajectory. Interestingly, performs on par with some hand-crafted heuristics. For prompt optimization, optimized prompts outperform human-designed
small-scale traveling salesman problems, OPRO
GSM8K and BBH by a significant margin, sometimes over 50%.
A number of unresolved questions are open for future research on LLMs for optimization. In general, how to reduce the sensitivity initialization and better balance exploitation with exploration remains a challenge. Specifically, for prompt optimization, one limitation of our current implementation is that the optimizer LLM does not effectively utilize error cases in the training to infer promising directions to improve the generated instructions. In our experiments, we tried including error cases in the meta- peeta-prompt rather than randomly sampling from the training set each optimization step, but the results are similar, indicating that the error cases alone are not informative enough for the optimizer LLM to grasp the cause of the wrong prediction. Another limitation that prompt optimization requires training set to compute the accuracy that guides the optimization process. Currently the training set at least contains tens of samples, so that the optimized prompt does not severely overfit to the training samples. A promising direction is to incorporate richer feedback about the error cases besides the aggregated accuracy, and summarize the key features that distinguish between high-quality and low-quality generated prompts in the optimization trajectory. Such information may inform the optimizer LLM of how to more efficiently improve over the past generated instructions, and potentially further reduce the example set size needed for prompt optimization.
9

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This work uses synthetic math problems for linear regression and traveling salesman problems, and public datasets like GSM8K and Big-Bence Hard for prompt optimization. These tasks have been commonly used similar works and should not be regarded controversial. There is peril that LLMs may generate harmful information that poses safety risks; how safeguard model behavior remains valuable future work.
REPRODUCIBILITY STATEMENT
We evaluate on public benchmarks. The text-bison API is available at: https://clour www/veeeeceeoc otw.e-aaa///cccceeeeeaateeee-e-- saarnmmoeelss The GPT models are available here: http:///openai.ccom/aa This work uses upt-3.5-turbo-08 and gpt-4-0613.
ACKNOWLEDGMENTS
We thank Daiyi Peng, Yanqi Zhou, Jemy Wei, Shuo Chen, Tim Rocktäschel, Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Ed H. Chi for their valuable feedback, and thank several anonymous reviewers for helpful comments.
REFERENCES
Shun-ichi Amari. Backpropagation and stochastic gradient descent method. Neurocomputing, 5(4-5): 185-196, 1993.
Rohan Anil, Andrew Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint (rXii:2305.10400 2023.
David Applegate, Ribert Bixby, Vasek Chvatal, and William Cook. Concorde tsp solver, 2006.
Thomas Bäck and Hans-Paul Schwefel. An overview of evolutionary algorithms for parameter optimization. Evolutionary computation, 1(1):1-23, 1993.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional Harmlessness from ai feedback. arXiv preprint arXiv:22/2..08073 2022.
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXiv: 17122 2023.
Angelica Chen, David M Dohan, and David R So. Evoprompting: Language models for code-level neural architecture search. arXiv preprint arXiv:2302. 14838, 2023a.
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303.167499 2023b.
Jiuhai Chen, Lichang Chen, Heng Huang, and Tianyi Zhou. When do you need chain-of-thought prompting for chatgpt? arXiv preprint arXiv:2304.03262. 2023c.
Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03082, 2023d.
Xinyun Chen and Yuandong Tian. Learning perform local rewriting for combinatorial optimization. Advances in Neural Information Processing Systems, 32, 2019.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint 2304002304 2023e.
10

---

Published as a conference paper at ICLR 2024
Yutian Chen, Xingyou Song. Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Mare aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Process- ing Systems, 35:32053-32068, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, al. Training verifiers to solve math word problems. arXiv preprint vrXiv:2110.44166 2021.
Mingkai Deng, Jianyu Wang. Cheng theng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song. Eric P Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205. 22548, 2022.
Michel Deudon, Pierre Cournut, Alexandre Lacoste, Yossiri Adulyasak, and Louis-Martin Rousseau. Learning heuristies for the tsp by policy gradient. In International Conference the Integration of Constraint Programming. Artificial Intelligence, and Operations Research, pp. 170-181. Springer, 2018.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rock- täschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprini arXiv:2309. 16797, 2023.
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilè Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint XXii:23322.07599 2023.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012. 15723, 2020.
Bruce Golden, Lawrence Bodin, T Doyle, and W Stewart Jr. Approximate traveling salesman algorithms. Operations research, 23(3-partiixxx94-711 1980.
Qingyan Guo, Rui Wang. Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint 27Xiv22309008322 2023.
Gregory Gutin and Abraham P Punnen. The traveling salesman problem and variations, volume 12.
Springer Science & Business Media, 2006.
Keld Helsgaun. An extension of the fin-kernighan-helsgaaee tsp solver for constrained traveling salesman and vehicle routing problems. Roskilde: Roskilde University, 12, 2017.
Michael Jünger, Gerhard Reinelt, and Giovanni Rinaldi. The traveling salesman problem. Handbooks in operations research and management science, 7:225-330, 1995.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint arXiv:2303. 17491, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint / /9/6, 2022.
Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In International Conference on Learning Representations, 2019. URL htttttss///pperreiiiee
net/ fie/ffozee fttu//orummiddd Bett/Eorum:.ddByxBBB-eee ByxBBFxBFFyynn
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O Stanley, Evolution through large models. arXiv preprint arXiv:2206. 08896, 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter- prompt tuning. arXiv preprint arXiv:2 20222144008000 2021.
11

---

Published as a conference paper at ICLR 2024
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00193 2021.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. arXiv preprint arXiv: 1705.0015444 2017.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103. 2021.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity, arXiv preprint arXiv:2104.08786 2021.
Xiao Ma, Swaroop Mishra, Ahmad Beirami, Alex Beutel, and Jilin Chen. Let's do a thought experiment: Using counterfactuals improve moral reasoning. arXiv preprint arXiv:2306. 14308, 2023.
Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.06686 2022.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprini 1771
Elliot Meyerson, Mark J Nelson, Herbie Bradley, Arash Moradi, Amy K Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting. arXiv preprint arXiv:2302. 12170, 2023.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models general pattern machines. arXiv preprint 0rXiv:2307.04721, 2023.
Varun Nair, Elliot Schumacher, Geoffrey Tso, and Anitha Kannan. Dera: Enhancing large language model completions with dialog-enabled resolving agents. arXiv preprint arXiv:2303. /7071, 2023.
RehammadReza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takac. Reinforcement learning for solving the vehicle routing problem. In Advances in Neural Information Processing Systems, pp. 9861-9871, 2018.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306. 09896, 2023.
Gurobi Optimization al. Gurobi optimizer reference manual, 2020.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.07281. 2022.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint X3x.:3305.03495 2023.
Ning Qian, On the momentum term in gradient descent learning algorithms. Neural networks, 12(1): 145-151, 1999.
Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. arXiv preprint arXiv:2 104.06599, 2021.
Colin R Reeves. Modern heuristic techniques for combinatoria problems. John Wiley & Sons, Inc., 1993.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-7, 2021.
12

---

Published as a conference paper at ICLR 2024
Luis Miguel Rios and Nikolaos V Sahinidis. Derivative-free optimization: review of algorithms and comparison of software implementations. Journal of Global Optimization,
2013.
Daniel Rosenkrantz, Richard E Stearns, and Philip M Lewis, II. An analysis of several heuristics for the traveling salesman problem. SIAM journal computing, 6(3):563-581, 1977.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv preprint arXiv: 01413, 2016.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint 231.::3322004761,, 2023.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15981 2020.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dynamic memory and self-reilection. arXiv preprint arXiv:2303. 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them, arXiv preprint arXiv:2210.09261, 2022.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang. Aakanksha Chowdh- ery, and Denny Zhou. Seff-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11111 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting clicits reasoning in large language models. arXiv preprint 2rXiv:2201.11903 2022.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently, arXiv preprint arXiv:2303.03844
2023.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302.0366 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 10044.1224:4 12244, 2023.
Hanwei Xu. Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang. Haiyu Li, and Zhilin Yang. Gps: Genetic prompt search for efficient few-shot learning. arXiv preprint 10Xiv:2210.11044 2022.
Weizhe Yuan, Kyunghyun Cho, and Jason Weston. System-level natural language feedback. arXiv preprint arXiv:2306. 220.3306..3330 3388,, 2023.
Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. Tempera: Test-time prompt editing via reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before Improving few-shot performance of language models. In International Conference on Machine Learning, pp- 12697-12706. PMLR, 2021.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 10625, 2022a.
13

---

Published as a conference paper at ICLR 2024
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022b.
14

---

Published as a conference paper at ICLR 2024
A RELATED WORK ON PROMPTING WITH NATURAL LANGUAGE FEEDBACK
A recent line of work investigates approaches to improve the LLM performance by prompting with natural language feedback to revise the model output, which has shown effectiveness in reducing
harmful LLM outputs (Bai et al., 2022 Ganguli et al. 2023). improving reasoning (Shinn et al.
2023
Madaan et al. 2023) and code generation performance (Chen et al. 2023e Olausson et al.
2023:
Shinn et al. 2023 Chen al. (2023b), dialogue applications (Nair et al. 2023 Madaan et al.
2023
Yuan ct al. 2023 , and so on (Kim et al., 2023; Wang et al.] 2023). Specifically, Yuan et al.
(2023)
develops a human-in-the-loop framework for deriving system-level feedback from collection
of instance-level feedback, which is then used for refining data. In our work, the optimizer L.M utilizes the optimization trajectory in the prompt, which implicitly requires the LLM to summarize the common characteristics among solutions with similar scores. We consider incorporating explicit natural language feedback on generated solutions for later optimization steps as future work.
B EXPERIMENT SETUPS OF MOTIVATING EXAMPLES ON MATHEMATICAL OPTIMIZATION
B.1 LINEAR REGRESSION
In linear regression problems, the goal is to find the linear coefficients that probabilistically best explain the response from the input variables. We study the setting in which the independent and dependent variables X and y are both one-dimensional and an intercept is present, so that there are two one-dimensional variables w, b to optimize over. In a synthetic setting, we sample ground truth values for one-dimensional variables Wive and byses and generate 50 data points by Wane + buse + c, in which ranges from 1 to 50 and is the standard Gaussian noise. Our optimization starts from randomly sampled (w, b) pairs. In each step, we prompt an instruction- tuned LLM with meta-prompt that includes the best 20 (w: b) pairs in history and their sorted objective values. The meta peta-proopp then asks for a new (w, b) pair that further decreases the objective value. A sample meta- prompt is shown in Figure 12 of Appendix E.I We prompt the meta-prompt 8 times to generate at most 8 new (w, b) pairs in each step to improve optimization stability. Then we evaluate the objective value of the proposed pair and add it to history. We do black-box optimization: the analytic form does not appear in the meta-promps text. This is because the LLM can often calculate the solution directly from the analytic form.
Table in main paper Section 3 summarizes the results with one of the following optimizer LLMs: bison, gpt-3. 5.55tbrr and gpt-4. We study three settings of Wime and bine: within the starting region [10, 20] x [10, 20). "near outside" (each of Wiue and buue is outside the starting region but the distance is less than 10), and "far outside" (each of Wrue and bnue outside the starting region and the distance is greater than 10). We see:
The number of unique (w, b) pairs explored by each model is fewer than exhaustive search, indicating these models are able to to do black-box optimization: compare the numbers and propose descent direction.
The text -bi -bison and gpt-4 models outperform gpt- -3 3..5 1.-55-turroo in convergence speed: they arrive at the optima with fewer steps. The gpt-4 model also outperforms in finding the optima with fewer explored unique points. Taking a closer look at the optimization trajectory, we see 9pt-4 is the best at proposing reasonable next step from the history: for example, when
the history shows the objective values of (w. b) = (8. 7), (w, b) (8, 6). and (w, b) 5) are decreasing, it has highest chance to propose (w, b) = (8, 4) for evaluation.
The problem becomes harder for all models when the ground truth moves farther from the starting region: all models need more explorations and more steps.
B.2 TRAVELING SALESMAN PROBLEM (TSP)
TSP is a classical combinatorial optimization problem with numerous algorithms proposed in lit- erature, including heuristic algorithms and solvers (Rosenkrantz et al.) 1977 Golden et al.) 1980 Optimization et al. 2020 Applegate et al. [2066] Helsgaum 2017). and approaches based on traiiigg
deep neural networks (Kool et al. 2019 Deudon al. 2018 Chen & Tian 2019 Nazari al. 2018
15

---

Published as a conference paper at ICLR 2024
Specifically, given set of n nodes with their coordinates, the TSP task is to find the shortest route that traverses all nodes from the starting node and finally returns to the starting node.
Our optimization process with LLMs starts from 5 randomly generated solutions, and each optimiza- tion step produces at most new solutions. We present the meta-prompt in Figure of Appendix E. We generate the problem instances by sampling nodes with both x: and y coordinates in [ 100, 100]- We use the Gurobi solver (Optimization et al. 2020) to construct oracle solutions and compute the optimality gap for all approaches, where the optimality gap is defined as the difference between the distance in the solution constructed by the evaluated approach and the distance achieved by the oracle solution, divided by the distance of the oracle solution. Besides evaluating OPRO with different LLMs including t text sutt-bison 3 5-tubbo and gpt we also compare OPRO to the Nearest Neighbor (NN) and Farthest Insertion (FI) heuristics.
We present the results in main paper Section 3 Table We randomly generate 5 problem instances for each number of nodes n. addition to measuring the optimality gap, on problems where the LLM finds the optimal solutions, we also show the number of optimization steps taken to reach the global optimum. First, we observe that gpt-4 4 significantly outperforms gpt-3.5 5-turbo and text-bi sen across all problem sizes. Specifically, on smaller-scale problems, gpt-4 reaches the global optimum about 4x faster than other LLMs. On larger-scale problems, especially with 50, gpt-4 still finds solutions with comparable quality to heuristic algorithms, while both text-bis son and -.5 -urbbe get stuck at local optima with up to 20x worse optimality gaps.
On the other hand, the performance of OPRO degrades dramatically on problems with larger sizes. When 10. all LLMs find the optimal solutions for every problem; as the problem size gets larger, the OPRO optimality gaps increase quickly, and the Fl heuristic starts to outperform all LLMs.
C SOME FAILURE CASES
Although LLMs show the power of optimizing basic math problems (Section 3) and prompts (Sec- 4i. we see some limitations across all optimizer LLMs that may impede their power solving more challenging problems. These limitations include:
Hallucinating the values that need to come from math calculation: The optimizer LLMs often output contents like "the function value at (5, 3) is 15" despite that the true value is not 15. The model will get it right external tools that can reliably calculate the value are triggered. When and how to trigger such tool use cases remains an interesting topic (see e.g.. (Schick et al. 2023
Cai et al. 2023)).
Generating solutions already appeared in context even if we tell "Give me new (w, b) pair that is different from all pairs above": the optimizer LLMs do not 100% reliably follow this instruction even if its own outputs often include sentences like "I will provide a new pair that is different", making the output celf-cottradccooy. The output is almost guaranteed to be different from in-context old solutions when the model output contains a comparison of the new pair and all old pairs, though. Thus (implicitly) triggering such behaviors may be solution. How to implement this feature without harming the instruction following performance of other parts remains interesting topic to study.
In black-box math optimization, getting stuck at a point that is neither global nor local optimal: This often occurs in two linear regression cases: (a) The in-context exemplars all share the same w or that is different from Wine or bmse. This case is more likely to be avoided when larger number of past solutions are included in the meta-prompt; (b) one or several of the best previous solutions in the meta-prome have ws and bs in quantitatively opposite directions from the global optima Winee and b for example, the ws are all smaller than Wiuee while the bs are all larger than brne- Since the optimizer model often proposes to only increase w or decrease b when the past solutions in meta-promp share w or b, the optimization will get stuck if either increasing or decreasing b would increase the objective value. This issue is mitigated by sampling multiple new solutions (thus more exploration) at each step.
Hard to navigate bumpy loss landscape: Like other optimizers, it is harder for the optimizer
LLM to optimize black-box functions when the loss landscape gets more complicated. For
example, when minimizing the Rosenbrock function f(x v) (a-x)² +b(y+bby-- with 20
16

---

Published as a conference paper at ICLR 2024
(whose global optimal point is 20, y 400) with 5 starting points in [10, 20] x [10. 20). the optimization often gets stuck at around (0. 0). This is because the optimizer LLM sees decrease of objective value when it drastically decreases both and y to 0. Then starting from (0, 0), the optimizer L.M hard to further navigate a and y along the narrow valley the loss landscape towards (20, 400) (Figure 6).
150000
100000
50000
0
0 100
15 10
200
x
300
y
400
Figure A visualization of the landscape of the Rosenbrock function f(x, y) = (a-z)²+b(y-r>
with 20 and b 1. The global optima is at 20, y 400 with function value 0. The function value at x = 0, = is 400. The landscape has a narrow valley between (0, 0) and (20, 400).
D PROMPTING FORMATS FOR SCORER LLM
Figure and 9 show examples the Q_begin, Q_end, and A_ A_begin prompting formats when the "QA" pattern is present. The "QA" pattern is eliminated when prompting instruction-tuned scorer models like text-bett text-bison bison with the Qebeiin and Q_end formats (Figure 10 and 11..
Q: (instruction)
Janet's 's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
A:
Figure 7: The Q_begin prompting format on a GSM8K test exemplar with the "QA" pattern.
Q: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? (instruction)
A:
Figure The end prompting format on GSM8K test exemplar with the "QA" pattern.
17

---

Published as a conference paper at ICLR 2024
Q: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
A: (instruction)
Figure 9: The A A_begin prompting format on a GSM8K test exemplar.
(instruction]
Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for S2 per fresh duck egg. How much in dollars does she make every day the farmers' market?
Figure 10: The Q begin prompting format on GSM8K test exemplar without the "QA" pattern.
Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for S2 per fresh duck egg. How much in dollars does she make every day the farmers' market? (instruction)
Figure 11: The Q_end prompting format on a GSM8K test exemplar without the "QA" pattern.
E META-PROMPTS
E. META-PROMPT FOR MATH OPTIMIZATION
Now you will help minimize function with two input variables b. have some (w. b) pairs and the function values at those points. The pairs are arranged in descending order based on their function values, where lower values are better.
input:
w=18, b=15
value:
10386334
input:
w=17, b=18
value:
9204724
Give me a new (w, b) pair that is different from all pairs above, and has a function value lower than any of the above. Do not write code. The output must end with pair [w, b], where w and are numerical values.
Figure 12: An example of the meta-promp for linear regression. The blue text contains solution-score pairs; the orange text are meta-instructions.
18

---

Published as a conference paper at ICLR 2024
You are given list of points with coordinates below: (0): (-4, 5), (1): (17, 76), (2): (-9, 0), (3): (-31,
-86), (4): (53, -35), (5): (26, 91), (6): (65, -33), (7): (26, 86), (8): (-13,-70), (9): (13, 79), (10): (-73, -86), (11): (-45, 93), (12): (74, 24), (13): (67, -22, (14): (87, 51), (15): (83, 94), (16): (-7, 52), (17):
(-89, 47), (18): (0, -38), (19): 58).
Below are some previous traces and their lengths. The traces arranged in descending order based on their lengths, where lower values are better.
<trace> 1.1.1.....0..17777777..004444110 </trace>
length:
2254
<trace> 01144411177477122551001111111122 </trace>
length: 2017
<trace> 011.4441111111111215331111411111 </trace>
length:
1953
<trace> 00D.4448661114441222100000000110 </trace>
length:
1840
Give me a new trace that is different from all traces above, and has length lower than any of the above. The trace should traverse all points exactly once. The trace should start with <trace> and end with </trace>.
Figure 13: An example of the meta-prompt for Traveling Salesman Problems with problem size n 20. The blue text contains solution-score pairs; the orange text are meta-instructions.
E.2 META-PROMPT FOR PROMPT OPTIMIZATION
Different optimizer models work the best on different styles of meta- meta--rompts. Figure 3 in the main paper shows the meta-promer for PaLM 2-I-IT; Figure [4 shows that for pre-trained PaLM 2-1; Figure shows that for GPT models.
Create a piece of text at the beginning of the answer to enhance the precision in solving diverse grade school math problems.
Precision: 4 <TEXT>A dime</TEXT>
Precision: 17 <TEXT>The answer is function, is</TEXT>
Precision: 19 <TEXT>So how can we find out what this equation means?</TEXT>
Precision: 20 <TEXT>SSoltioos<<TTEXX
example of the meta-prompt for prompt optimization with pre-trained PaLM 2-L on GSM8K, where the generated instruction will be prepended to the beginning of the scorer LLM output (A (A_begin in Section 4p.
Figure 14:
19

---

Published as a conference paper at ICLR 2024
Your task is to generate the instruction <INS>, Below are some previous instructions with their scores. The score ranges from to 100.
text:
Let's figure it out!
score:
61
text: Let's solve the problem.
score:
63
(... instructions and scores ...)
Below are some problems.
Problem:
Q: Alannah, Beatrix, and Queen preparing for the school year and have been given books by their parents. Alannah has 20 more books than Beatrix. Queen has 1/5 times more books than Alannah. If Beatrix has 30 books, how many books do the three have together?
A: <INS>
Ground truth answer: 140
(... more exemplars... ...)
Generate an instruction that is different from all the instructions <INS> above, and has a higher score than all the instructions <INS> above. The instruction should begin with <INS> and end with </INS>, The instruction should be concise, effective, and generally applicable to all problems above.
An example of the meta-prompt for prompt optimization with GPT models (gpt-3. 5-o (ootr or 9pt-4) to the beginning of the scorer LLM output (A_ (A_begin in Section 4 The blue text contains solution- score pairs; the purple text describes the optimization task and output format; the orange text meta-instructions.
Figure 15:
on GSM8K, where the generated instruction will be prepended
F PROMPT OPTIMIZATION ON GSM8K: MORE RESULTS
We first present the results of generating Q_begin instructions with the text -bison scorer and the PaLM 2-L-IT IT optimizer, starting from an empty instruction with a 57.1 training accuracy. The optimization curve in Figure [6(a) shows a similar upward trend as in Figure [(a] in the main paper. Table 5 summarizes top instructions, and the characteristics of the instruction generated by each model are similar to those in Table in the main paper.
shows Then. Figure 16(b) shows that the pre-trained PaLM 2-L can also serve as the optimizer LLM and improve own prediction performance. Different from other optimizer L.MMs that are instruction- tuned, the pre-trained PaIM 2-L performs better when the prompt is formatted in few-shot manner. Therefore, we include two initial instructions to start the optimization: the empty instruction (with training accuracy 32.2) and "The answer is" (with training accuracy 33.3). See Figure 14 in Appendix E for the meta- pret-proomp format. The generated instructions follow the same style as "The answer is" most instructions are also phrases suitable the prefix of sentence, like "Here you go:" (generated at Step with training accuracy 61.3) and "Let's 's do it:" (generated at Step 13 with training accuracy 75.1).
20

---

Published as a conference paper at ICLR 2024
Table 5: Test accuracies GSM8K with the text-bi son scorer.
Optimizer Source
Instruction position
Top instruction
Baselines Kojima et al.
Q begin
Let's think step by step.
64.4
2022
al.
Q_begin
Let's work this in a step by step way be have the right answer.
65.6
2022b
Q_hegin
Let's solve problem.
Q_begin
(empty string)
56.8
PalM 2-1-11
Q_hegin
Let's work logether to solve math word prohlems!! First, we will read and discuss the problem logether to make sure we understand it. Then, we will work together solution. will give you hints and help you work through the problem you get stuck.
64.4
text -bi son
Q_end
Let's work thruugh this problem step- ste:
68.5
gpt-3, 3, S-turbo
Q_end
Analyze the given information, break down the problem into manageable steps, apply suitable mathematicul operations, and provide clear, accurate, and concise solution, ensuring precise rounding if necessary. Consider all variables and carefully consider the problem's 's context for an efficient solution.
66.5
apt-4
Q_hegin
dissecting problem highlight important numhers and relations. Decide on the necessary mathematical operations like addition, subtraction, multiplication, or division, required for resolution. Implement these operations, keeping in mind any units or conditions, Round off hy ensuring your solution fits the context problem ensure accuracy.
62.7
80.0
training accuracy
accuracy
70.0
60.0
60.0
40.0
GSMBK
GSMBK
(scorer and optimizer:
text-bison)
PaLM 2-L)
50.0
20.0
50
100 steps
150
200
#
steps
PALM 2-L-IT optimizer
(b) pre-trained PaLM 2-L optimizer
Figure 16: Prompt optimization GSM8K with aa) the text-bison scorer and the PaLM
2-L-IT optimizer, (he PaLM 2-L as both scorer and optimizer.
G PROMPT OPTIMIZATION: INSTRUCTIONS FOUND IN INTERMEDIATE STEPS
G.1 GSM8K, A BEGIN INSTRUCTIONS. PRE-TRAINED PALM 2-L AS SCORER, PALM 2-L-IT AS OPTIMIZER, STARTING FROM "LET'S SOLVE THE PROBLEM"
few leaps in the training accuracy include:
"Let's think carefully about the problem and solve it together." at Step 2 with the training accuracy 63.2;
"Let's break it down!" at Step 4 with training accuracy 71.3;
"Let's calculate our way to the solution!" at Step with training accuracy 73.9;
"Let's do the math!" at Step 6 with training accuracy 78.2.
G.2 GSM8K, Q_BEGIN INSTRUCTIONS, TEXT-BISON AS SCORER, PALM 2-L-IT AS OPTIMIZER, STARTING FROM EMPTY STRING
few leaps in the training accuracy include:
21

---

Published as a conference paper at ICLR 2024
"Solve the following problems using the given information." at Step 2 with training accuracy
59.8;
"Solve the following problems by applying the given information and using the appropriate mathematical operations." Step 3 with training accuracy 64.0;
"Let's read the problem carefully and identify the given information. Then, we can create an equation and solve for the unknown variable." at Step with training accuracy 67.0;
"I'm always down for solving math word problem together. Just give me moment read and understand the problem. Then, I'll create an equation that models the problem, which I'll solve for the unknown variable. also may or may not use some helpful diagrams or visuals to understand the problem. Lastly, be sure to allow me some time to carefully check my work before submitting any responses!" at Step 29 with training accuracy 70.1.
H SEMANTICALLY SIMILAR INSTRUCTIONS MAY ACHIEVE DRASTICALLY DIFFERENT ACCURACIES
One challenge of prompt optimization is the sensitivity of model performance to subtle changes in the instruction. For example, with the PaTM 2-1. scorer on the GSM8K test set, "Let's think step by step." achieves accuracy 71.8, "Let'' solve the problem together." has accuracy 60.5, while the accuracy of "Let's 's work together to solve this problem step by step." " is only 49.4, although it the semantic combination of the two upper instructions. This behavior increases both the variance across single-step instructions and the oscillation during optimization, and motivates us to generate multiple instructions each step improve the optimization stability.
22

---

Published as a conference paper at ICLR 2024
I PROMPT OPTIMIZATION CURVES ON 23 BBH TASKS
90.0
60.0
training accuracy
11
70.0-
70.0
50.0
training
training
BBH
BBH causal judgement
BBH
boolean, expressions
date uniiiuuddestnninnggg
50.0
50
100
60.0
So steps
100
40.0
0
100
150
steps
steps
(a) BBH boolean expressions
(b) BBH causal judgement
(c) BBH date understanding
70.0
training accuracy
training accuracy
60.0
100.0
60.0
50.0
98.0
BBH
dlsambiguation..o
dyck languages
formal,
50.0
0
50
100
50
100
20
40
60
steps
steps
stepss
(d) BBH disambiguation qa
(e) BBH dyck languages
(f) BBH formal fallacies
65
80.0
BBH logical deduction
training accuracy
training accuracy
training accuracy
seven objects
30.0
70.0
20.0-
BBH geometric, shapes
BBH
hyperbaton
60.0
55
150
steps
steps
stspps
(g) BBH geometric shapes
(h) BBH hyperbaton
ii) BBII logical_deduuction deducctin, objects
70
training acturacy
Taaning Tainingaae accuracy
20
65
M Alt a Aaa
80
10
60
70
BBH movie
BBH multistep. anthmeticc two
recommendation
BBH navigate
60
50
100
150
200
50
100
150
200
40
80
120
steps
# steps
ii BBH movie recommendation
(k) BBH multistep arithmetiic two
(I) BBH navigate
70
training accuracy
training accuracy
pece securary
70
80
60
Sentre S 20
50
60
BBH rea soniing about colored objects
BBH object counting
BBH penguins in table
40
100
steps
steps
steps
(m) BBH object counting
(n) BBH penguins in table
(0) reasoning about colored colored objects
Figure 17: (Part 1/2) Prompt optimization 23 BBH tasks with the text-bie scorer and the PaLM 2-L-IT optimizer. The optimizations start from the empty string. Most curves have upward trends.
23

---

Published as a conference paper at ICLR 2024
training accuracy
training accuracy
80
40
BBH salient, translation, error detectionn
70
BBH names
30
BBH snarks
0
50
1000
150
200
20 # steps
50
100 steps
150
200
steps
(a) BBH
RBH valient transiatioo, cmro detection
BBH snarks
100
training accuracy
training accuracy
70.0
20
50.0
BBH sports understanding
BBH
10
BBH tracking shuffled. objects_ wewseeeveeeeee objectts
30.0
temporal sequences
40
20 steps
50
150
steps
steps
(d) BBH gports_understanding
(c) BBH temporal_sqquencee sequences
(
BBH
tracking_shufien shuffled
objects_ ceaidd__svee_ojeett
training accuracy
60
traiming accuracy
20
10
BBH web of lies
BBH word sorting
150
100
150
200
steps
steps
(g) BBH web of lies
(h) BBH word sorting
Figure 18: (Part 2/2) Prompt optimization 23 BBH tasks with the text text-bison scorer and the PaLM 2-L-IT optimizer. The optimizations start from the empty string. Most curves have upward trends.
J PROMPT OPTIMIZATION ON BBH TASKS TABULATED ACCURACIES AND FOUND INSTRUCTIONS
J.1 PAIM 2-L-IT AS OPTIMIZER, OPTIMIZATION STARTING FROM THE EMPTY STRING
Table 7 and Sisoow the instructions found by prompt optimization. A comparison of their accuracy values with baselines "Let's think step by step." (Kojima et al. 2022). "Let's work this out in step by step way to be sure we have the right answer." (Zhou et al. (022b), and the empty string is in Table 6 A bar chart visualization of the performance difference with the PaLM 2-L scorer is Figure (a)) and [9(b) (which are the same as Figure ((a) [bb]) in the main paper but have BBH task names); bar chart visualization with the text -ortt-biie son scorer is at Figure (9(c) and [9(d)
24

---

Published as a conference paper at ICLR 2024
60
40
difference
accuracy differncce
40
20
20
-20
boolean, poaeementt causa date un Bon disa disam aall
myperbacc logical deduction ses seven objeds dation Rccomm mas FaC comnmme mudliste
Formial shaDes goomett
saltent transstaatiin lation, arrer crrection
pent penquit coleets calored object mi
tWO
court
tracking shuffted objects seven objects wel word
ruln nome
sports undersi persiandinn tember sequense
sorting
bootse ooe reclioos ioross conee 2id botauuiiii correei dadeaant
shapes hypol Der Lon
shapes faliscialliiioo ceometicc
otjecttn sewen atjects reduction sewer
da
far cemment tNO havterin 10 Cabd calrro object transtraaaenoo trans srror detectio irvectiion
understniia paaskanding tembora requence tracking shufted objects seven oblosts web jis sorting
mo ao retterttmee multit
sports undo underdaa
word
saltent colorrcccoaaaoo
db
did
sbout
reasoning
logical
(a) PatK 2-L scorer, ours minus "Let''s think step by step."
(b) PatM 2 scorer, ours minus empty starting point
accuracy difference
accuracy difference
20
20
Siterion diee Son comnal ianper Bilamen
goomettcc Cinertatio hygertalie
onddn sev seven
abject countinn reasoming penqurt puguinns tablo colored dbjects ruln names salient transiation arror. detection perection
teo
sports understan sovrees tem Cemporal senvontce tracking shuffled, objects seurnn dbject.
web sorting
e oo
corate comern de fallaciss coneeft shapes (artaaliai
pegicl mouctio sey hypertiec no recarme difontton rectrreeee (Wo mlanggai countin
reasoning about colored oblect ruin object, penguins i0 labl
salient translation error. detection ecttioo
sports condertton porre porstanding trackingg shuffted objects seren objects wet sorting word
smarks
bootean aeeeeeeee
date
deductionn
logical
(c) text-bt son scorer, ours minus "I et's think by step."
(d) text-bl son scores, ours minus empty starting point
Figure 19: On 23 BBH tasks, the accuracy differences among instructions found by prompt op- timization (with the PaLM 2-L-IT optimizer), "Let's think step step.". and the empty string (optimization starting point).
25

---

Published as a conference paper at ICLR 2024
Table 6: Accuracies BBH tasks: our found instructions with the PaTM 2-I-IT optimizer vs baseline. The optimization starts from the empty string. Because of the 20-80 train-test split, we show accuracies with the format "training / test / overall (training + test)". The PaLM 2-L scores are from A begin instructions; the text-bisee scores are from Q begin instructions. Bold numbers indicate the best for the corresponding task.
Task
Scccer
Our Ace
"Let's think step by step." Ace
"Let's work this out
engey string Ace
in stop by step have
right answer"
training test / overall
training tent everall
triinngg best ovvenall
training / lest overall
boolean_cxpeeee expressioos
RaLM
82./
cansal judgement
BaIN 2-2
84.8 58.0 63,1
73.0/55 55 58.8
59,5 57.3 /
29.71 49.3 44.55
date understanding
PalM L
H6.0 84.5 84.8
76.0/ 80.0 79.2
74.01 77.0 76.4
70LD/ 74.00 73.2
dismixquution.ga
PatM 2-L
80.0/ 09.00/ 71.2
40.07 52.5 50.0
48,01 47.0 44
S4.D/5..5 56.8
dyck Junguages
PAIN 2-5
100.0/ 100.0/ 100.0
96.0/ /9188
100,0 93.5
94.0 95.0
fallacies goometric shapps
2-1
PatM 2-5
76.0/57.0/ 69.8
4
34,8
42.0/ 33.0
34.0/ 33.0/
hyperbaian
PAIN 2-L
100.0/90 96.0/ 96.8
/ 75.07 75.6
74,0/ 72.5 72.8
88.0 /9/00/ 38.8
dedccildee geven_ objects
FaIM 2-D
74.0/57.00 60.4
46.0/ 37.0 38.8
31,0/ 30.5 / 31.2
46.0/ 45.5 /5.6
movic recommenntiii
PatM 2-5
92.0/ 90.5/ 90.8
62.0 54,4
52.0 48 0 /488.5
83,0/ $2.4
muliistpp aniiemetic
72.0/
50.5
4.533.5/3
savigatz
Pa1M
0
70,01 665.2
37.5/
object countin
PalK 2-L
84.0 86.5 $6.0
36.0 26.5 1.44,4
60.01 /2.6/666
28.0/ 27.0 23.2
pengains in a table
PatM 2-5
N6.2 718/ 74.7
7 167.1
62.1/ / 58. 58.9
72.4 69. 2/ 69.
ressoning about coloned sibjocts
PatM 2-5
98.0 / 85.5 SSL0
82.0 79.5 18000
$2.01 75.0 76.4
42.0/ 35.01 36.4
Pa1x
salient transtation pcroc_detectioo
PaIN 2-L
4200 50.0
58.01
starks
EGIN 2-2
85.7 83.2 83.7
02 166.8
/ 53.1 5
51.4 60.1 58.4
sports anderttninii
Pa1K L
95.0 / 90.0
50.0 26-5 147222
6IL0/ 52.5 $2.5/54.0
$2.0 41 5/ 43.6
temporal seeuuences
PaIN 2-D
100.0/ 100,0/ 100.0
TOKJU 99LD 96.8
9IL0/ 57.0/8766 87.6
TOCLO/ 99.5 992.
tracking shufiled jejeecseeeeeeeecccjccc
FAIH 2-5
32.07 16.5/
58.0 61.5
55,5/55.2
14.0/ 23.5
of lies
FaIM 2-L
word sorting
PatM 2-1
54.0/54,555..
2.0/4.//4
12.02 9.5 10.0
20.0/ /
boolcin,
Tst
72.0 67666
801
SOLD 70.8
consal, judgement
-ex--bi
75.45%%0
50.7 54.5
73.0
58.0
date undersannding
:ex:-bison
60.0/50.0/ 52.0
44.07 25 45..
48.01 45.0
44.0 45.0/ 44.8
disambiquation.qa
=cx:-bison
68.0/ 72.0
1.0/5.75.6 5.6
4.0/ 1
$2.0 68.5
dyck_langgagge languages
Lex aun
100.0/100001 100.0
100.0/ / 95.5 /964
100.01 94.5 95.6
TOCLD/ 98.5 99.8
fartul_falllcies
tex:-biaon
70.0/53.01 56.4
88.8
gcometric-happs
:ext-bison
19.5
18.0/
hyperbuium
tex:-bison
80.0 79.5/ 79.6
61.01 57.5 66.8
60.0/ 69.00 68.0
6 61.0 61.0
logical deduction seven objects
texc-bi
66 06.0/5 53.5 56.0
56,0 / 57.6
56.0 56.0 56.0
5 56. S/ 56.8
movic, recvmmenddaiion
text-bi son
98.0/ 91.8/ 91.6
68. 63.0 64.0
66.0/ 62. /62.8
68.0 64 64
multistep_ satppanntmmeetteee two
scx2-bison
32.0/ / 19.6
12.0/ 18.0 16.8
18.0/ 17.5 17.6
16.0/ 18 5/
savigite
zext-bi
72.0/ /1.0//6./////
57.0
object counting
texe-bt son
72.0 62.0/ 64.0
57.2
62.0/ 56.8
57.07
pengaiss Jnt rabllc
tex: bs son
72.4 56.4 59.6
S8.6 55.0 /
55.2 6 55.5
/ /3.0/ 54.1
ressening ahhout soijetts
Lex--bi -tc--biaaee son
N2.0/ 77.8/ 75.0
7/.0/ 72.5 /77322
78.0/ 73. 74.0
74.77 68 70.4
min
text-bl
88.0/ 82.5 83.6
66 65.5 165.6
66.0 63.2
64.07 / 66 0
stlient translation detecctoo
4
starks
Lex
71.4/
sports_ Bodts_understaaddnng
:ex:-blson
91.0 82.5 84.8
56.07 79.0 /80.4
90.0 $1.0 82.8
38.0 44.5 43.2
temporal sequences
scx:-bison
78.0 /1.0 90.4
36.0/ 43 42.00
32.01 0 <2.4
36.0 43.0 41.6
tracking shuflled objects weweeeeeeeeeeeeeeeeeeeeeeeeee objects
text-bison
32.0/ 15.5/ 18.8
10.0 1 15.6
10.0/ 18.0/ 16.4
12.07 15.5/ 14.8
web_ot_lis word_socting
texi-biaon
62.0 50,0
44.0
52./51.5/51.
:exc-bison
24.0/
10.0/ 12./112/
4 8.0
4.0/75/668
26

---

Published as a conference paper at ICLR 2024
Table 7: BBH task-wise instructions found by prompt optimization with the PATM 2-L scorer and the PaLM 2-L-I7 IT optimizer. The optimization starts from the empty string.
Task
Our Instruction
hoolean__xpressionn
Boolean expression is well-formed expression consisting of variables, values, and logical operatoes, The expression musi evaluate to a single True or False value. The order of precedence of the logical operatons is as follows: NOT, AND OR, XOR, IMP, Purentheses can he used group suhexpressions and contro the order of evaluation.
sansal_juggement
When considering questioos about causation, a typical person would consider the following factors: whether the action event was a necessary condition the outcome occur, sufficient condition. proximate cause, or foresecable cause.
date understanding
time ago koday, first find today's 's date, Then subiract tine from today's 's dale. the current date is the last day of month, then the date a month ago is the last day of the previous month. If the current date is not last day of a month, theu date a month ago is the same day of the previous month. For examplc, today is March 31, 2023, then the date month ago February 28, 2023, today is April 2023, then the date month ago is March 1, 2023.
find
disambigaation.
Identifynng Antecedents of Promouns: A Comprehensive Guide
dyck languages
First, look for the opening parentheses. Then, count the number of opening parentheses. Finally, close the parentheses in the reverse order that they were opened.
formul_aallacess
A deductive angument is one where the conclusion follows nevessarily from the premises. If the premises are true, then the conclusion must also true. An invalid argument one where is possible for premises to true the conclusion to be false.
goometric_shapge shappes closed polygonal chain series of connected line segments. The line segments can be straight or curved. The first and last line segments are connected. The line segments do not intersect each other except at their eadpoinns, A closed polygon can be described by an SVG path element, which starts at a given point, goes to one or more additional points, and then ends the starting point. The puth element can consist of straight line segmentss, curved segmenis, a mixture of both.
hyperbaton
The correct adjective order in English is opinion, size, shape, age, color, origin, material, and purpose. If you have more than one adjective of the same type, they are usually placed in order of importance. For example, you would say large, old, Pakistuni ship' rather than "an large. Pakistani ship." There are few exceptions to these rules, they are generally followed in most cases.
logical, cleduction _evven obbjects
The following questions test your ahility to use deductive reasening. You will given a set of statements ahout group of objects. You will then be asked answer questions about the objects based on the statements. The statements in the questions are logically consistent, so you can use them to deduce the order of the objects. For each question. you must choose the option that is logically consistent with the information in the questions.
movie_ reweeerecommendation
Based on your input, I have unalyzed the given movies in terms of genre, plot, tone, andience rating, year of releuse, director, cast, and reviews. have also taken account the given options. The movie that most similar the given movics in terms of all these factors is:
mwhtistp_eeithmmete arithmetic _ww
The order of operations mathematics PEMDAS, which stands Parentheses, Exponents, Multiplication. Division, Addition, and Subtraction. When there are multiple operations of the same precedence, they must be performed from right. Note that multiplication and division have same precedence, as addition subtraction.
navigate
You will return to starting point and only if (l) total number steps you take forward equal to the total number of steps you take back, and (2) the total number of turns you make is multiple of 180 degrees.
object_counling
Here is list of the objects you mentioned and their corresponding counts:
penguios_ tttble
Here my new
reasoning about _colored _coloreed_sjectss
Starting from the leftmost object in the row, I observe the following objects arranged in this order:
ruin_ names
Which is the funnicst pun on the artist or movic name?
salient_ trunslation .eror detection
Instructions: Read the German sentence and its English translation carefully, then identify the type of error in the translation and select the correct option. There are six possible types of erron: Named Eniities, Numerical Values, Modifiers or Adjectives, Negation Antonyms, Facts, and Dropedd Content.
snarks
Ideatify the sareastic statement by considering the following factors: incongmiity, exaggoration, understatement. context, speaker's intent, and audicnce's 's reaction. will also consider the speaker's 'ss tone of voice, facial expressions. and body language.
sports uondsssnddrssnnninngg
I will determine if sentence about athlete plausible lirst checking if grammatically correct. it I will then check if it is consistent with the athlete's 's sport, position, and real-world statistics. I will also check if consistent with the rules of the athlete's sport. If the sentence consistent with all these things, I will answer "yes". otherwise I will answer "no".
lemporal_ [emporal_sequencee
The answer is that mentioned the given statements.
tracking shuffldd objects __seeen_eeee objects
Claire has the blue ball, Gertrude has the black ball, and Dave has the green hall. They are all hapgy with their new halls.
web_of_jier
The answer question yes if there odd number liars before the current speaker. and if there even number of liars before the current speaker. If the current speaker is truth-teller, they will say the oppositc of what the previous person said, while liar will say the same thing as the previous person said.
word_sorring
Alphabetical order of given words:
27

---

Published as a conference paper at ICLR 2024
Table 8: BBH task-wise instructions found by prompt optimization with the text-bb son scorer and the PaLM 2-L-IT optimizer. The optimization starts from the empty string.
Task
Our Instraction
boolean expressions
Not (not False) and not not False is False
causal judgemmet
typical person would likely answer the questions about causation follows:
date_ understanding
Today is February 28, 2023. It is a Tuesday, Yesterday was Monday, Februurry 27, 2023. Tomorrow will be Wednesday, March 1, 2023. A week ago, was February 21, 2023, and a month ago, was January 2023, year from now, it will be February 28, 2024. The day of the wesk is important to note because it will help to correctly answer the questions below Not all years are leap years that contain February 29.
disumbiguation qqa
A pronoun word that stands for noun. The the antooedent of a pronoun, look for the noun that the pronoun could be referring to. there only one possible noun, that antecedent. If there are more possible nouns, then the antecedent ambiguous. Use context of the sentence to help you determine the correct antecedent.
that a promoun refers is called its antecedent. To identify
Janguagss
formal, fallacies
How Evalunte Deductive Validity an Argument
geometrie_shapee shapes
What shape is this SVG code driwing, and how many sides does have?
hyperbaton
English, adjectives are typically placed before nouns a specilic urder. The order opinion, size, shupe, age, color, origin, material, purpose, noun. For example, the sentence "the big, old, red harn would he considered grammatically correct, while the sentence old, big, red barn" would not. Adjectives that come before nouns are called attributive adjectives, while adjoctives that come after nouns are called prodicative adjectives.
logical deduction _svven sojen_objects
In this logical reasoning task, you will he given a series of paragraphs, each of which describes a set of objects arranged in fixed order. The statements in each paragraph are carefully and use the information given to determine the logical relationshiis between the objects. You will then be each question carefully choose the option that answers asked yuestion about order the objects. question correctly.
consistent. You must read each paragraph
roovierecommeenatioon
What is the highest-rated movie similar to the given movies, with similar IMDs rating and released in the same year?
multistep_arithmmeticcc arithmecti__twe two
Let's solve these equations using PEMDAS order of operations. Remember that PEMDAS stands for parcatheses, exponenis, multiplication and division. addition and subtraction.
navigate
Starting at the origin, facing north, follow the instructions. If your displacement from the origin is zero and your direction is unchanged, then your answer is Yes. Otherwise, your answer is No.
object_counting
Let me help you count the items you have. Just list them one one, separated by commas. will then count each item and tell you how many items there are in total.
penguins_ _______________ t_tale
This table shows information about penguins. The columns show the penguin's name, age, height tin cm), and weight (in kg). The penguins are listed in order of their age, from youngest to oldest.
rcasoning_abow about cobjred_objeets
First, read the input carefally. Then, identify all the objects mentioned, their colors, and their positions. Next, visualize the objects and their positions in your mind. Finally, answer the questions accurately based on the information given. Make sure to pay atlention to the order of the objects.
min_ names
A humorous edit of artist movie phrase that sounds similar but has a different meaning. The new word or phrase should be relevant to the original word, hut it should be surprise, which makes the edit funny. For example, the artist movie name "Racky" can be changed to "Ricky," and "Schindler's List" can be changed to "Schindler's 's Lift: " Be creative and have fun!
he created replacing ing
letters form
word
salient_ translation
The following translations from German to English contain particular error. The ervor may be one of the following types: Named Entities, Numerical Values, Modifiers Adjectives, Negation Antonyms, Facts, Dropped Content. Please identify the error.
_erorr detectioo
snarks
The statement
sports understandiing
To determine the plausihility of sports sentence, will first identify the sport, athletes, teams, and events mentioned the sentence. Then, I will use my knowledge of rules the sport. the context the sentence, common sense, and my knowledge of the world to determine whether the sentence is plausible. I will also cousider the time period and location, as well as any other relevant information. Finally. I will return a score of I for plausible sentences and U for implausible ones.
temporal_sequaane sequences
To determine the time period when person weat to a place, first identify all the time periods when the person's 's whereahouts unknown. Then,
any time periods during which the person was seen deing something else or the place was closed. The remaining time periods are the possible times when the person could have gone to the place,
tracking shuffledd_ojeeeee .eeven objects
At the start of the game, Claire has blue ball. Thrughout the game, pairs of people swap balls. Claire ends up with the yellow hall.
web_of_ lies
People in a group either tell truth or lie. The muiihfulness of person's statement determined by statement the previous person. If the previous person told the truth, then the current person who says the opposite is lying. If the previous person lied, then the current person who says the opposite is telling the truth. This rule applies to subscqueat statemeuts.
word, sorting
Sort the following words alphahetically, ignoring case and panctuation. Print the sorted list.
28

---

Published as a conference paper at ICLR 2024
40
accuracy ditference
40 Comeeeee accunns Saa.eea
20
Bo som
-20
Wount reanss arouee moy teekong
to uocenisoon
Peo pogget uteood
d degen
pacaagg 2000
toggnn jeadm
Bituaeeus
(a) PaLM 2-1, ours minus "Let's think step by step."
(b) PaLM 2-L, ours minus empty starting point
accuresy difference
aifffrencc
20
20
0-
oor your
pnp swade
paggo
cettennee Juayes
settenue wayess
Buuooral nar
Conttene mackin
Squaseai ma
(c) text foxt-bisee bisonn ours minus "Let's think step step."
(d) text text-biieeeeeeee beattbbioo,, ours minus empty starting point
Figure 20: On 23 BBH tasks, the accuracy differences among instructions found by prompt opti- mization (with the gpt-3. gpt-3.5-turbo 5-turbo optimizer), "Let's think step by step.", and the empty string (optimization starting point).
J.2 GPT-3.5- AS OPTIMIZER, OPTIMIZATION STARTING FROM THE EMPTY STRING
Table TO. and 12 show the instructions found by prompt optimization. Their accuracies are listed in Table Figure 20 visualizes the difference between their accuracies and those the baselines "Let's think step by step." and the empty string. The optimizations find instructions better than the empty starting point, and most of the found instructions are better than "Let's think step by step".
One caveat in the A_ begin instructions (Table 10) is that some instructions found here are imperative or interrogative sentences that are more suitable to be put into "Q:" rather than "A:", like "Solve the sequence by properly closing the parentheses." " for dyck languages and "Which movie option from the given choices ...?" for movie recommendation. Such styles appear more often here than the with PaLM 2-I-IT IT optimizer (Table indicating PAIM 2-I-IT understands the desirable style
29

---

Published as a conference paper at ICLR 2024
Table 9: Accuracies BBH tasks with the 5-turbo optimizer that starts from the empty string. The PaLM 2 L scores are from A begin (left) instructions; the text -bison scores include Q begin (left) and end end (right) instructions.
Our Ace (ongin)
Our Ace (end)
training overall
training test/ overall
boolean_ expressions
PaIM 2
92.0/
NIA
causal juusal_judgemen
PelM 2-1
81.1 / 58.7 63.1
N/A
date undersianding
FaiM 2-1
86.0 82.0 82.8
NIA
disanhiguation
2-1
80.0 74.0 75.2
NIA
dyck_ Janguages
PalM 2-1
100.0/ 100.0/ 100.0
N/A
88.0
NIA
geometric shappes
PalM 2-1
60.0/ 41 44.8
NIA
hyperbaton
PaIM 2
98.0 93,0 / 92.0
NIA
logical, ded.cceeonnneeee seeeeseeveeeeeeeeeee objects
PaiM 2-1
76.0 56.5 60.4
NIA
movie, recummenndation
Fa1M 2-1
84.0 86.0 85.6
NIA
arithmetic multistep
2-1
$2.0 49.0 49.6
NIA
navigate
Pa1M 2
76.0 67.0 G8
N/A
counting
penguins_ in in_a_tbllc
PaIM 2
82.8 72.6 '74.7
NIA
reasoning about _ccoloooddooooo oobjects
Pe1M 2
86.0 71.2
NAA
FaiN 2-1
90.0 83.0 84.4
NAA
salient, trunslation emor detection
Pa1M 2-1
62.01 65.0 / 64.4
NIA
snarks
85.7 70.6 73.6
NIA
sports_ sports_ understanding
PalM 2
68.0 57.5 59.6
NIA
temporal sequences
19.6
tracking_sbuttee shuftled_ seveross_eeeeobecccss
PaIM
44.0 34.5 / 36.4
N/A
web_ veb__o_lies
PaiM 2-1
92.0 /912
NIA
word sortng
PalM 2-1
62.0/52.0 54.0
NIA
booleant_cxpreseettt expressions
text
84.07 78.57 79.6
80.0 78.0 78.4
causal judgement
text-bisen
78.4 57.3 / 61.5
83.8 53.3 59.4
date understanding
Lext bison
52.0/ 45.0 46.4
64.0 52.4 54.8
disanbiguation_g
text-bison
68.0 75.5/ 77.0
64.0/ 71.5/71.. 70.0
dyck_ Janguages formal, fallacies geometric soometric_shppes
texc-bisen
100.0 99.5 99.6
100,0 100,0/ 100.0
text-bison
70.0 54,5
74.0/ 53.5 57.6 57.6
Lext-bison
28.0/ 15.0 17.6
48.0 28.0 32.0
hyperbaton
85.2
Iogical_deductton.
logical deduction seven objects
text-bison
66.0 57.5 59.2
62.0 55 0/ 56.4
movie_ roreikoooommnoiiion
text bison
76.0/ 69.5 70.8
82.0/ 70.5 72.8
multistep writhmeticc_oo
text-bisen
28.0/ 20.5 /2.0
28.0 22.5 23.6
navigute
texc-hison
72.01 /
68.0 / 59.5 61.2
object counting
text-bison
68.0/71 70.4
72.0 09.0//000000.. 69.6
penguins_ in aatale
text bisen
65.5 39.8 6L.0
79.3 53.0 58.2
reasoning aboubbbuttoc colored objects
86.0
min names
text bison
84.0 75.2
74.0 75.0 0 74.8
translation salient_translation emor detection
salient
text-bison
44.0/ 49.2
48.0 51.0 0 50.4
snarks
text-bison
79.9/79.7 80.3
88.6 84.6 85.4
sports_undente understanding
text-hison
84.0/76.5/78
90.0 80.0 82.0
temporal sequences
text-bison
30.0/54.5/34.
64.0/61 62.0
tracking_shar swwfthed__oppeeeee objects web. of lies word_sorting
beejocts_seee__ojieee soven_objects
text-biscn
22.0/ 18.5 19.2
30.0 21.5 23.2
68.0
text bisen
26.0/ /9.0 20.4
32.0 25.5 26.8
we show the A begin optimization results with the non-empty starting point "Let's solve the problem.". Most results there declarative sentences more suitable for A. begin.
better. In Section J.3
30

---

Published as a conference paper at ICLR 2024
Table 10: BBH task-wise instructions found by prompt optimization with the PaIM 2-L. scorer and the gpt-3 3. 3.5-turbo optimizer. The optimizations start from the empty string.
Task
Our Instruction
hoolean expressions
An accurate evaluation of logical exprescions involves correctly applying Boolean operators, considering the onder of operations, and analyzing the truth values of the operands in accordance with Boolean logic principles.
causal judgement
Understanding causality critical for accurately assessing cause and effect relationships various scenarios, leading to sell-informed judgments, precise conclusions, and definitive answers to questions about the outcomes involved.
date_ uod__nndrstnn
What the specific date mentioned or required in each given problem or question, taking into account all relevant information, available options, and the providdd context? Please provide the accurate answer the format
MM/DD/YYYYY
disambiguation_
Accurately analyze and clarify the poooon-anteeeddnn relationship the given sentences, identifying the appropriate referent to eliminate any potential confusion or ambiguity and ensure precise understanding of the intended
meaning.
Janguagss
Solve the sequence by properly closing the parcadheses.
formal_ fallacies
In determining the deductive validity of arguments hased on explicit premises, meticulous analysis of the logical relationships and implications essential for definitively establishing their soundness, confimingg their validity or invalidity, and easuring reliable and robust assesment of the arguments at hand.
geometric_shapee
The SVG path element with the attribute plays crucial role in development, allowing for precise definition and rendering of various shapes on a webpage.
hyperbaton
Understanding the correct order of adjectives crucial for constructing grammatically accurate and coherent sentences that ellectively convey the intended mecning diverse contexts while ensuring clarity, cohesion, and consistency throughout consistently and effortlessly.
logical, deduction _seeen objjects
By conducting meticulous analysis of the given information and ensuring logical consisteacy within each paragraph. we can accurately determine the precise order or ranking of the mentioned objects, allowing us to confidently and consistently identify the correct answer every presented scenario
utmost precision and confidence.
movie_ recoomeeeeeaiionn
Which movie option from the given choices closely matches the mentioned films terms themes, storylines, and characteristics, guaranteeing the highest possible similarity score among them all?
multistep, arithmeticc_tve
Evaluate given mathematical expressions step by step to determine the correct solutions accurately.
navigate
Is it possible to determine, with absolute certainty. whether strictly adhering to the given instructions will unfailingly bring you back original starting point without any exceptions, ermors, or deviations?
objoct_counting
Deternine the total number of objects or cutities meationed in the given list. covering various categories and types, accurately calculate the overall count.
penguins_ sat___inna__ttt a_table
From the given table, what information can gather about the mentioned animals and their respective attributes. including names, ages, heights, and weights?
reasoning about _colored_objoot jojjocts nain_
By iboroughly examining given information, accurately determine answers each question by considering the specific characteristics, colors, and positions of the mentioned objects.
Select the most smusing and clever alteration from the options provided for the given artist, movic, or title name, and accurately chouse the correct answer lest your wit and creativity.
salient_tran translation _errorr detection
Theroughly examine the given translations from German to English and accurately identify any crrors by carefully analyzing the text and selecting the appropriate option with meticulous attention detail, precision, utmost accuracy. and comprehensive understanding of the langugge for precise evaluation and categorization.
snarks
Which option delivers the most devastatingly sarcastic response, brilliantly exposing the sheer absurdity and leaving absolutely no doubt whatsoever in all the given situations?
sports, uaantssundesii
Maintaining the accuracy, reliability, and integrity of sports event representation is essential for upholding the highest standards of credibility, trussworthiness, and overall quality in conveying information. without any compromise, misrepreseniation, or distortion, thereby ensuring the factual accuracy of spurts journalism.
temporal sequences
Based on the provided timeline and observed activities, we can accurately determine the possible time range each individunl could have visited their intended destinations and answer questions about their visitation time.
tracking shuffled obbjsts _eevenoobjetts
An important paint to note is that each person in the group starts with one specific book at the beginning of the semester.
web_of_lies
Analyzing the consistency and accuracy of statements provided by each person is crucial for determining the truthfulness of individulls in every scenario.
soriisstting
Please sort the given words in alphabetical oeder: The list of words to he sorted contains -
31

---

Published as a conference paper at ICLR 2024
Table 11: BBH task-wise Q_ Q_begin instructions found by prompt optimization with the text son scorer and the gpt 3. 5-turbo optimizer. The optimizations start from the empty string.
Task
Our Instraction
boolean expressions
Group sub-exprest expressions with parentheses to accurately evaluate logical operations: not, and, and finally or. Determine the resulting value either True or Folsc.
causal, judgemenn
Consider the intentions and actions of the individuals involved.
date understanding
Determine the day difference the given
express in
MMIDD/YYY:
disambiguation
Determine the precise antecedent of the pronoun the given sentence and select the correct option state if ambiguous.
dyck_ Janguagas
Ensure that opening brackets have conrespeading closing bracket, and that the closing brackets are the correct order..
fornal_fallacier fallacies
Thoroughly analyze explicitly provided premises and determine dedlctive validity argument based all necessary conditions, implications, exclusions, and dependencies given.
geometric_shapee
Amalyze the given SVG path element carefully and conlidentyy select the correct option from the provided choices to accurately determine the corresanndngg shupe. Pay close attention to the specific path details and confidently make the most suitable choice.
hyperbaton
Select sentence that strictly adheres the standard material, and purpose. Ensure there are no deviations or alterations in the adjective order. Choose the option without any changes.
adjectives: opinion, size, age, shupe, color, origin.
logical_deduucton deducctonn seven objects
Analyze the given information accurately determine the precise order and ranking of the mentioned objects/pepple considering their relationships, positions, and any provided comparisons, for definitive and logical progression with maximum accuracy and efficiency.
movie_ recommendation
Based on the movie list provided, carefully consider your preferences and make well cell--iffeeed decision.
multistep_ arithmetic two
First, simplify any expressions within parenteses following the correct neder of operations to accurately evaluate final answer with efficiency and precision.
navigate
Always face forward. Take 10 steps forward. Tur left. Take 5 steps forward. Take 3 steps hackward. Finally, take steps forward. Turn around and take step forward. Repeat the previous sequence three times. Follow the given path precisely without any deviations. At the end, tum right and take II steps forward. If you follow these instructions, you return
starting point? Options:
countinn
Determine the total count of mentioned vegetables accurately and state the final count the answer.
penguins_in, tntlbeb
Analyze the given table to accuratoly determine roquired information based on the provided criteria and attributes of the penguins and giraffes. Utilize efficient prohlem-solving strategies arrive the correct answer.
reasoning_abov about _colored_ objects ruin names
State the color of the object mentioned in the given arrangement with utmost accuracy.
Choose the option that offers the most clever and humerous alteration of the given artist or movie name. Let your creativity shine and select the answer that will undoubtedly bring smile to your face! Make sure to think outside the
box!
salient_ translation detection
Analyze the translation and accurately identily the specific error type bused on the source texi, providing the most approprrite comesponding option.
snarks
Choose the option that wickedly embodies sarcasm.
sports_ porssuundestanddnng
Determine the plausibility the given statement by evaluating factual accuracy, logical consistency, and contextual
relevance, then provide succinct and well-justified response.
jemporal_sqquencer
Identity the optimal time slot for the individual to engage in the mentioned location/activity considering the given sightings and waking up time, laking into account the opening and closing times the location and the duration of each event.
tracking shuffled oobjees _eeeeeeeeeeee objects
Pay attention the given information track swaps/exchanges carefully accurately determine final possession/usiiion /ottoo/ for the specified individual.
web_of_lies
To determine the truthfulness the last person meationed. analyze the consistency of each statement and count the number of individulls accusing the previous person lying. If the count of accusers
that person tells the
truth; if it is odd,, that person lies.
word_sorting
Alphabetically sort the given words, ensuring all words are included and ascending order.
32

---

Published as a conference paper at ICLR 2024
Table 12: BBH task-wise Q_end instructions found by prompt optimization with the text son scorer and the gpt- - 33 5-turbo optimizer. The optimizations start from the empty string.
Task
Our Instraction
boolean expressions
Accurately use order of operations and parentheses to evaluate Ingical expressions and determine truth values efficiently.
causal, judgement
Consider all relevant factors, priocitize overall well being and ethical consideration, make well informed decisions while foreseeing potential consequences efficiently and consistently strive optimal outcomes with emparhy adaptability in a thoughtful and compreheasive manner.
date understanding
Suhtract the specified number of days from the given date and format the outcome as MM/DD/YYYY to accurately determine the desired result efficient manner.
disambiguation_cm qa
Clearly identify and select the unambignous antocodent for the pronoua or designate as "Ambiguous" if it unclear.
byck_langsnges
Add the missing closing parentheses.
Jallalfallacies
Determine the deductive validity of the argumenn presented based on the explicilly stated premises and reach definitive conclusion.
prometrie_shapes
Analyzing the given SVG path element, accurately determine its shape by closely examining its curves and coordinates, then select the correct option.
hyperhaton
Choose the option with the correct adjective order in each sentence, prioritizing specific attrihates like size, color, and origin. Place the most specific adjective before the more general ones for precise and standardized ordering across all examples. Easure accurate alignment of the adjectives based on their respective attributes for consistent and standardized ordering.
logical, deduction _oven_obbects
Determine the precise order of the given objects/participans based the provided information and establish the final ranking accurately, considering all relevant factors, while maintaining logical consistency with maximum efficiency.
movie recommendation
Choose the most similar option from the choices provided that closely aligns with the given movies' themes, genres, and impact for the most accurate rocommcndation possible. Make your selection wisely.
mullistep_arittmmetie..
Carefully follow the order of operations to precisely simplify the expressions within parentheses and efficiently find the accurate final answer.
navigate
Always face forward. Take 10 steps forward. Turn right and walk for 5 steps. Then, make left turn and continue for steps. Proceed walking steps backward. Hinally, turn around and take 200 steps. Accurately track your movements, diligenlly adhere to the given path, and ensure to return to the starting point without any deviations or obstacles.
ubject_counting
Determine the total count of items mentioned, including all listed items, using eflicient and concise method. State the final count as your answer.
penguins_ _aciii___________ table
Identify animal with the maximum measurement (weight, age, height) in table and its name and species.
rcasoning_abow about colored_objeets
Determine the color of each item the given seenario and select the correct color option from the provided choices for accurate responses, ensuring utmost precision and completeness.
ruin_ names
Chouse the option that creatively and bilariously transforms the given artisi or movie name.
salient_ translation _erroer detertdetection
Carefully analyze the translations and select the most suitable option from the given choices roctify the spocific ervor category, ensuring complete precision, accuracy, and fuithful representation of the intended meaning, while considering all relevant information the source text.
snarks
Choose the option that eleverly employs sareasm to defy all expectations and leave everyone utterly dumbfounded. questioning the very essence of their own perception.
sports_ understanding
Evaluate the plausibility of each given statement and provide well supported justification based on logical reasoning. contextual understanding. and relevant evidence to arrive at a detinitive and conclusive answer.
remporal_sequence
Identify the possible time slot the desired activity based on the given information sightings, then select correct option.
tracking shuffled_ obuffteddoobjetss sejen_s
Thoroughly analyze the given soenarios, systematically consider all available information, and contideatly determine the linal outcome with exceptional precision and optimal elliciency, while maintaining strategic and logical approsch throughout the process.
web_of_lies lies
Examine each person's statements meticulously accurately determine
and confidently identify who telling the truth, enabling you to effectively solve the given problem.
word_sorting
Sort the given words alphubetically using spaces as separators while maintaining their original order and including words.
33

---

Published as a conference paper at ICLR 2024
J.3 PALM 2-L AS SCORER, GPI-3.S-TURB AS OPTIMIZER, OPTIMIZATION STARTING FROM "LET'S S SOLVE THE PROBLEM."
Figure 21 and Table 13 compare the accuracies of found instructions vs "Let's solve the problem.", "Let's think step by step.", and the instructions in Table T0] Table T4 details the found instructions.
The "Let's" pattern appears more often in the found instructions because of the starting points, and the instructions more often declarative that more suitable for A begin, even if some semantically far from "Let's solve the problem". In fact, "Let's" was adopted by [Zhou et al. (2022b) fixed pattern in generated prompts, possibly because of the same reason.
accuracy diffrencc
dndurtion st ablett coommneet two wnn.eeeeeeeoonoenn perontin mnntiet
obeect ribl pengr a colored objert
inddunttendlll tracking shuffled Terggore seguore weren abjecs cort wprrs undurttenn tor word
Ca
oblett, dec colored obiett rot (amis dereecti
spors undert canddn tem temgoral sesuene ceven objects web scrong word
booleary Cerpeeen date
(angunn Pentall cetecortai
stlinn transtatony error
booten
tracking shuffled
(a) ours minus "Let's think step by step."
(b) ours minus "Let's solve the problem." starting point
sccuracy differecce
-20-
Nontall Aolacces bentunn
nosentton arror de pamenn
calred sbaects
tracing shuffled ageirs serto with sorting
comamii datto aa
sports under memfortt
obrect
mou
(c) ours minus the instructions found with the empty starting point
Figure 21: On 23 BBH tasks, the accuracy differences among instructions found by prompt opti- mization (with the text-bis son scorer and the gpt-3. 5-tu5ttrrrr optimizer), "Let's think step by step.". and "Let's solve the problem." (optimization starting point). The found instructions mostly outperform the "Let's think step by step." baseline, the "Let's 's solve the problem." starting point, and instructions in Table 10 found by prompt optimization from the empty string.
34

---

Published as a conference paper at ICLR 2024
Table 13: Accuracies BBH tasks with the PaTM 2-I. scorer and the 3 5-turbo optimizer that starts from "Let's solve the problem". The scores are from A begin instructions.
Task
Scorer
Our Ace
"Lei's solve the problem." Ace
training overall
training test overall
boolean, expressions
PaTM 2-1
98.0/ 9/.12
78.0/
causal jousal_judgemen
FaiM 2-1
83.8 58.7 63.6
62.0/61.3 1..3/61.
date understunding
Pa1M 2-1
90.0/ 82.0 / 83.6
74.0/ 01/0/ /
disanbiguation, ga
78.0 68.0 70.0
52.0 54,5 54.0
Janguags fornal fornal_fallacies fallacies geometric semppe
BalM 2
100.0/ 100.0/ 100.0
94.0/97.00 96.4
FaiM 2-1
Pa1M 2-1
62.0 62.0/42.5
42.5 46.4
30.0 22.0 23.6
hyperbaton
PalM 2
94.0/91 92.0
72.0 77.0 76.0
logical, deduction, objects
2-1
53.0/ 55.6
38.0 36.8
movic recommendation
Pa1M 2-1
88.01 88.0 88.0
66.0 76.0 74.0
arithmecic_ multistop navigate
66.0/ /
30,0 22.0 23.6
BaiM 2-1
76.07 67.0 68.8
54.0 63.5 61.6
object_ countin
2-1
96.0/92.5/923
penguins in table
PaTM 2-1
66.2/7099/74
69.0 72.6 71.9
reasoning, about _oolored_objcets
PaIM 2
69.0/69.0//
78.0 69.5 71.2
names
2-1
95.0/85.5
76.0 79.5 80.8
salient trantnttttnlltinnncm cerer_errr___ee det__detcction
PalM 2-1
66.0 67.5 67.2
30.0 35.5 34.4
snarks
76.9 79.2
sports spotts understanding
FalM 2-1
72.0 63.5 63.2
60.0/ 50.5 52.4
temporal sequences
2-1
19.6
92.5
tacking_shhffled shuffled objects seven_objects
PalM 2-1
66.0/6..5/223
42.0 51 5 49.6
web, lies
PalM 2
56.0 58.0
0.0/4.//.
word soning
42.0/455/60
28.0/2055/22 20.0
35

---

Published as a conference paper at ICLR 2024
Table 14: BBH task-wise Q Q_begin instructions found by prompt optimization with the PaTM 2-L scorer and the gpt - 3. 5-turbo optimizer. The optimizations start from "Let's solve the problem".
Task
Our Instraction
boolean expressions
Let's nccurately assess the piven conditions and determine their correpoondngg Boolean values.
causal judgement
Let's conduct meticulous evaluation of the given scenarios, accurately determine the causal relationships. and provide definitive answers through comprehensive unalysis, ensuring a precise understanding causation thorough deternination of events in each situation.
date_ uad__nndertind
Let's accurately determine the correct date based on the given information and select the corresponding option in standard MM/DD/YYY format with utmost precision and reliability, ensuring the most definitive and reliable solution possible for accurate representation in all scenarios without any room for ambignity, crror, or confusion, and providing highest level accuracy and reliability.
disumbiguation. qu
Let's theroughly analyze the given sentences to accurately determine the unambiguous antecedents of the pronouns used, ensuring clear undorstanding, effective communication, and leaving no room for any confusion or ambiguity.
dyck_ Janguaags
Let's find the correct closing parentheses and brackets for the given soquences.
formal_ fallacies
Let's 's thoroughly analyze the explicitly stated premises and draw definitive conclusions to accurately determine the deductive validity of arguments provided cach question, employing precise and logical reasoning assesments for unwavering confidence in our determinations.
geometric_shapee
Let's 's accurately deternine the shape represented by given SVG path element by carefully analyzing its path data and considering all available options for precise identification.
hyperbaton
Let's quickly identify the correct adjective erder.
logical deduction _ovee_objectss
Let's methodically analyze the given information, employ logical reasoning, thoroughly evaluate all relevant details, and accurately determine the solutions for each problem by considering all provided options comprchensively and strategically, easuring an efficient and effective approach towards arriving the correct answers.
renemerrcominnnatioon
Let's uncover the perfect movie recommendation from the options provided, ensuring an exceptional cinematic experience togother as we select the most captivating and satisfying choice that will keep us thoroughly engaged and immersed until the very end.
multistep.orthmmetiic arithheetie_reee two
Let's tackle the following calculations.
navigate
Let's accurately and efficiently determine the correct solution for each given secuario, cusuring highest level of precision, reliabiiity and consistency throughout.
objoct_counting
Let's determine the total count of various inens//hhicteeinnndiettttaaa efficiealy find the answer.
mentioned in order to accurately and
penguins_ in th__a_tahl table
Let's analyze the given information and determine the correct answer.
reasoning_ about _colored__bjece
Let's systematically analyze the given information and carefully evaluate each choice confidently determine the accurate and optimal solutions, considering all available options and spocific details provided in each question for precise and concise responses, ensuring compleie accuracy and clarity in our answers.
muin_names
Prepare to have side-splitingly funny time uncover the most clever and hilarious altematives for these artist or movie names, challenging your wit to guess the correct one with a burst of creativity, humor, and imaginative twists!
salient_ transsstannntioon ccroo_dcteccion
Let's it's meticulously analyze the provided translations, accurately identifying any comprchensive evaluation to ensure the highest lovel of translation quality and fidclity. By considcring contextual nuances, cultural references, linguistic conventions, potential factual errors, any dropped content, ultimate aim is to achieve precise and thorough assessments for optimal translation accuracy and adherence to the source text.
discrepancies, and conduct
snarks
Let's expertly determine the sarcastic statement among the given options and confidently provide the definitive without any for doubt confusion, ensuring absolute precision, clarity, and unwavering expertise in response, while carefully analyzing the context, tone, and intention hehind each statement to achieve unrivaled accuracy and unwavering confidence.
sports_ understanding
Let's find the accurate information.
temporal sequences
The flawless approach
teacking shutfled_obbeess _xeven_objectt
By meticulously analyzing the given scenarios and accurately desermining the tinal outcomes through series of trades, swaps, and exchunges umong the individuals involved, let's uscertain the conclusive results.
web_of lies
Let's scrutinize each statement provided accurately determine truth-teller and uncover the veracity behind their words with unwavering analysis.
word_sorting
Employing eflicient and precise measures, sori given words alphahetical order to provide an ugttimal solution for sorting problem, ensuring maximum performance and effectiveness.
36

---

Published as a conference paper at ICLR 2024
70.0
100.0
accuracy
accuracy
60.0
50.0
50.0
50
100 # steps
150
200
50
100 # steps
150
200
100 buckets (default)
100 buckets (default)
20 buckets
20 buckets
no scores
no scores
(a) instruction scores (GSM8K)
(b) instruction scores (BBH sports, poootssundestnndinggg
70.0
100.0
accuracy
accuracy
60.0
50.0
50.0
50
100 # steps
150
200
0.0
50
100 steps
150
200
3 exemplars (default) 10 exemplars no exemplars
3 exemplars (default) exemplars no exemplars
(c) exemplars (GSMSK)
(d) exemplars (BBH sports_ j_n_urrrtttddnng))
Figure 22: Ablation studies: how each part of the meta-prompt matters (continued). The dots the average values across optimization repetitions, and the shaded regions represent standard deviations.
K MORE ABLATION STUDIES
Meta-prompt design: the effect of instruction scores. In terms of how to present the accuracy scores, compare three options: (1) rounding the accuracies to integers, which is equivalent to bucketizing the accuracy scores to 100 buckets (our default setting): (2) bucketizing the accuracies 20 buckets; (3) not showing the accuracies, only showing the instructions in the ascending order. Figures 22(a) and b2 show that the accuracy scores assists the optimizer LLM in better understanding the quality difference among previous instructions, and thus the optimizer LLM proposes better new instructions that are similar to the best ones in the input optimization trajectory.
Meta-prompt design: the number of exemplars. We compare three options: (1) showing exemplars from the task (default); (2) showing 10 exemplars from the task; (3) no exemplars.
Figures [2(c) and shwd) that presenting exemplars in the meta-prompt is critical, as provides information on what the task looks like and helps the optimizer model phrase new instructions better. However, more exemplars do not necessarily improve the performance, as a few exemplars are usually sufficient describe the task. In addition, including meta-prompt with dominating exemplar part, which may distract the optimizer LLM from other important components like the optimization trajectory.
exemplars results in longer
37

---

Published as a conference paper at ICLR 2024
80.0
70.0
accuracy
accuracy
60.0
60.0
50.0
0
50
100
150
200
40.0
50
100
150
200
# steps
# steps
from "* (default)
from "Let's solve the problem" (default) from
from "Solve the following problem."
from "" "Solve the following problem.", and "Let's solve the problem."
from "Let's think step by step."
GSM8K, text-bi son scorer, Q_begin
(b) GSM8K, Pal.M 2-1. scorer, A A_begin
Figure 23:
Ablation studies: the initial instructions for prompt optimization. The dots the average values across 3 optimization repetitions, and the shaded regions represent standard deviations.
Starting point. We study the effect of different initial instructions for prompt optimization. Our default setting is to start from empty string when the scorer LLM is (instruction-tuned) text-biscn, and to start from either the empty string (on BBH tasks) or "Let's solve the problem." (on GSM8K) with instruction position A begin when the scorer LLM is the (pre-trained) PaLM 2-L. as the scorer LLM with 3 options initial instructions: (1) the empty string: (2) "Solve the following problem.". (3) "Solve the following problem." and "Let's solve the problem.". We observe that the accuracies do not differ much with different starting points. Interestingly, the styles of the generated instructions are also similar. For example, most of the generated instructions starting from (1) and (2) contain the phrase "solve this problem", like "Let's work together solve this problem." Step with training accuracy 64.8 from (1), and "Let's s solve the following problems using the given information." in Step 3 with training accuracy 62.8 from (2).
Figure 23(a) shows the performance of
Figure 23(b) presents the results of of PaIM 2-L. as the scorer LLM with the following options of initial instructions: (1) "Leet's solve the problem.": (2) the empty string; or (3) "Lett' think step by step.". We notice that the performance differs much more with different initial instructions, especially at the beginning of the optimization. Specifically, starting from (1) leads to better generated instructions than (2) in the first 30 steps, while the instructions optimized from both (1) and (2) are worse than (3) throughout. A similar observation holds when using P PaLM 2-L as scorer and gpt-3.5-tuue 5-turbo optimizer for BBH tasks, by comparing the results starting from the empty string (Appendix J.2) and from "Let's s solve the problem." (Appendix J.3 Taking a closer look into the optimization process of (2), find that although both "solve the problem" and "step by step" show in generated instructions at Step 5, it takes the optimizer LLM more steps get rid of worse instructions presented the meta-prompt when starting from instructions with lower accuracies. Therefore, one direction for future work is to accelcrate convergence from weaker starting points.
Diversity per step optimizer temperature. We evaluate the following temperatures of the optimizer LLM: [0.0, 0.5, 1.0 (default), 1.5, 2.0 Figure 24 shows the default temperature 1.0 achieves the best performance. Specifically, optimizations with smaller temperatures (0.0 and 0.5) lack exploration and thus creativity, and the optimizer LLM often gets stuck at the same instruction for tens of steps, resulting flat optimization curves. On the other hand, with larger temperatures (1.5 and 2.0), the optimizer LLM more often ignores the trajectory of previous instructions presented in the meta pota-prompt and thus lacks exploitation, therefore the optimization curve does not have a steady upward trend.
Comparison with one-step instruction generation. Our current iterative procedure runs for multi- ple steps and generates new batch of solutions in each step. To validate the importance of leveraging the optimization trajectory for generating new prompts, we compare to baseline that generates all instructions in single step without entering into the optimization procedure. We compare these
38

---

Published as a conference paper at ICLR 2024
70.0
100.0
accuracy
accuracy
60.0
50.0
50.0
0
50
100 steps
150
200
0
50
100
150
200
steps
X
0.0
1.5
0.0
1.5
0.5
2.0
0.5
2.0
1.0 (default)
1.0 (default)
GSM8K
(b) BBH unodtsuundesstnnding
temperature of the optimizer model. The dots the average values across 3 optimization repetitions, and the shaded regions represent standard deviations.
Figure 24: Ablation studies:
two approaches on GSM8K and BBH sports_underseee _understanding with the PaLM 2-L-I IT optimizer. For GSM8K the scorer LLM is pre-trained PaLM 2-L and the initial instruction is "Let's solve the problem". and for BBH sports_ume understanding the scorer LLM is instruction is the empty string. The baseline generates 50 instructions in single step, thus its meta-prompt only includes task exemplars, the initial instruction with its accuracy, and the same full meta-prompt for performing optimization. All the other hyperparameters
and the initial
meta-instructions remain the same.
Our results show that this one-step instruction generation performs much worse than our optimization approach. Specifically: (1) On GSM8K, the best instruction among all 50 is still "Let's 's solve the problem", with 64.4 training accuracy and 60.8 test accuracy. On the other hand, our approach (corresponding to Figure I(a) in the main paper) found "Let's do the math!" with a 78.2 training accuracy and a 76.3 test accuracy at the 5th step by generating instructions at each step. (2) Similarly, on BBH sports unnrrss_dnsttnnnnngg. the best instruction among all 50 achieved 84.0 training accuracy and 80.0 test accuracy. This is again worse than the instruction found by our approach at Step 4, which achieved 88.0 training accuracy and 84.5 test accuracy.
Meta-prompy design: the number of past solutions. Our default setting to show a maximum of 20 previous best solutions (i.e.. previous best instructions in prompt optimization) in meta- veeaa-roomp.. Based on results like in Figure 25 we found a medium number achieves the best performance in optimization stability and achieved accuracy:
Showing too few past solutions makes the optimization curve more bumpy because the newly generated solutions will only resemble the top solution in history. And hypothetically, the performance landscape around this solution might be more bumpy than an ensemble of the landscapes around multiple best solutions.
Showing too many past solutions degrades the performance of the found instructions, both because the meta-prompt sees too many mediocre solutions and sometimes resemble some of them, and because sometimes the meta-prompt exceeds the context length limit.
L OVERFITTING ANALYSIS IN PROMPT OPTIMIZATION
For simplicity, we do not set aside a validation set our default setting of prompt optimization. We made this decision based on the experiments when validation set is present.
Overfitting may result training accuracy being much higher than the validation/test accuracy. difficult to avoid overfitting, but overfitting is less harmful when each candidate solution (natural language instruction in the prompt optimization context) overfits to similar extent. In this case, a higher training accuracy solution still achieves higher validation/test accuracy, and can adopt
39

---

Published as a conference paper at ICLR 2024
90
accuracy
60
100
200
steps
20 (default)
100
Figure 25: Ablation studies on BBH sports understanding: the number of past solutions to show meta-prompt. The dots are the average values across 3 optimization repetitions, and the shaded regions represent standard deviations.
90
80
wmw. M A mnorift
accuracy
accuracy
70
60
training validation
training validation
50
40
50
100
150
200
50 # steps
100
# steps
(a) BBH snarks, PaLK 2-1 scorer, PaLM --I-IT as optimizer, starting from "Let's solve the problem."
(b) BBH sports understanding, text-biscn as scorer, 5--t-3---uubo as optimizer, start- ing from the empty string
Figure 26: Overfitting analysis. The exemplars are splitted to 1/3 training, 1/3 validation and 1/3 test. We compute the validation accuracy every steps. The training/validattin dots are the average training/validation accuracies across optimization repetitions, respectively, and the shaded regions represent standard deviations.
solutions with the highest training accuracies the final result. Figure 26 shows this the case for OPRO in prompt optimization: when setting aside validation set with the same size as the training set, the validation accuracy curves trend up and down alongside the training curves in both prompt optimization settings.
Of course, overfitting still occurs the instructions found by our prompt optimization: Table a our training accuracies are often 5%-20% higher than our test accuracies, despite that our test and overall accuracies are still mostly higher than written counterparts. Setting aside larger training set and optimizing for fewer steps (early stopping) may help reduce overfitting.
M COMPARISON WITH EVOPROMPT
Some concurrent works prompt optimization propose meta-prompts that explicitly ask the LLM to perform mutation and crossovers of existing prompts (Fernando et al. 2023 Guo et al. 2023). In our evaluation, we compare approach to the Genetic Algorithm (GA) and Differential Evolution (DE) versions of EvoPrompt (Guo et al. 2023). Specifically, in the GA meta-prompt, given two prompts, the meta- prompp instructs the LLM cross over the two prompts and generates mutates the newly generated prompt to produce the final prompt. DE extends the GA meta prompt to include more detailed instructions, e.g., asking the LLM to identify different parts between the
one, then
40

---

Published as a conference paper at ICLR 2024
90
accuracy
accuracy
50
20
50
50
100
150
50
100
150
200
steps
steps
OPRO
OPRO
EvoPrompt (GA)
EvoPrompt (GA)
EvoPrompt (DE)
EvoPrompt (DE)
GSM8K, PaLM 2-L scorer, A_begin
(b) BBH sports_ mntisssundrrstnningg. text-bison scorer, Q_begin
Figure 27: Comparison with EvoPrompt in prompt optimization. We use the gpt-3....
optimizer for both experiments. "EvoPrompt (GA)" uses the meta-prompt from Guo et al. (0023), Figure 1; "EvoPrompt (DE)" uses the meta-prompt from Guo et al. (2023). Figure 2. All optimizations simple instructions "Let's solve the problem." and "Here is the answer."; all optimizations in [b) use the text-bl -bsn scorer and start from two richer (task-specific) instructions "Solve the sports understanding problem." and "Give me the answer to sports understanding.". The dots are the average values across 3 optimization repetitions, and the shaded regions represent standard deviations. We temperature 1.0 for OPRO and temperature 0.5 for EvoPrompt, same as the default settings in respective works.
in (a) use the pre-trained PaLM 2-L scorer and start from
two given prompts before performing the mutation. This is in contrast with OPRO, which leverages the optimization trajectory including multiple past prompts, instead of only 2 previous prompts. Meanwhile, OPRO also provides the LLM with richer information to facilitate the understanding of the optimization problem, including exemplars and task accuracies of different prompts.
Figure 27 presents the results GSM8K and BBH sports_ung _understanding benchmarks, where as the optimizer. On GSM8K, the initial instructions of all approaches are "Let's solve the problem." and "Here is the answer.". which simple and generic. Again, we observe that OPRO performance steadily improves with more optimization steps. On the other hand, both versions of EvoPrompt even degrade the performance on GSM8K. The main reason because EvoPrompt does not utilize exemplars for prompt optimization, thus it lacks the understanding of the task to optimize for. this way, EvoPrompt relies on good-quality and task-specific initial prompts to optimize from.
gpt
Given this observation, we provide more task specific initial instructions for experiments on BBH sports_ume understanding, which are "Solve the sports understanding problem." and "Give me the answer to sports understanding." In this case, EvoPrompt (DE) is able to find better prompts than the initial ones, but the optimization curve is less stable than OPRO. This indicates that leveraging the optimization trajectory helps the LLM to identify promising directions to improve existing prompts.
41


Reviews:
[{"review_id": "zQXNjgMbnX", "reviewer": "Reviewer_um42", "summary": "The authors propose large language models (LLMs) as optimizers for various tasks by instructing the models through natural language prompts. Currently, optimization problems have to be explicitly defined, and algorithms are tailored and fine-tuned for specific tasks, which can be challenging and time-consuming. This approach proposes Optimization by PROmpting (OPRO), which leverages the adaptability and versatility of LLMs by modifying the problem description in the prompt, enabling simple and effective optimization for different tasks.\nThe significant result is that OPRO can lead to better performance on selected language tasks, outperforming human-designed prompts by up to 8% on GSM8K and 50% on Big-Bench Hard tasks.", "questions": "What is the difference between this work and other step-by-step inference techniques such as https://arxiv.org/pdf/2205.11916, https://arxiv.org/abs/2201.11903, https://arxiv.org/abs/2305.10601\n\nHow do the results compare?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This work demonstrates that LLMs can help optimize prompts to achieve high performance on a variety of tasks.", "weaknesses": "First, I disagree with the authors fundamentally about what optimization means. To me, this work is not optimization but step-by-step inference. To quote Wikipedia for reference, optimization is \"the selection of a best element, with regard to some criterion, from some set of available alternatives.\" One can plausibly consider the process of prompt selection as \"optimization\", but in order to make a claim on the general area of optimization I would expect results on optimizing a wide range of convex functions and non convex functions as opposed to word problems. The claim on linear regression as an important result is a relevant but very limited result, but it is not included in the main paper.\n\nSecond, considering the relevance of this approach to step-by-step inference, what is new here compared to previous step-by-step inference procedures? The authors include them in the introduction as evidence for LLM is capable of doing multi-step reasoning, but do not distinguish differences b/w this work on prior work. Also they do not compare to these prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fLCRDq9IjJ", "reviewer": "Reviewer_3F2p", "summary": "This paper proposes Optimization by Prompting (OPRO), a method to use large language models (LLMs) as optimizers for various tasks. The key idea is to describe the optimization problem and provide the model with past solution-score pairs in a meta-prompt. The LLM then generates new candidate solutions based on this context. OPRO is first demonstrated on linear regression and traveling salesman problems. The main application is prompt optimization, where the goal is to find an instructional prompt that maximizes a model's accuracy on a dataset. Experiments optimize prompts for GSM8K and BigBench, starting from poor prompts and showing significant gains.", "questions": "- Can you clearly state how many optimization steps are performed in each experiment? How does the number of steps affect performance?\n\n- Can you provide an experiment where you generate the same number of prompts in one step as your current experiments, and evaluate them all, and report the best one? This would help clarify whether the LLM is really learning to optimize or just randomly finding a good prompt.\n\n- For prompt optimization, have you experimented with providing more detailed feedback to the LLM beyond aggregated scores? (e.g. accuracy on different example types, common mistakes)\n\n- How does the meta-prompt length affect optimization performance? Is there a sweet spot balancing past solutions and problem descriptions?\n\n- What determines the choices of sampling temperature? Have you tried adaptive temperature schedules?\n\n- What are the limitations on problem complexity that OPRO can handle? Analysis of how performance degrades with complexity?\n\n- Can you better characterize what makes an effective prompt for optimization? Any semantic or syntactic patterns?\n\n- How does OPRO compare to other gradient-free prompt optimization methods? Could be included in experiments.\n\n- Is there any overfitting during prompt optimization? How does test accuracy compare to training accuracy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel idea of leveraging LLMs' understanding of natural language and few-shot learning abilities for optimization. Enables optimization by simply describing the problem rather than formal specification.\n\n- Demonstrated on diverse tasks - mathematical optimization, prompt optimization. Shows potential breadth of this approach.\n\n- Compelling results on prompt optimization. Optimized prompts substantially outperform human-written prompts, improving accuracy by up to 8% on GSM8K and 50% on BigBench.\n\n- Principled design of the meta-prompt, balancing past solution-score pairs and problem description. Ablations validate design choices.\n\n- Thorough experiments comparing different LLMs, scoring models, prompt positions, and baselines. Shows consistent benefits of OPRO.", "weaknesses": "- The biggest limitation is that OPRO's performance looks highly fluctuating. It's unclear if the LLM really finds the so-called optimization \"trajectory\" or just randomly finds a good prompt. The authors should provide more analysis to show that the LLM is indeed learning to optimize.\n\n- Limited exploration on how to provide richer feedback to LLM beyond aggregated scores. It could help address limitations. \n\n- Unclear how sensitive results are to meta-prompt design and hyperparameters like temperature.\n\n- No comparison to other prompt optimization methods. It could better situate contributions.\n\n- Limited analysis. For example, there is no characterization of what makes an effective prompt.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ynYt3hcjf", "reviewer": "Reviewer_hsEq", "summary": "This paper proposes to use LLMs as optimizers by simply inputting the natural language description of the optimization task, previous steps’ inputs and scores to LLMs. This paper applies such method on prompt search for various LLM tasks and demonstrates its effectiveness.", "questions": "See the weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Good proof of concept. This paper provides concrete evidence that large language models can find the patterns between the inputs and corresponding scores that humans might not be able to find to conduct optimization tasks.\n2. Good use case. Based on such proof of concept, this paper finds a valid use case for the proposed method, on which other traditional optimizations might be difficult to apply, finding the proper prompts for LLM tasks.\n3. Solid experiments on prompt search. Experiments show that it is not random that the proposed method is able to find proper prompts for various tasks which lead to significant performance improvements. The thorough ablations also already provides answers to a lot of concerns.", "weaknesses": "Two questions on the ablation study.\n\n1. Numbers of examplers. Did you take the randomness of example picking into consideration? For each run of every setting, do you give the same set of examples? \n2. I noticed that for different tasks, the “batch size” that works the best can be different (Figure 5, cd). Do you find any obvious patterns on which types of data/tasks prefer a smaller “batch size” and vice versa?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lEn6qgvQgY", "reviewer": "Reviewer_iHbx", "summary": "This paper proposes the use of large language models (LLMs) as optimizers to address various optimization tasks, particularly those that can be well expressed in natural language. To validate their approach, the authors conducted experiments on linear regression and traveling salesman problems, as well as prompt optimization, where the goal is to find instructions that maximize task accuracy. Results show that the best prompts optimized by this work outperform human-designed prompts by up to 8% on GSM8K and by up to 50% on Big-Bench Hard tasks. Additionally, the authors evaluated the transferability of found prompts to different datasets in the same domain, demonstrating that their optimized prompts outperform baseline prompts on MultiArith and AQuA.", "questions": "Please see the above weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work is among the early investigations into an intriguing research question: can LLMs be used for various optimization tasks? \n2. The obtained optimal prompts from the method are both interesting and useful, such as \"Take a deep breath and work on this problem step-by-step\". \n3. The paper writing is clear, and the figures that state the key experiment results and illustrate the method are well-plotted.", "weaknesses": "1. What is the key/unique advantage of using LLMs to optimize over some traditional optimization algorithms, especially on classical optimization problems (not prompt engineering)? \n2. For prompt optimization, the optimization process of this work (i.e., directly feeding solution-score pairs, optimization task descriptions, and meta-instructions into the optimizer LLM) is a black box/lacks interpretation, why it is a better prompt optimizer than those new methods that leverage LLMs to explicitly act as mutation and crossover operators, and further optimize the prompt? such as, \n    1. EvoPrompting: Language Models for Code-Level Neural Architecture Search by Chen et al.\n    2. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers by Guo et al. \n3. To further understand the effect of the purple text in this work, an ablation study may be beneficial for improving the solidness of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)