Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization
Authors: Xingchen Wan
Ruoxi Sun
Hootan Nakhost
Sercan O Arik
Venue: NeurIPS 2024 poster (2024)
Keywords: instruction optimization
prompting
in-context learning
Primary area: natural_language_processing

Abstract: Large language models have demonstrated remarkable capabilities but their performance is heavily reliant on effective prompt engineering. Automatic prompt optimization (APO) methods are designed to automate this and can be broadly categorized into those targeting instructions (instruction optimization, IO) vs. those targeting exemplars (exemplar optimization, EO). Despite their shared objective, these have evolved rather independently, with IO receiving more research attention recently. This paper seeks to bridge this gap by comprehensively comparing the performance of representative IO and EO techniques both isolation and combination on a diverse set of challenging tasks. Our findings reveal that intelligently reusing model-generated input-output pairs obtained from evaluating prompts on the validation set as exemplars, consistently improves performance on top of IO methods but is currently under-investigated. We also find that despite the recent focus on IO, how we select exemplars can outweigh how we optimize instructions, with EO strategies as simple as random search outperforming state-of-the-art IO methods with seed instructions without any optimization. Moreover, we observe a synergy between EO and IO, with optimal combinations surpassing the individual contributions. We conclude that studying exemplar optimization both as a standalone method and its optimal combination with instruction optimization remain a crucial aspect of APO and deserve greater consideration in future research, even in the era of highly capable instruction-following models.

Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization
Xingchen Wan, Ruoxi Sun, Hootan Nakhost, Sercan Ö. Arik Google Cloud AI Research (xingchenw, ruoxis, hootan, soarik)@google.com com
Abstract
Large language models have demonstrated remarkable capabilities but their per- formance is heavily reliant on effective prompt engineering. Automatic prompt optimization (APO) methods are designed automate this and can be broadly categorized into those targeting instructions (instruction optimization, I0) vs. those targeting exemplars (exemplar optimization, EO). Despite their shared objective, these have evolved rather independently. with IO receiving more research attention recently. This paper seeks bridge this gap by comprehensively comparing the performance of representative IO and EO techniques both isolation and combina- tion on a diverse set of challenging tasks. Our findings reveal that intelligently reusing
input-output pairs obtained from evaluating prompts on the validation set as exemplars, consistently improves performance on top of IO methods but is currently inver--nvesttigaee. We also find that despite the recent focus on IO, how we select exemplars can outweigh how we optimize instructions, with EO strategies as simple as random search outperforming state-of-the-art IO methods with seed instructions without any optimization. Moreover, we observe a synergy between EO and IO, with optimal combinations surpassing the individual contributions. We conclude that studying exemplar optimization both as a stan- dalone method and its optimal combination with instruction optimization remain a crucial aspect of APO and deserve greater consideration in future research, even in the era of highly capable intructtion-oolowing models.
Introduction
76.1
Optimized instruction + Optimized exemplers
Significant advancements in large language models (LLMs)) have revolutionized various natural language pro- cessing tasks [9, 4, 15). One notable aspect of I.LMs, however, is their sensitivity to the input "prompts," which has given rise to the burgeoning field of prompt engineer- ing [26, 38]. On black-box LLMs where we can neither modify access internal parameters, prompt engineer- ing involves crafting input prompts that effectively guide LLMs generate desired outputs. Starting from manual processes requiring human expertise, the complexity and volume of prompts have necessitated the development of automatic prompt optimization (APO) methods aiming to streamline and automate prompt generation, thereby alle- viating the burden of manual intervention. Broadly, since prompts consist of instructions and exemplars, we may roughly categorize APO into instruction optimization (IO) and exemplar optimization (EO) approaches. IO focuses 38th Conference on Neural Information Processing Systems (NcurIPS 2024).
Optiiiing sg
72.0 exomplas only
Optimized examplars
cutow-form
Optimized Instruction + Random eammplers
70
eodsting IO
Moat
681
works
Optimized Instruction exemplors
60.3
Initae inste uction No scam plars
accuracy
Figure 1: Average performance over >20 tasks on PaLM 2 We compare and combine APO targeting exemplars and instructions, and find that how we opti- mize exemplars (orange) can eclipse how we optimize instructions despite current research favoring the latter (blue and pur- ple), whereas optimizing both is the best (cyan) within similar budget.

---

on refining the textual instructions provided to LLMs that contain task- information (i.e., to teach). whereas EO emphasizes the selection of relevant examples to guide model behavior (i.e., to show). Partially driven by the improved instruction-follonn followigg ability of LLMss the research attention has increasingly shifted towards IO, especially using L.LMs themselves as optimizers [66, 35, 48].
While EO and IO approaches address the similar overarching problem, they have evolved somewhat independently, with a few exceptions [14, 47]. Indeed, as we elaborate in $2, EO approaches are often based on simple, handcrafted templates without explicit instruction optimization [21, 45], while 10 methods seldom optimize exemplars and often rely on random validation set samples [35], require additional fixed exemplars on top of the validation set [66, 16], or consider the "zero-shot" setup with no exemplars at all [48]. Whereas the lack of 10 in EO methods is somewhat understandable as many EO approaches predate instruction finetuning [49] and, subsequently, instruction coiructionfooloiinnn models that are sensitive to instructions, the inverse is much less so: concretely, almost all existing IO approaches already require a labeled dataset as the validation set, and are therefore, by definition, not "zero-shot". With the inputs, labels, and, if applicable, model-generated intermediate outputs (e.g.. reasoning steps) the subset of the validation set that the model has answered correctly, we already have a set of exemplars as a free side-product whenever we perform IO, independent from and on top to any additionally provided, human-annotan EO, such as mentioned in Pryzant et al. [35], is the goal to focus on one objective at a time'. However, given the common practical goal of and the interplay between EO and IO [30], we argue they should not be treated separately - it is instead critical to understand their relative importance and combined impact, and, where necessary, optimize them jointly for the best performance-cox balance.
exemplars. A common argument for not focusing on
This is, to our knowledge, where there is a gap the literature that we aim to bridge. To do so, on diverse suite of challenging BIG-Bench and MMLU tasks, we compare the performance gain brought by various representative, state-of-the-art (SoTA) 10 and EO methods on the fairground with PaLM 2, Gemini (1.0/1.5) and GPT models to foster better scientific understanding of different APO techniques. While IO comfortably improves the baseline prompts before any instruction or exemplar optimization, this, at best, portrays an incomplete picture. Under the same setup, with simple yet effective EO methods on the gener-geeeeeted exemplars on the validation set, we show:
Intelligently incorporating exemplars generated by the target model itself on the validation set significantly and consistently improves performance on top of recently proposed IO methods;
The performance gains realized by choosing appropriate exemplars via methods as simple as random search can eclipse the improvements brought by SoTA instruction optimization. As a concrete example, as shown in Fig. 1, with a simple optimization routine exemplars, seed instructions before any optimization outperform optimized instructions obtained with complex IO but with random exemplars most commonly used.
There exists a synergy between EO and IO, and optimally mixing-anddmatchiin 10 and EO is greater than the sum of its parts under comparable computational budget.
SoTA 10 might be itself implicitly generating exemplars, and these exemplars, while somewhat unintentional, contribute more to the performance than the rest of the instruction that I0 methods are meant to optimize.
While arguably receiving less research attention recently, exemplar optimization remains crucial design consideration in APO. Even in an era of highly capable instruction-followinng following LLMs, the significance of exemplar optimization should not be relegated an afterthought, and better exemplar optimization both as a standalone tool and as a combinable component with IO is crucial for APO.
2
Preliminaries
Prompts are natural language inputs to LLMs. Denoting an input task query as x. a few-shot prompt P(x)) may be represented as P(x)= = [1, ei, xr] where I denotes an instruction and {e1, e denote k exemplars (or interchangeably, demonstrations)) each of which is a concatenation of other queries and their outputs (including both the final answer and any possible intermediate outputs) which resemble the current query z or may otherwise guide the LLM to better handle the current task, and we show a common prompt template organizing these components in Fig. note that not all components are required: c.g., zero-shot prompts feature no exemplars.
I "The proposed algorithum is about optimizing the language of prompts, as opposed to selecting the best examples for few-shot learning."
2

---

Automatic prompt optimization (APO) aims to automatically design P(z) via optimization. We broadly consider black-box API-only LLMss The proposed framework assumes a validation datasel Daal= == {< Ut) Fe where T2 and y/ represent validation inputs and targets, a performance metric (e-g., accuracy) g(::)) ,), and aims to find the optimal prompt P (x) to be used at test time, which is empirically the maximizer on Dval:
Solve the following math problems by following the steps:
1. When multiplying or dividing two negative numbers, the result will be positive.
2. When multiplying or dividing a positive and a negative number, the result will be negative.
3. When adding or subtracting negative number, is the same as adding or subtracting its positive counterpart.
('(x)= = arg max D(a (9 (P()))(())) P(-N
(1)
where fLLM(-) denotes a textual LLM output given input and denotes the search space, whose definition allows broad cate- gorization of APO methods into instruction optimization methods targeting instructions in Fig. 2, exemplar optimization methods tar- geting exemplars in Fig. 2 and approaches that tackle both.
((6 5- -3 -7+ -2--7)
(6-0 -3) (6- -2-
3 (13 + -2 1818 18 <answer-54clanswer>
Exemplar optimization (EO). Efforts to optimize exemplars started soon after the discovery of in-context learning (ICL) [6] via retrieval-based approaches to identify the closest labeled examples [28, 61, 51, 64], influences and sensitivity [31, 81. and learning- based approaches [58, 55]. Toolkits like DSPy [21] adopt EO as the main APO paradigm. Works [45, 46, 60, 22, 41, 59] have also extended EO to model-generated exemplars in L.MS and multimodal models. Lastly, by framing EO from an active learning angle, Margatina et al. [29] provide a comprehensive understanding and comparative analyses. These works, however, principally analyze different EO strategies only, nor do they analyze from the angle of APO. Many of these works also primarily focus on and draw findings from earlier and simpler tasks that are arguably less challenging SOTA LLMs.
(1-
1.8-900-1=0+1+1
2-9--7-------47- 144
<answer-15-lanswer>
Figure 2: An example prompt: instruction I describes the task; exemplars (e1,..., ex. k
in the figure) pro- vide demonstrations and en- able ICI.; both are prepended to the query x before receiving the LLM responses.
Instruction optimization (IO). On black-box LLMs. the origin of IO may be traced to discrete prompt search [12. 40, 53, 62, 34] which prepend optimized tokens. which can be viewed as form of "instructions", to inputs. However, these approaches do not necessarily yield interpretable prompts and most of them require output signals (such as logits) beyond strictly black-box access. Thus, recent advances have shifted towards utilizing an LLM itself to generate natural language instructions for iterative optimization on Dval in Eq. 1. The seminal works is APE [66], which employs the LLM to iteratively cull top-performing instructions on Dval and paraphrase them until convergence. Similar evolutionary frameworks are widely used in follow-up works [16, 19, 63] and alternative formulations like Bayesian optimization (BO) [7] and neural bandits [25] were also used. Another line of works [35, 48, 56, 42] employ reflection, directing LLM to articulate reasons for errors to iteratively improve instructions. Other approaches like OPRO and its variants [54, 27]. treat the LLM as a black-box optimizer, tasking it with generating new instructions based on the trajectory of previously evaluated instructions and their performances without explicit intaainnstuuttons.
Combining EO and IO. As discussed in $1, there is relative dearth of work combining EO and I0 despite their shared objective. Specifically, even when the labeled dataset Dval is a prerequisite of virtually all IO methods, it is primarily used to estimate the expectation in Eq. I only rather than to construct exemplars in a principled way: For instance, ProTeGi [35] randomly samples exemplars from V while OPRO [54] uses them only for instruction induction [18]. Other works [16, 66, 48]
either use no exemplars or fix exemplars and only optimize the instructions - for challenging reasoning tasks, these methods require numan-annotater in addition to Dvals which arguably runs counter to the goal of automatically designing prompts without human intervention in APO. A few exceptions exist: PromptBreeder [14] employs "context shuffling" to co-evolve exemplars and instructions, while Mixture-of- Prommts (MoP) [47] aligns exemplars with multiple prompting "experts" for joint optimization. However, these works still focus their optimization effort on instructions: PromptBreeder emphasizes complex mutation operators for IO while providing only basic EO frameworks, whereas MoP chiefly focuses on 10 with the bulk of its contribution being assigning optimized instructions to different exemplar groups, rather
chain-of-thouse (CoT) exemplars [50]
ii.. only textual outputs are available; parameters, gradients, and intermediate outputs like logits are neither modifiable or accessible as of June 2024, many SoTA models like Gemini [15, 36] and the most advanced variants of GPT-4 [1] fall into this category.
3

---

than optimizing the exemplars themselves. Other works [13, 57] also include both exemplars and instructions in the search space, but they require information beyond strictly black-box outputs to some extent. Lastly, several works have analyzed the interplay between ICL. and instructions [30] or prompt templates [39], but they mainly characterize the performance variation as an issue deserving attention. We, however, consider the APO setup specifically, and argue that such an interdependence presents an opportunity through holistically considering instructions and exemplars. Concurrent to our work, Agarwal et al. [2] and Opsahl-One et al. [32] also study the joint optimization of instructions and exemplars, and in many cases reached conclusions corroborating our findings, demonstrating the community's ' growing awareness on the importance of the subject of focal interest to this paper.
3 Understanding Instruction Optimization and Exemplar Optimization
While studying 10 and EO independently has academic value, the practical goal ultimately for both is to optimize the performance of LLMs. Hence, I0 and EO, as two dominant genres of APO methods, present practitioners with the challenge of selecting or combining them to maximize cost-performance benefits. We aim to meet this by evaluating EO and IO in the context of APO by answering the following: 1) What is the relative importance and performance impact of EO and IO, both in isolation and when combined together? 2) How do we make the optimal use of the limited data and computational budget under the current APO framework?
3.1 Experimental Setup
We perform thorough experiments employing various EO and IO methods individually and in combination. We use the PaI.M 2 text- text-bison-002 [4] and Gemini 1.0 Pro/1.5 Flash [15, 36] as the target models, but we will also validate key findings on GPT-3.5. Modern IO methods often employ another, usually more potent optimizer model for to generate and/or critique instructions; we use PaLM text -ortt-n000 (for text-bison target model), Gemini 1.0 Ultra (for Gemini 1.0 Pro target model) or Gemini 1.5 Pro (for Gemini 1.5 Flash target model). We evaluate on tasks selected from
Hard (BBH) [43], collection of diverse tasks considered to be challenging to LLMs - the suite itself and datasets of similar task types are frequently used in many recent APO works [48, 54, 19. 14. inter alia]: the tasks include numerical reasoning, commonsense problem- solving, logical deduction, linguistic manipulation, machine translation, and tabular reasoning, among others. For all tasks, we use 20% of data as validation set and the remaining 80% for testing, the latter of which is held-out and unavailable to the L.M at search time (see App. A for implementation details). We also test some of our key findings on the MMLU benchmark [17], a set of 57 tasks frequently used to gauge the general problem-solving abilities of LLMs we use the official val and test splits for validation and testing, respectively. We consider the following IO strategies:
1. No IO: we use the seed instruction Io "Let's 's think step by step." [23] without any optimization.
2. APE [66] is the seminal work for LL-M-as-a--instruccii -nninnsruuttion--ee -pttmmize and uses an evolutionary algorithm design: at each iteration, we evaluate a population of instructions on the validation set and the optimizer is asked to generate a new population by paraphrasing the top-performing instructions. This process iterates until convergence.
3. ProTeGi [35] collects samples that the target [.LM answers incorrectly on Dyat under the current instruction and directs the optimizer LLM to reflect and critique it. The optimizer model is then asked to update the instruction by summarizing and abstracting the feedback. Additionally, at each iteration, ProTeGi also paraphrases instructions similar to APE (referred to as "Monte Carlo sampling") and uses beam search to identify the most promising instructions.
4. PromptAgent [48] is similar to ProTeGi but it features a more advanced planning agent using Monte Carlo tree search [10].
5. OPRO [54] implicitly optimizes instructions. Starting from the seed instruction, at each iteration, OPRO provides the optimizer model a concatenation of previously evaluated instructions and their validation scores. Instead of explicitly requesting paraphrasing or reflection, OPRO treats the optimizer L.M as a black-box optimizer and simply asks the optimizer model to "come up with a better instruction" given these information.
The above methods are selected as each of them represents the state of the art of a genre of approaches as outlined in $2 and collectively represents 10 techniques whole. We initialize each method at the seed instruction and ensure they consume the same amount of compute measured by the

---

12%
25
25
MC
20
20
20-
122%
19%6
-150
192
Task indox
Taskindex
45%
Taok inkindox
4 4%%
325%
+2%%
+28%
20%
100%
10
+35%
10
+28%
10
+0%
5
5
30%
65.0
25
50
75 100 25 50 75
100
25 50
75
100
Test accuracy
Test eccurecy
Test accuracy (%)
Figure 3: Appropriate EO improves over any or no I0: Task-specific BBH performance with no instruction optimization (left) and with SoTA 10: APE (middle) and ProTeGi (right) before and after applying exemplars found via Mutation ($3.1) on PaLM 2. Dashed and solid lines denote the average performance before and after exemplars, respectively. Task index is determined by the ascending order of test accuracy under seed instruction. Refer to additional visualization in App. B.3.
number of prompt evaluations on Dval m (we cap m = 32 except for "No IO" which requires no iteration). We also compare against PromptBreeder [14] in later section, as it features a much more expansive search space and requires significantly more than 32 iterations before convergence. After obtaining the optimized instruction I' (or Io if no 10 is performed), we perform EO. At this point, we emphasize that our setup should not be confused with the "few-shot" setup considered by some prior works [66, 16] which require additional human- annotated exemplars with reasoning traces to elicit CoT behavior. We perform EO only from the exemplars self-generated by the target model (also referred to as "bootstrapped few-shot" in DSPy [21] and "reinforced ICI," in concurrent works like Agarwal et al. [3]) and do not assume exemplars are given al the start of APO (i.e., we do not assume the presence of initial {ei....,ee e in P(x)). We consider the following EO strategies:
1. No EO: no exemplars are used; this is typically referred to as "zero-shot" in the APO literature.
2. Random: we randomly sample k input-output pairs from De(I") S D the subset to the validation set that the target LLM predicted correctly under I* and the output in this case includes any intermediate output the LLM generates before the final answer.
3. Nearest: We use the same DI( as above, but instead of sampling randomly, we retrieve top-k input-output pairs whose inputs are most similar to the current test input based on text embedding cosine similarity. We use the Gecko embedding [24].
4. Diversity: We use DT( but select the k input-output pairs closest to the centroids via k-means clustering, similar to the approach in Zhang et al. [60] to promote diversity in exemplars.
5. All exemplars (Gemini 1.5 target models only): With the advent of long-context models like Gemini 1.5, we may also fit the entire set of D. into the context and perform no selection at all.
The above heuristic- based EO strategies do not use D whereas aptimizatiion-aasee EO can utilize it similarly to IO. Instead of generating instructions, we select the exemplar combinations with the highest validation accuracy for testing [33, 20, 21]. Unlike I0, which creates new instructions via an optimizer model, EO selects from pre-generated outputs and does not require an optimizer model. Formally, we focus on optimizing exemplars conditional on 1" from 10 (or To if no 10 is involved)?:
E" {e arg max, of EE E B(G.n)) ( [f
(2)
s.t. I" = arg max IEI
B( Dea (T,LM 2;)
We include the following optimization-basee EO methods that differ in search strategy:
6. Random search: Following the EO procedure DSPy [21], we randomly sample m combinations of k exemplars: {Ei,... ..., Em} where each Es = {e V E {l,..., m.. We evaluate each combination on the validation set and use the best for testing.
7. Mutation: We also implement mutation-based baseline, initiating with population of Q combinations for T = m/Q generations, where Q = S. Each generation starts with a randomly
'We performed EO after 10 to optimize the exemplars generated by the best instruction; we also tested the inverted order (i.e.. EO before 10) and interleaved optimization (detailed in App. B.8) and found the results to be largely robust to these design choices.
5

---

Table I: Average BBH accuracy of all 30 EO-IO combinations with PaLM 2 (text-bison-002) target model and PaLM 2 (text-uni comncooo-000 0001) optimizer model. The last row/column show the max improvement over the No 10 and/or No EO baseline of the respective row/column. The background shades indicate cost in terms of prompt evaluations on Dval by the target model: gray cells requires evaluation
Dval (m = 0): blue cells perform iteratively optimize instructions or exemplars; orange cells iteratively optimize exemplars m times on top of optimized instructions.
32 evaluations to
Exemplar optimization (EO)
Max A over No EO
No EO
Random
Nearest
Diversity
R.S.
Mutation
Instruction Optimiza- tion (10)
No 10
60.30
66.91
66.09
66.74
71.16
72.92
+12.63
APE
64.96
69.11
69.01
70.81
75.88
76.25
+11.28
ProTeGi
68.13
70.81
70.01
69.25
75.90
77.29
+9.16
ProriptAgent
65
67.65
67.82
67.35
72.51
72.77
+7.11
OPRO
63.04
68.50
68.33
67.57
73.02
73.06
+10.01
Max over No
+7.83
+3.89
+3.92
+4.07
+4.74
+4.37
Table 2: Average BBH accuracy of seed instruction (No 10) and ProTeGi (best IO strategy from Table 1) with different EO strategies using Gemini 1.0 Pro target model and Gem- ini 1.0 Ultra optimizer model. Refer to Table I for further explanations.
Table 3: Average MMLU accuracy
of No 10 and ProTeGi with different EO strategies with
target model and text- eext-unicory optimizer
model. See App B.2 for Gemini results.
No E0
Random
Nearest
Diversity
R.S.
Mutation
E0
No EO
Random
R.S.
ABO
63.14
71.12
75.77
+12.63
No
65.77
72,06
72.75
+6.98
ProTeGi
65.91
72.72
72.13
72.64
78.27
79.01
+13.10
ProTeGi
69.73
70.82
72.31
+2.58
A
+2.77
+1.60
+2.95
+4.83
+2.50
+2.52
10
+3.96
-1.24
-0.44
Table Average BBH accuracy of seed instruction (No I0), APE and ProTeGi (top IO strategies from Table 1) with different EO strategies using Gemini 1.5 Flash target model and Gemini 1.5 Pro optimizer model. Refer to Table for further explanations.
No EO
Random
Nearest
Diversity
All
R.S.
Mutation
A AN
10
75.07
80.02
81.71
81.52
80.43
83.25
82.42
+8.18
APE
77.52
81.20
83.71
81.55
81.20
85.04
84.76
+7.54
ProTeGi
80.39
82.40
82.61
82.29
83.52
84.47
84.49
+4.10
1O
+5.32
+2.20
+2.00
+0.77
+3.09
+1.9
+2.34
initialized first generation, similar to random search. For subsequent generations, we populate with mutations of the best-performing combination from the previous generation, E Each mutation involves swapping one exemplar for another input-output pair from De.
For all EO methods except for "All exemplars", we use m 32, k = 3 for all main experiments, but we also test with h = (1,3,5 5, 10, 20) in App. B.4. Given the 10 and EO strategies, we experiment on each of the 1O-EO combinations.
3.2 Results and Analyses
On BBH, we aggregate the results in Table for PaLM models. We also experiment on Gemini models with no 10 and with ProTeGi (best overall 10 technique from the PaLM 2 results) in Table 2 (Gemini 1.0) and Table 4 (Gemini 1.5). We also validate key findings in representative datasets on GPT-3.5 (Table 13, App B). On MMLU, we present results in Table 3 (PaLM 2) and App. B.
(Gemini). Below, we highlight and discuss the key insights.
Insight 1: We should almost always perform EO whenever we perform APO.
One of the immediate findings from the tables that comparing the first column against others, any EO consistently improves test performance, with any or no instruction optimization. In Fig. 3, we further show that the EO not only benefits at an aggregated level but also leads to significant and almost unanimous improvements across diverse tasks.
While exemplars improving performance may not seem surprising, is worth noting that as mentioned, in this case they are iid-proodcct of evaluating instructions on Vval with no additional data costs. Thus, we argue that for practical purposes under the framework of Eq. 1, barring unusual constraints

---

(and task-specific PaLM 2/ BBH
Gemin TO Pro/ BBH
Gomini 1. Flach BBH
PRLM 2/ 2/MLLU
Gamiii 1.0 Pro (MMLL
1126
90-
80
VarTest
143
70
0.07
8D-
204
1.22
Bina Besl IO Bust ES
Base
Be Best ES
Bose
Best O Best Uade
BCst Best
Base Besi ID Best ES
Figure 4: Optimized exemplars generalize better than optimized instructions. Comparison of vali- dation accuracy and test accuracy over different model-task combinations. The generalization gap, which is the difference between validation and test accuracy, is marked on each figure. The better generalization of EO is exemplified by the smaller generalization gaps in all cases studied.
like extreme restrictions in context length, which This might restrict applicability of IO methods too as many So'A 10 methods also generate long prompts, and/or extreme long-contex: tasks where it is not possible to fit exemplars in the context window, there is little incentive to consider the "zero-shot" setup without exemplars and little incentive not to perform EO, given that current APO setup requires Daal anyway, regardless of whether we use them as exemplars. Thus, is by definition, not "zero-shot" and is not directly comparable to true zero-shot methods requiring no labeled data. Furthermore, there is also the risk that "zero-shot" results neither reflect nor accurately predict the full potential of the LLM, what performs well under zero-shot does not necessarily performs well when a better EO strategy (e.g., PromptAgent in Table I and ProTeGi in Table 3) is used. Lastly, since obtaining labeled data can be costly, intelligently reusing them as exemplars also represents more judicious use of scarce resources compared to only using them to evaluate a metric for optimization.
How we select exemplars may outweigh how we optimize instructions, and selecting exemplars via iterative optimization consistently outperforms alternative strategies.
Insight 2:
Exemplar optimization outweighs instruction optimization. Despite the recent focus the commu- nity places on IO, we find that how we select exemplars outweighs how we optimize instructions in the model-task combinations we investigate. With reference to Tables breakdown in Fig. 5), we find that if we optimize instructions or exemplars (i.c.. the blue cells) under roughly compute-matched budget, prompts without instruction optimization but with optimized exemplars (e.g., the "No 10 + Mutation" combination) outperform prompts with SoTA instruction optimization but without optimized exemplars (e.g., the "ProTeGi + Random" combinations) in an overwhelming majority of cases. In fact, on a separate set of experiments performed on the PaLM 2 models, we find this to be true even after halving the evaluation budget of EO (see App. B.6), and optimization on exemplars as naive as random search can outperform 10 methods that are significantly more complicated and expensive. Further substantiating this argument are that:
1) In isolation, EO boosts performance more effectively than IO: for example, with reference to Table 2, compared the seed prompt, using the best EO strategy (Mutation.fire row) alone increases the average performance by >11% compared to approximately 8% using the best 10 strategy (eri. first column);
2) When combined, benefits of EO and 10 stack up but are largely attributable to EO: under "Mutation" (last column), the best EO strategy, the performance gap between the best and worst I0 strategies shrinks to less than 4%, suggesting that instructions might be less critical if the LLM is provided with good exemplars after all.
We observe similar conclusions for different models and task combinations. In fact, on MMLU (Table 3) featuring much smaller validation splits, we observe that judicious exemplar optimization completely eliminates the performance gap caused by 10 under zero-shot, with No 10 even surpassing SoTA IO. Interestingly, as we show in Fig. 4 where we further consider the difference between validation accuracy, which is the empirical objective function in Eq. 1, and the test accuracy, which the reported metric that represents the generalization power of the optimized prompt, optimized exemplars generalize better than optimized instructions under all model-task combinations considered. On MMLU tasks (two rightmost plots in Fig. 4), IO even improves validation performance comparable to or better than EO, but the validation improvement does not generalize to the test set. These imply that the superior test performance of EO cannot be solely attributed to more effective search space P or optimization strategy in Eq. 1, and the performance gap might not be completely closed by advancing optimization only.
Optimization-base based EO outperforms heuristics. Between the different EO strategies, we find that optimization-bant based EO vastly outperform the alternatives: e.g. in all tables, ProTeGi with optimized
7

---

10:
ProteGi
ES
Randam
Mutation
Test acc. Test acc.
boclean,
dsantiguation
disambiguationn
formal falaccees
jogical deduction
deduction (3)
90
70
70
60
80
70
80
60
60
50
50
70
navigate
rerooiing about colored odjects
snarks
sports understanding
75
trocking shuflled objects 15)
75
tracking shuffied osjects (5)
70
90
60
70
80
80
50
50
70
25
25
Figure 5: tass-specffic BBH performance of selected IO-EO combinations with PaLM 2 (first 12 tasks; refer to App. B.3 for all other tasks/modells. Note that 1) Proper EO almost uniformly improves performance and 2) With appropriate exemplars, seed instructions with no optimization (third bar from the right) can often perform on par better than SoTA IO but with standard random exemplars or no exemplars commonly used in the literature (first six bars in each figure).
(a) web of lies (55%- 97%)
(b) trackingg wwwfssshuffeeeeeeeo _bbjectssssssseeeee (7) (38% 86%)
Maybelle tells the truth. Antwan says Maybelle tells the truth. Audrie says Antwan truth. Ryan says Audrie lies. Delfina says Ryan lies. Does Delfina tell the truth?
G: Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude dancers square dance. At the start of song. they each have a partner. Alcc is dancing with Ophelia, Gertrude is dancing with Karl.
dencing with Melisse. (text amitted),
Throughout the song. dancers often trade partners. First. Dave and Claire switch partners. Then, Alice and Eve switch partners.... (text
A: Let's think step by step.
Maybelle tells the truth.
omitted) the end of the dance, Fred is dancing with
Antwan says Maybelle telk the truth. Antwan tells the truth.
Options:
(A) Ophella
Audrie says Antwan tells the truth So Audrie tells the truth.
(text omitted)
(G) Karl
Ryan says Audrie lies. So Ryan
Delfina says Ryan lies. So Delfina tells the truth. <answer- cansser-Truu-dannswe Trueclanswer>
A: Let's follow step-by-step process. Let's follow the steps one by
(c) snarks (73%- 87%)
1. Dave and Claire switch partners.
Dave was caacinn with Sam,, and Claire was dancing with Jamie.
Q: Which statement sarcastic?
now. Dave is daacing with Jamie, and Claire is dancing with Sam.
Options:
Alice and Eve switch partners.
(A) If it's Britannica it must be True..
Alice was dancing with Ophelia, and Eve was dancing with Patrick. So now, Alice dancing with Patrick, and Eve dancing with Ophella.
(B) If it's on reddit it must be True..
A: We should move forward in stages.
At the end of the dence, Fred is dancing with Melissa. canswero(B)odansweee
<answer-(BB-llnnweer
Figure 6: "Winning" exemplars that led to exceedingly high performance: (a, b) LLM improves by almost 50% from imitating and chaining the patterns in the optimal exemplars. (c) When CoT hurts performance, optimal exemplars encourage LLMs to override instructions and answer directly. Refer to App. B.9 for examples of LLM responses when these exemplars are applied.
exemplars outperforms random exemplars, which is the default design in Pryzant et al. [35], by more than 6% in both Table 1 and 2 and more than 2% in Table 4. Interestingly, shown by the "All" column in Table 4 for Gemini 1.5 and App. B.4 for Gemini 1.0. naïvely scaling the number of exemplars may not be the most effective - the fact using the entire D. underperforms optimized exemplars, which are a subset of D... highlights the relevance of EO even for modern LLMs capable of handling long context windows. On the other hand, heuristic-basee based selection like Diversity and Nearest do not consistently outperform simple random baseline, echoing previous findings [33].
Imitation of task-dependent winning patterns outweighs elaborate descriptions. We further present representative prompts in Fig 6 and how LLM responds to them in App. B.9: Gener- ally, we find that even detailed and task-specific instructions do not regulate the LLM's behav- ior as effectively as exemplars, which enable imitation. For example, for challenging tasks like srrrriinnssufflleeee _ed__eddeeeeeeeeeeeeeeeeeeeeeeee jetts and web_of_ oi_eies, optimization basedd EO discover "winning" tem- plates that, when followed, improve performance massively. Even when SoTA I0 methods may often state the answer steps equivalently in words, we find LLMs to simply respond better to exemplars from which they can copy behaviors. On the other hand, for tasks where CoT--style answering are known to be unhelpful (e.g., snarks) [43], the optimized exemplars are invariably those giving direct answers; when prepended to the test queries, LLMs tend to prioritize exemplar imitation over instruction following to answer directly despite triggers like "We should move forward in stages". These highly task-dependent "winning" patterns that vary from elaborate step-to-step reasoning to direct answering may also explain why heuristic-baser EO fares worse to data-driven approaches, since there might not be a single heuristic universally useful for all tasks.
Concluding remarks. We argue that the findings are highly significant for the future of APO. First, they point to a need of re-balancing; without disparaging the value of IO, we argue that EO is at least
8

---

tracking_
Trackingsshuffled shuffled objects (7)
multistep_ auutittep_aithmmtii
snarks
100
80
75
acc (%)
80
60
75
50
60
40
50
25
0
#
Iterations 32
O
# Iterations 32
O
# Iterations
32
O
# Iterations 32
PaLM (text- bisen (02)
Gemini 1.5 Flash
Eval. budget m
32
64
Eval. budget in
32
64
10 Budget mioo
32
24
16
8
D
J2
10 Budget ihto
32
24
16
8
32
EO Budget
16
21
32
32
EO Budget moxo
0
32
Aug. test acc. (%)
73.26
74.49
76.14
72.92
76.25
Ang. test ace. (%)
83.251
84.00
8482
83,71
Aug. test rank 3
4.50
3,63
3.44
2.88
3.60
2.94
Arg. test rank J
4.04
3.65
3.29
3,00
3.58
3.44
Figure 7: Mixing--ann aaneing--nn--nnn and- matching EO and 10 outperforms either alone under a similar budget. Top figure: Validation accuracy vs. # evaluations on Dval with PaLM 2 in selected tasks if we optimize instructions only (via APE), exemplars only (Mutation), or both (first evals for IO (purple shade) + remaining 24 for EO). Gray dashed lines denote the performance of Tu- Bottom table: Test accuracy averaged across all tasks for different IO/EO budget allocations for PaLM 2 and Gemini 1.5. UUsed best APE results without EO that incur additional evaluations. Refer to App. B.7 for all per-task results and additional results on Gemini 1.0 Pro and other instruction optimizers.
equally crucial and should not be relegated to an afterthought. Second, we note that the EO strategies studied are in no way exhaustive. In fact, in contrast the sophisticated search and instruction generation approaches adopted by IO methods, they can even be considered elementary. Yet, they deliver comparable more significant improvements. Thus, anticipate advanced methods that more effectively optimize exemplars would yield even greater enhancements: some techniques in IO may be adapted to EO with little-to-no modifications. For example, many recent advances in IO adopt an evolutionary search framework- while our "Mutation" baseline can be seen as an elementary version of it in EO, it should be also straightforward to use more advanced search strategies and operators or use techniques like LLM LLM-genrrated paraphrasing on top of selected exemplars. Other search techniques, such as sample-efficient combinatorial BO [5, 44, 11, 65], can be uniquely suitable for the EO setup, which is itself a combinatorial optimization problem'. Furthermore, while we used a fixed set of exemplars (i.e., task-wise selection), it might also be fruitful to further explore in the direction of instance-wise selection [5]. Lastly, as discussed, the presence of (often large) generalization gap also suggests the importance to consider generalization alongside optimization, which seems to be the chief focus thus far; it might be promising to investigate analogies of well-tested in classical machine learning like regularization and cross-validation in APO.
Optimizing both instructions and exemplars greater than the sum of its parts, even under a comparable computational budget.
Insight 3:
For most of the results obtained, we note that iteratively optimizing both instructions and exemplars led to the best performance. This naturally leads us to answer the second research question: whereas experiments in Tables 1 and 2 expend additional cost by optimizing exemplars top of the optimized instructions, we show that 1) such a combinable benefit does not simply root from the additional com- pute and 2) optimally mixing-and--ng matching I0 and EO leads to significant performance improvement with negligible additional overhead.
Concretely, we budget a total m = 32 prompt evaluations on D where we use first m,o= iterations optimizing instructions and the remaining mgo optimizing exemplars. We summarize the results in Fig. 7, where we find that 1) any mix-and-mand EO only (i.e., mro=0 or 32), and 2) the best allocation bridges the gap or almost bridges the gap compared to the combination that uses twice as many prompt evaluations (last column) interestingly, in this case the optimal allocation also roughly reflects the relative contribution of 10 and EO to the overall performance improvement in Table 1. We show in App. B.7 that the above findings hold for other target models and instruction optimizers, and we also give detailed examples and explanations of the mechanism leading to this synergy. Additionally, in App. 10 we compare this simple routine against ProdptBreeder [14], which is one of the few existing I0 methods that supports EO via an
outperforms 10 or
"Wu et al. [52], a concurrent work, precisely explored a combinatorial BO solution.

---

optional "context shuffling" routine. It, however, mutates the exemplars purely stochastically rather than optimizing from the validation metric. We show that despite the simplicity, our algorithm converges to comparable or better solutions while incurring fraction of the PromptBreeder cost, which often requires hundreds of evaluations before convergence. Finally, we note that the presented way to combine optimization-base based IO and EO is proof of concept and room for future improvement can be vast and we experiment several other alternative ways to combine them in App. B.8, but we defer thorough investigations to a future work.
Insight 4: SoTA I0 methods may be inadvertently relying on exemplars already.
Beyond inspecting performance metrics only, we also examine the actual instructions and exemplars discovered. While detailed prompts are available in App. C, we highlight a key observation that adds a new dimension to our discussion: SoTA IO strategies may inadvertently utilize exemplars already, Despite IO methods not typically explicitly focusing on exemplars, we find them to frequently generate texts resembling exemplars within instructions through feedback and reflection processes. For instance, PromptBreeder employs "Lamarckian mutation" to reconstruct instructions from input- output pairs, while ProTeGi prompts the optimizer model to analyze target model errors. These operators, though varied, all involve taking actual validation exemplars as inputs to optimizer models. As exemplified by Fig. 8, while the original intention may have been to abstract task-level instructions, the model occasionally incorporates these exemplars verbatim in the instructions. Whereas these "quasi-exemplars" may seem unintentional, we observe that they are surprisingly common in high- performing instructions and often contribute more to the performance than the actual instructions themselves.
Full prompt 84.5%
Quasi axomplors only: 80.5% (-4.5%)
We argue that the findings here provide further evi- dence corroborating our insights obtained so far and our suggestions advocating explicit EO. Indeed, in contrast to explicit optimization of the exemplars, the aforementioned mechanism of exemplar discovery via 10 is entirely opportunistic and, depending on interpretation, an unintentional artifact. For exam- ple, the quasi-exemplars in Fig. 8 almost certainly originate from the optimizer model in ProTeGi tak- ing a convenient shorteut by incorporating critique into the instruction verbatim (note the presence of traces like "Label: a Prediction: b" which suggests a previous mistake by the target model), which should not happen if the optimizer model per- fectly executes the intended task of abstracting these critiques. Thus, we argue that instead of relying on the opportunistic exemplar generation via I0, explic- itly optimizing for exemplars can be more preferable, as shown throughout this study.
Removing guasi eecemppppeee ecemplers: 71.0% (-13.5%)
Let's address this syslematicalyy To detemiine the correct onder of vdjectivs in sentence. we need to followe specific order. The genaral order of adfectiona is follows::
1. Opinion adjectives
Size adject vis
Purpose adjectives
of contusion,, you can unilize the ROYAL moeemonc
aid:
R- Relarive size (big smal0: O Oginion fovautiful uglyt Age
lyouung oldiA A Shape (equin, roundly Color (red, blus))
applying this
ascertain the correct order adjectivves
let's examine the esamplee provided. itt Opt (A smail archaic eatter sock archaic smal leather Predictionc
"Whichsenter the adjactive order.
The comeect
adjections this "archalc "squaro." "lather? Raferring the ROYAL meamnoncc the shane. matoria:" Therafore. cornect acjeectivess if "smmll square. the appropriate order wather..
"small"
esamples cmmiteee)
Figure 8: The best instructions might actu- ally be exemplars: Best instruction discov- ered by ProTeGi on hyperbaton where spon- taneously discovered "quasi-exemplar highlighted. We also edit the instructions to either remove or retain the higlighted parts and find these quasi-exemplart rather than the rest of the instruction, drive the perfor- mance. See App. B.11 for more examples.
4 Conclusion
are
We present comprehensive evaluations on the SoTA 10 and EO methods, both individually and combined. We demonstrate that EO can be a potentially more crucial element in APO, revealing a beneficial syn- ergy through joint optimization with IO. We also find that the high performance of SoTA IO methods can themselves driven by implicit yet spontaneous exemplar discovery. Overall, we advocate for further research into EO, both as an independent approach and in conjunction with IO, even for highly-capable One limitation is that although the tasks we consider are fairly diverse and findings on it are already of value given the widespread interest just on these tasks only, they are not exhaustive, omitting tasks like open-ended longer-form generation and metrics like safety & harmfulness which important for responsible use of LLMs. As is the case for any inductive study deriving insights from experiments, there is also possibility that the findings may not fully generalize to other tasks and/or models. Expanding to include these aspects would be important for future work.
cittuuttionfolloviiii
modern models.
10

---

Acknowledgements
We thank all colleagues from Google Cloud AI Research for their feedback. We would also like to thank the anonymous NeurlPS reviewers and area chairs, whose valuable comments have helped to improve our paper.
References
[1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Agarwal, E., Dani, V., Ganu, T., and Nambi, A. Promptwizard: Task- aaske agent-driven prompt optimization framework. arXiv preprint 2tt.:2205.8366 2024.
[3] Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Chan, S., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J. D., Chu, E., al. Many-shot in-context learning. arXiv preprint arXiv:2404. 10188 2024.
[4] Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D.. Passos, A., Shakeri, S., Taropa, E.,
Bailey, P., Chen, Z., et al. Palm technical report. arXiv preprint arXiv:2305. 10403, 2023.
[5] Baptista, R. and Poloczek, M. Bayesian optimization of combinatorial structures. In Interna- lional Conference on Machine Learning, pp. 462-471. PMLR, 2018.
[6] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] Chen, I.., Chen, J., Goldstein, T., Huang. H., and Zhou, T. Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2.366.33088 2023.
[8] Chen, Y., Zhao, C., Yu, Z., McKeown, K., and He, H. On the relation between sensitivity and accuracy in in-context learning. arXiv preprint xrXiv22209.076611 2022.
[9] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
[10] Coulom, R. Efficient selectivity and backup operators in monte-carlo tree search. In Interna- tional conference on computers and games, pp. 72-83. Springer, 2006.
[11] Daulton, S., Wan, X., Eriksson, D., Balandat, M., Osborne, M. A., and Bakshy, E. Bayesian optimization over discrete and mixed spaces via probabilistic reparameterization. Advances in Neural Information Processing Systems, 012112770-127777 2022.
[12] Deng, M., Wang, J., Hsich, C.-P. Wang, Y., Guo, H., Shu, T., Song. M., Xing, E.,, and Hu, Z. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3369-3391, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2000 10.1665/v1720222mmmmmmmaa URL httpp::/aacaatthhllgg.oooo .rge:.orrr///g. 2022 emnlp-main 222.
[13] Do, X. L.,, Zhao, Y., Brown, H., Xie, Y., Zhao, X., Chen, N. F., Kawaguchi, K., Xie, M. Q., and He, J. Prompt optimization via adversarial in-context learning, arXiv preprint arXiv:2312.02614 2023.
[14] Fernando, C., Banarse, D., Michalewski, H., Osindero, S.. and Rocktäschel, T. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309. 2023.
[15] Gemini team, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., al. Gemini: family of highly capable multimodal models. arXiv preprint 2rXiv:2312.11800 2023.

---

[16] Guo, Q.. Wang, R., Guo, J., Li. B., Song, K., Tan, X.. Liu, G., Bian, J., and Yang, Y. Connect- ing large language models with evolutionary algorithms yields powerful prompt optimizers. International Conference on Learning Representations, 2024.
[17] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.00 03300, 2020.
[18] Honovich, O., Shaham, U., Bowman, S. R., and Levy, O. Instruction induction: From few examples to natural language task descriptions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1; Long Papers), pp. 1935-1952, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18653/1/2023..km acl-ooog.... URL org/2023 acl-long 108.
attps://aclanthonn
[19] Hsieh, C.-J., Si, S., Yu, F. X., and Dhillon, I. S. Automatic engineering of long prompts. arXiv preprint arXiv:2311.10117, 2023.
[20] Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., and Zaharia, M. intensive Demonstrate-ssarcch strrrtttttsscrrr---edttttt Composing retrieval and language models for knowledge- -owllege-intensive nlp. arXiv preprint arXiv:2212. 14024, 2022.
[21] Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. DSPy: Compiling declarative language model calls into self-improving pipelines. International Conference on Learning Representations, 2024.
[22] Kim, H. J., Cho, H., Kim, J., Kim, T., Yoo, K. M., and Lee, S.-g. Self-generated in-context learning: Leveraging autorsegressive language models demonstration generator. arXiv preprint 2rXiv22206.08086
2022.
[23] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22241 2022.
[24] Lee, J., Dai, Z., Ren, X., Chen, B., Cer, D., Cole, J. R., Hui, K., Boratko, M., Kapadia, R., Ding, W., al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint vrXiv:2403.20322 2024.
[25] Lin, X., Wu, Z., Dai, Z., Hu, W., Shu, Y., Ng. S.-K., Jaillet, P., and Low, B. K. H. Use your instinct: Instruction optimization using neural bandits coupled with transformers. arXiv preprint trXiv:2310.02905. 2023.
[26] Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing, ACM Computing Surveys, 55(9)11-35, 2023.
[27] Liu, S., Chen, C., Qu, X., Tang, K., and Ong, Y.-S. Large language models as evolutionary optimizers. arxiv preprint arXiv:2310.1900 2023.
[28] Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 8086-8098 2022.
[29] Margatina, K., Schick, T., Aletras, N., and Dwivedi-Yrs Yu, J. Active learning principles for in-context learning with large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 501 5011-5034, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653vv 1/2033/ findings-emmlp.334 URL pttttp:///acaanntollos wrg.://...g222222 findings-emnnnaa eetpssemmnl..... 334.
[30] Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., and Stanovsky, G. State of what art? a call for prottp I1 evaluation. arXiv preprint arXiv:2401.000 00595, 2024.
[31] Nguyen, T. and Wong, E. In-context example selection with influences. arXiv preprint 1014, 2023.
12

---

[32] Opsahl-Ong K., Ryan, M. J.. Purtell, J., Broman, D., Potts, C., Zaharia, M., and Khattab, O. Optimizing instructions and demonstrations for multi-stage language model programs, arXiv preprint arXiv:2406. 1695, 2024.
[33] Perez, E., Kiela, D., and Cho, K. True few-shot learning with language models. Advances in neural information processing systems, 34:1 1011-115111111 2021.
[34] Prasad, A., Hase, P., Zhou, X., and Bansal, M. GrIPS:
Gradient-free, bdit-based instruction search for prompting large language models. In Vlachos, A. and Augenstein, I. (eds.), Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3845-3864, Dubrovnik, Croatia, May 2023, Association for Computational Lin- URL httpt::ttcccaaaaaaaaaa
guistics. doi: 10. (/655//12022.eacl-mma eacl-main, 277.
org/2023.
[35] Pryzant, R., Iter, D., Li, J., Lee, Y., Zhu, C., and Zeng. M. Automatic prompt optimization with "gradient descent" and beam search. In Bouamor, H., Pino. and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7957-7968, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18865/v1222 emnlp-main.494. URL https:://aaaanttollllll
patp::/aalaaatttte aalaaaaaaanoooyoooooooooooooolal org/202 emnlp-mane 494.
[36] Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R.. Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.0553 2024.
[37] Rubin, O., Herzig, J., and Berant, J. Learning to retrieve prompts for in-context learning. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V. (eds.). Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655-2671 1, Seattle, United States, July 2022. Association for Computational Linguistics. doi: acl aclanthology. org/2022 maacl-maine 191.
1/255//1/002.mmaclll waa.aaaallmiinn9999 191. URL https://
[38] Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., and Chadha, A. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024.
[39] Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. Quantifying language models' sensitivity to spurious features in prompt design How learned to start worrying about prompt formatting. arXiv preprint arXiv:2310. 11344, 2023.
[40] Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222-4235, 2020.
[41] Stanić, A., Caelles, S., and Tschannen, M. Towards truly zero-shot compositional visual reasoning with ilms as programmers. arXiv preprint arXiv:2401.01974, 2024.
[42] Sun, H., Li, X., Xu, Y., Homma, Y., Cao, Q., Wu, M., Jiao, J., and Charles, D. Autohint: Automatic prompt optimization with hint generation. arXiv preprint arXiv:2307.07415, 2023,
[43] Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:22 00.011 2022.
[44] Wan, X., Nguyen, V., Ha, H., Ru, B., Lu, C., and Osborne, M. A. Think global and act local: Bayesian optimisation over high sigh-dimensional categorical and mixed search spaces. In International Conference on Machine Learning. pp. 10663-10611 PMLR, 2021.
[45] Wan, X., Sun, R., Dai, H., Arik, S., and Pfister, T. Better zero-shot reasoning with self- adaptive prompting. Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 3493-3514, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18633/1/2023.1 findings--nadl.222. URL https: / /aclaantoooogyy...... org/2023. findings- 216.
13

---

[46] Wan, X., Sun, R., Nakhost, H., Dai, H., Eisenschlos. J., Arik, S., and Pfister, T. Universal self-adaptive prompting. In Bouamor, H., Pino. J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7437-7462, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v 12003.. emnlp-main.461. URL https: http://aaaaaattooooeoeee org/2023. emnlp-ma 1-m.m...61..11
[47] Wang, R., An, S., Cheng, M., Zhou, T.,, Hwang, S. J., and Hsieh, C.-J. Mixture-of-experts in prompt optimization. International Conference on Machine Learning, 2024.
[48] Wang, X., Li, C., Wang, Z., Bai, F., Luo, H., Zhang, J., Jojic, N., Xing, E. P., and Hu, Z. Promptagent: Strategic planning with language models enables expert-level prompt optimization. International Conference on Learning Representations, 2024.
[49] Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https: //pp////ep//eeeeee//e//e
gEZrGCozdqR.
[50] Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V. Zhou, D., al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24887 2022.
[51] Wu, Z., Wang. Y., Ye, J., and Kong, L. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Proceedings of the 6 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1423-1436, 2023.
[52] Wu, Z., Lin, X., Dai, Z., Hu, W., Shu, Y., Ng. S.-K. Jaillet, P., and Low, B. K. H. Prompt optimization with ease? efficient ordering-aware automated selection of exemplars. arXiv preprint arXiv:2405. /6122, 2024.
[53] Xu, H., Chen, Y., Du, Y., Shao, N., Wang. Y. Li, H., and Yang. Z. Gps: Genetic prompt search for efficient few-shot learning. arXiv preprint arXiv:2210. 17041, 2022.
[54] Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models optimizers. In International Conference on Learning Representattons, 2024.
[55] Ye, J., Wu, Z., Feng, J., Yu, T., and Kong, L. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pp. 39818-39833, PMLR, 2023.
[56] Ye, Q., Axmed, M., Pryzant, R., and Khani, F. Prompt engineering a prompt engineer. arXiv preprint rrXiv:2311.05661 2023.
[57] Zhang, T., Wang, X., Zhou, D., Schuurmans, D., and Gonzalez, J. E. Tempera: Test-time prompting via reinforcement learning. International Conference Learning Representations, 2023.
[58] Zhang, Y., Feng, S., and Tan, C. Active example selection for in-context learning. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference Empirical Meth- ods in Natural Language Processing, pp. 9134-9148, Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguistics. doi: 10.1865...11 URL https: http///caaaaathooc org/ emnll-manlmmmmm..e maai-main... aamain.622 622.
622..
[59] Zhang, Y., Zhou, K., and Liu. Z. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36, 2024.
[60] Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. International Conference on Learning Representations, 2023.
[61] Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning. pp. 12697-12700 PMLR, 2021.
14

---

[62] Zhou, H., Wan, X., Vulić, I., and Korhonen, A. Survival of the most influential prompts: Efficient black-box prompt search via clustering and pruning. In Bouamor, H., Pino, J., and Bali, K. (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13064-13077, Singapore, December 2023. Association for Computational Linguisties. doi: 10. 10613311111223.. findings-emnlp.8700 URL https http///acaaaatlooooo/oo org/2023. f findings-ennnn emn1p 870.
[63] Zhou, H., Wan, X., Liu, Y., Collier, N., Vulić, L., and Korhonen, A. Fairer preferences elicit improved human-alle -ainned large language model judgments. arXiv preprint arXiv:2406.13370 2024.
[64] Zhou, H., Wan, X., Proleev, L., Mineu, D., Chen, J., Heller, K., and Roy, S. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. International Conference on Learning Representations (ICLR), 2024.
[65] Zhou, H., Wan, X., Vulić, I., and Korhonen, Autopeft: Automatic configuration search for parameter-efficient fine-tuning. Transactions of the Association for Computational Linguistics, 12:525-544 2024.
[66] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human-level prompt engineers. In The Eleventh International Conference Learning Representations, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and instroduction accurately reflect contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We clearly discuss limitations in the Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "LLimitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We fully disclose the datasets (all datasets are publicly accessible) and the mod- els used (all models have public API). We include all discussions about the implementation details in Appendix. A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g.. with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While we aim to apen-source our experimental code in the future, we cannot open source the codebase at the time of submission. However, many of our experiments derive from existing methods, which can be reproduced by running the respective, open- sourced codebase which we outline in Appendix A.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.co wwtpccccppppccccttcccccccccccccc http:////guiieeeeeeeuubbiiiiiie Cwtp//oooeeeteeeeeeeeeeeee
swi.iiosioopyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We fully disclose the datasets (all datasets are publicly accessible) and the mod- els used (all models have public API), include all discussions about the implementation details in Appendix. A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While we do not repeat each experiments more than once due to the com- putational constraints. we perform experiments on a large set of tasks, and insights are largely drawn from the aggregated results averaged over 70+ different tasks to improve the reliability of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include all necessary details in Appendix. A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tttu::///eurpps httprripp.ppppppcccceee httpppulic/thiiauuiiiieee.eeeeee
Answer: [Yes]
Justification:
We have reviewed and agreed to the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We include such discussion in the Conclusion.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper does not release new generative models or scraped datasets but rather derive from existing models and datasets. Therefore, we do not anticipate such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We list the license and terms use in Appendix A. Original creators/owners of all assets used in the paper are properly credited with citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com condaataeess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets. We entirely use assets that have already been made available.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22

---

Appendix
A Implementation Details
Input prompt templates. In this section, we outline the input prompt templates we used for all experiments for reproducibility. For seed instruction (i.e., No IO), APE and OPRO, adopt the following template for all datasets.
Q: { QUEAY_ }}
( MNSWER_INSTRUCTEN
}}
A: < INSTRUCTION }} {{ 11m( }}
In the template above, QUERY_TEX: denotes the input text; INSTRUCTION denotes instruction
to be added, which is the principal optimizable component of IO methods; 11m() 1) denotes the location where LLM is prompted to generate an output. ANSWER INSTRUCTITNNNN
a special, task-
specific sentence to ensure the LLM generates the final answer in format that can be easily parsed. Specifically, for all multiple-choice questions-style tasks, has the following content:
Show your final ansver option bracketed between <anaver> and <\ansue>>.
For all other tasks, the content is: Show your final
CONSTRAINT }} bracketed betseen {{ FF
and <\
ansver>
where the content of MASK_SPECIFIC SPECCIFICCCNNNTTAAANT depends the task:
boolean boolean_expressteeeeee _expressions: (True or False only)
formal_fallacies: fallacies: (valid or invalid only)
navigate, sports coarrts_unerttadding causal cousal_judgement web_of_line lies: (yes or no only)
word sorting: (sorted words separated by spaces only)
all other lasks: None (empty string).
At test time, the final answer is extracted with the capturing pattern <answer>... ..........aasssan <annwerr> When exemplars are added, we use the following template:
D: DEMO QUERY TEXT }>
{ DNSSER_IISTRUCTION }}
A: (( INSTRUCTION }} (( DENO_1_ 1_NU_UT }
DEMO_2. 2.QU___
7 { ABSWER_ INSTRUCTION }}
A: { INSTRUCTION }} {{ DENO DENO_2_OUTPUT
10
12 Q: < QUERY TEXT }}
{f AUSWER_ INSTRUCITTT
14 A: { INSTRUCTION }} (( 11m() }}
where DEMO {it__iit_PP OUTPUT contains the entire response from the LLM the corresponding input (not the final answer only). For ProTeGi and PromptAgent, we follow the templates used in the respective original papers with the format that puts the instruction and answer instruction in front of the test query:
< INSTRUCTION }} { INSERUCTTST }>
< QVERY_TEXT }}
< 11a( }}
Accordingly, modify the template with exemplars to:
< INSTRuCTION }}
2 { DEMO_1_QVERY. TEXT }> {( INSVER_IXSTAUCTIO: }>
DEMO_1_OUTPUT -
_BXT < DEMO_2_QUERY. TEXT }} {( IXSTRR_IXT }}
{ DEMO_2_0UTPUT }}
11 < QUERY_TEXT TEXT }} { INSEER.INSTRUCTIN }}
12 { 11n() }}
23

---

Noting that instruction is stated once at the beginning only rather than repeated at each exemplar, in consistency to the original styles adopted by these papers.
Implementation details of IO methods. In this section, we describe the implementation details of the 10 and EO methods adopted. For all methods, we use greedy decoding (temperature = 0) for the PaLM 2 (text-bison--e - 002) or Gemini target models. Whenever an optimizer model is used, we use temperature = 1.0, top_k k = 40 and top_p- = 0.8. For both PaLM 2 and Gemini models, we use the Google Cloud Vertex AI API available at https: //cl:////loodd google. wwwc womm/ertex-ai.
APE: We adapt the official implementation available at https://githan www//oirrppp sereaticc_pompp_eengg__e
Instead of using "instruction induction" [18] from exemplars which is the primary initialization method introduced in the original paper in $3.1, Forward Mode Generation or Reverse Mode Generation, we opt for the third option, Customized Prompts where we initialize APE at the seed prompt "Let's think step by step" because 1) this ensures fairness in comparison with other methods and 2) we find initializing at the seed prompt actually leads to much better performance because it is well-known to induce step-by-step. CoT-style reasoning from the LLM. On the other hand, while the LLM induced initial instructions may describe the task better, the model often tends to utter the final answer without intermediate steps which observe lead to much worse performance: on the BBH tasks selected for experimentation in this work, instruction induction using the meta- meta- prompt provided by the APE paper only led to an average test accuracy of 56.7% on PaI.M 2 (text-bison-O00
which is even worse than using the seed prompt with no additional optimization. For APE, we use population size of 8 and allow for 4 generations.
OPRO: we adapt the official implementation available at nttps://githui com/ pogleeddeepinde /orro. At each optimization step, we follow the authors by asking the optimizer model to generate 8 candidate prompts and we budget for 4 steps.
ProTeGi: We use the official implementation available at https://gi //gp::////tttbb www/miimmmcccccc.......o comm/iitrooot///////ooooo///om/t LMOps. We set the initial prompt to the seed prompt, and we always use the entire validation set to generate the "gradients" (i.e., we use no mini-batching). We set the number of newly proposed instructions per optimization step to 8, where half of them come from "gradients" (i.e., new instructions generated by using the optimizer model to critique past mistakes made by the target model) and the other half come from "Monte Carlo samples" which are paraphrased/rewritee variants of the past prompts by the optimizer model. We again allow for 4 generations of mutations. PromptAgent: We use the official implementation at https: / //github....... com/ /am///nyuanaagCS..... Promtt We set the initial prompt to the seed prompt and use the default Monte Carlo Tree Search algorithm and set the number of maximum iterations to 32 to be consistent with the other methods described above.
Computational Resources. All experiments conducted in this work are accessible via public APIs where the underlying L.LMs are hosted from the server side. There is no computational resource requirement on the client machine except that one can access Google Colab and run Python 3.10.
Datasets. As discussed briefly in 1, we rely existing assets to perform experiments. Both the BIG-Bench Hard (BBH) dataset and the MMLU dataset we used are licensed under the MIT License (BBH: https:/ //gi tig. com/ Haa https://gi thub.
hate--HHHa///booiiianaaaaaa http///booonnnnnaaaaaae MMLU:
B
Additional Experimental Results
B.1
Detailed Results on BBH
In Tables 5 to 10, we show the per-task performance breakdown using the PaLM (text-bison-002) target model whose aggregated results are presented in Table In Table and Table 12, we show the results using the Gemini 1.0 Pro and Gemini 1.5 Flash target models whose aggregated results are presented in Table 2 and Table 4, respectively. The aggregated and task-specific breakdown results on the GPT-3.5 (gpt-3. 3.5 5-turbo -ur5o-0125) model are shown in Tables 13 and 14, respectively.
24

---

Table 5: Per-task test accuracy (%) of the PaLM-2 (text-bison-002) target model without exem-
plar optimization (No EO).
RO method
No FO
10 method
No 10
OPRO
APE
PeuTeGi
PromptAgent
boolean_ expeesssisss
73.50
80.00
80.50
81.00
77.00
jussa_judgenee
61.33
58.67
61.33
60.67
65.33
date unteesuddrst
70,50
67.00
74.00
70.50
66.00
disambiguation_c formal fallacies
66.50
68.50
62.50
64.50
69.50
48,50
49.00
$3,50
50.50
geometric_shne shappesssssss
60.00
59.00
56,00
52.50
46.00
hyperbaton
55,50
79.00
69.50
84.50
86.50
logical Delluuttoon_ive_ohee
49.00
50. S0
58.00
55.00
$2,50
logical deductionn severtioonseeen_eee seeeeeeeeeeeeeeeeeeeeeeeeeeeeeee objects
45.00
48.50
53.00
54.50
logicul deduction three_ three_objects
74.00
70.00
75.00
78.00
76.50
movic recommendation
56.50
67.00
57.50
71.00
82,00
multistep_arihhmetic__ arrthhmetie_tee two navigate
38.00
61.50
62.50
43.00
50.00
56.50
56.00
60.50
65.00
63.00
Wbjecl_counting
69.50
90.50
81.50
penguins_ i_a_lable
75.21
80.34
86.32
79.49
78.63
reasoning_abee aabout_colored objects
66.00
67,50
70.50
67.50
65,50
ruin names
69.00
73.50
70.50
79.50
74.00
salient_ translation eror detection
55.00
53.50
54.50
50.00
53,00
snarks
70.63
70.63
73.43
69.23
81,12
sports_ understandingg
75.50
76.00
75.50
85.00
73.50
cemporal_sequencer
87,00
86.50
90.00
99.00
90.00
tracking shuffled objects_ fiveeliveeoe_oeee__eee objects
29.00
40.50
31.00
43.00
38,50
tracking. shuffled objects_seven seven objects
36.00
37.00
38.00
51.50
33.50
tracking shuffled objects chhro__object
49.00
30.50
67.00
56.50
web of lies
55.50
53.50
71.00
82.00
70.00
woon_ssrting
75.50
73.00
76.50
75.50
72.00
Average test accuracy (%)
60.30
63.04
64.96
68.13
65.66
Table 6: Per-task test accuracy (%) of the PaLM-2 (text-bison-002) target model with random exemplar optimization (Random).
EO method
Random
10 method
No 10
OPRO
APE
ProTeGi
PrompiAgent
boolean boolean_cxpressiee expressions
86.00
88.00
89.50
84.00
86.50
causal jusgel_jddementtt
67.33
64.67
62.00
63.33
62.67
date wwddrsttaddinnggg
77.00
74.00
76.00
79.00
disambiguation__g
69.00
68.50
64.50
61.00
72.50
formall[aallacis) fallacies
58.00
56.50
59.00
$9.00
58.00
geometric_ shapes
57
57.00
60.50
50.00
46.50
hyperbaton
$1.50
71.00
72.00
77.00
logical, deduction, five objects
51.00
50,00
48.00
47.00
45.00
logical, _eeuueddd_dddeeeeeeee Berrilssseeeeeeeeeeeeeeeeeeeeeee _seven_objects
49.50
54,00
35.00
42.00
52.00
logical. deduction three_ oojject
73.50
70,50
74.00
72.00
73.00
movic recommendation callistte__rithheete two navigate
76.50
77.50
80.50
84.00
94.00
49.50
76.50
71.00
53.50
53,00
58.50
61.00
61.00
65.00
54,00
Dojjet_ccuniigg
78.00
85.50
92.00
97.50
97.50
penguins in a_ table
76.07
82.91
77.78
84.62
78.63
reasoning aboutaaooccooo
colored about_colored
_oojeet_objects
74.50
75.50
72.50
75.50
names
81.50
83.00
78.50
89.00
86.00
salient_tree translationnnnnoooo ceror detection
61.50
60.50
60.50
56.50
58.00
snarks
78.32
79.02
77.62
82,52
82,52
pports_understtamie
79.00
82.00
81.00
91.50
88 50
temporal sequuencs
94.00
95.50
91.00
94.00
tracking shuffled_ _bjjccts_s___eeeeee five objects
34.00
45.50
35.50
56.00
23.50
tracking shuffled objects seven objects
33.00
41.00
55.50
38.50
43.50
tracking_shate wwwfsshcuuobjeeeeeec obwoccssttrreeeoheeettooeee tatcc_oobcctttt..teeeee
52.50
33.00
58.50
63.00
56.00
weh of lies
65.50
70.50
86.00
97.50
47.50
word sorting
77.50
78.00
77.50
78.00
Average test accuracy (%)
66.91
68.50
69.11
70.81
25

---

Table Per-task test accuracy (%) of the PaLM-2 (text-bison-002) target model with nearest
exemplar optimization (Nearest).
EO method 10 method
Nearest
No 10
OPRO
APE
ProTeGi
PromptAgent
boolean hoolean_expressine expressions causal jusal_judgement jal__understanding disumbiguation_.e formal fallacies
86.50
86.30
87.50
76.50
77.50
68.00
62.67
67.33
66.67
65.33
81.50
77.50
78.50
77.50
62.50
71.50
60.00
59.00
68.50
58.00
60.50
58.00
56.50
54.50
gcometric shapes
66.00
62.00
63.00
54.50
57.50
hyperbaton
76.50
68.50
71.50
80.00
82,00
logical. deduction. five objects
49.00
46.50
44.00
38.00
46.00
logical. deduction seven, objects
41.50
48.50
36.50
43,00
44,00
logical deduccieonnnnnn three_ole sajec_s
68.50
72.50
71.50
72.50
76.00
mmoiierreommment
80.00
81.00
81.50
86.50
90.00
mullistep_annin catissep__antmmetii_tooo
51.50
71.00
70.50
63.00
navigate
51.00
56.50
51.00
51.00
48.00
object object_counting
80.50
84,50
92.50
92.00
96,00
penguins in a table
80.34
84.62
84.62
82.05
76.07
reasoning ahouut coloredd_ddcccecc colored_objects
73.50
75.50
77.00
69.00
71.50
ruin_ names
77.00
84,50
86.50
78.00
translationn eerieeerrrrrrrreeeeeeeeee detectinn
58.50
58.00
58.50
58.00
61.00
snarks
76.92
83.22
80.42
82.52
83,92
sports, understanding
82.50
83.00
82.00
89.50
83.00
temporal serpuorclssqqunccss
93.50
94,50
85.50
97.50
94.00
tracking tracking_shuffie shuffled objecct five objects
24.50
$1.00
29.00
49.50
31.50
laakiing_suflleee_heeeeeee c__jeoosjjssseeeeeeeeee _eeessss_see___eeeeeeeee tas___eeee___jeeee objects
34.00
41.00
52.00
63.00
42.00
tracking wttpsshhhulllojecittee wwwttterreeeeeteeeee
56.50
28.00
59.00
64.00
55.00
web_ lies
60.00
62.00
91.50
82.50
75.00
word sorting
80.00
81,50
81.50
79.50
Average test accuracy f
66.09
68.33
69.01
70.01
67.82
Table Per-task test accuracy (%) of the PaLM-2 (text-bison-002 target model with diversity exemplar optimization (Diversity).
EO method 10 method
Diversity
No I0
OPRO
APE
ProTeGi
PromptAgent
boolean hoolcan_expresssion causaljuudgement judgement causal
82.00
93.00
88.50
86.50
84,00
65.33
63.33
67.33
66.00
66.67
date sale_underslanding disambiguation
75.50
66.50
75.50
76.50
75.50
68.50
69.50
65.50
60.50
formal, fallacies
56.00
56.50
55.50
55,50
54.50
geometric. shapes
53.00
59.50
57.00
56.50
41.00
hyperbaton
80.50
66.00
70.50
80.50
87.00
logical. deduction, five objects
48.50
59.00
46.00
34.50
$1.50
logical. deduccion.
objects
57.50
58.50
38.00
46.00
50.00
logical, deduction three_ohe porec_o
74.00
71.50
75.00
72.00
73.50
movie movie_recommen ree_rrrmmmmenaiionn
72.00
76.00
79.00
84.00
88.50
55.00
72.00
72.00
64.50
$4.00
navigate
50.00
63,50
64.00
60.50
object Drbect_countin
84,00
86.00
93.00
86.50
97.00
penguins in table
72.65
66.67
85.47
76.07
76.92
reasoning about aal_aabboooorcdddd colored objects
68.50
72,50
80.50
69.50
74.50
ruin names
81.50
82,50
81.50
82.50
83.50
fraiintiaannlttionn...o cettiooo_errreeeeeeeee ceror_detection
55,50
62.00
snarks
79.72
83.22
83.22
86.01
83,92
sports_ www.erstaandinggggg
83.50
79.00
83,00
90.00
73.50
temporal sequences
92.50
89,50
96.50
99.00
92.50
tracking shuffled objects five tile_/ooeeeeeeeee objects
27.00
59.50
42.50
37.50
37.00
Beekingg_shuflee shufflledoobjectss..oooooe seven objects
44.50
33,50
47.00
$3.00
42.50
ohauu/llu__ojeetts_hhi web oeb_of_liee
68.00
33.50
62.50
60.50
$7.00
64.50
56.50
97.50
84.50
48.00
word sorting
75.50
77.50
78.50
77.00
78.50
Average test accuracy (%)*
66.74
67.57
70.81
69.25
67.35
26

---

Table Per-task test accuracy (%) the PaLM-2 text-bison-002) target model with random search exemplar optimization (Random search) with search budget m = 32.
EO method
Random Search (m 32)
10 method
No 10
OPRO
APE
ProTeGi PromptAgent
hoolean hoolean_expressioee expressions causal jusaa__udgement
85.50
91.50
89.00
83.50
84,50
68.67
68.67
64.67
63.33
64.67
date_ underuddesttnni disumbiguation_ee
74.00
79.00
78.00
70.00
65.50
66.50
68.00
72.00
formal fallacies
56.00
54,00
56.00
63,00
55.00
gcometric shapes
62.50
59.00
88,50
67.50
65.50
hyperbaton
80.00
83.00
82.50
85.00
90.50
logical. deduction. five objects
59.50
62.50
50.50
47.50
51.00
logical. deduction seven, objects
45.00
59.00
54.50
47.00
54,00
logical deduccieonnnnnn three_ole oojects
84,50
81.00
75.50
76.50
76.00
80.00
87.50
87.00
85.50
93.50
mullistepp_anine catissep__antmmetii_teo
53.00
73.50
75.00
75.00
navigate
65.50
67.00
65.50
68.50
66.00
object object_counting
94.50
99.00
98.00
98.50
98.00
penguins in a table
82,91
87.18
89.74
82,91
83.76
reasoning ahouut coloredd_ddcccccc colored_objects
73.00
80.50
74.50
76.00
76.50
ruin_ names
85.00
86.50
87.00
translationn eeri__errrrr_reeeeeeeeeeee detectinn
56.00
$4,50
62.50
56.50
58.00
snarks
82.52
83.22
87.41
84.62
87.41
sports, understanding
78.50
82.00
73.00
88.50
82.50
temporal sequunccss
93.50
92.00
95.50
98.50
96.50
tracking tracking_shuffin shuffled _bhocctsh_ess__eeeeeeeeeeeeeeee_ five objects
49.00
63.50
70.00
43.50
backinggshufllel_bbye c__jeoosjjssseeeeeeeeee _eeesssssse____eeeeeeeee tis___eeee___jeeee objects
40.00
48.50
63.50
70.50
39.50
tracking_shulln wttnsshhhulllojeclteee wwwttttrreeeeeeee
56.50
28.50
66.00
74.00
55.00
web_ _ies
97.50
77.00
99.50
98.00
95.00
word_ sorting
76.50
79.00
80.00
80.00
Average test accuracy f
71.16
73.02
75.88
75.90
72.51
Table 10: Per-task test accuracy (%) the PaLM-2 (text-bison target model with mutation exemplar optimization (Mutation) with search budget = 32.
EO method
Mutation (m 32)
10 method
No I0
OPRO
APE
ProTeGi
PromptAgent
boolean hoolcan_expreession causaljuudgement judgement causal
88.50
95.00
92.50
89.50
88.50
62.67
65.33
65.33
64.67
68.67
date dale_underslanding
81.00
76.00
79.00
80.00
78 55
disambiguation formal, fallacies geometric. shapes
70.50
69.00
66.50
67.50
71.50
60.00
50.50
55.50
57.00
57.50
64.00
64.00
86.00
74.50
54.50
hyperbaton
89.00
76.00
77.50
84.50
81.50
logical. deduction, five objects
55.50
55.00
53.00
53.50
62.50
logical. deduccion
objects
54.00
62.50
51.50
50.50
51.00
logical, deduction three_ole jorec_o
75.00
84,00
80.00
80.50
78.50
movie movie_recomme reeerrrrmmmenaiiinn calliisep__rithmmtti.___
84.00
86.00
89.00
87.00
95.00
51.50
775 00
75.00
82.50
55.50
navigate
58.50
67.50
67.50
66.50
object Dooent_counting
92.00
92.50
97.50
99.00
99.00
penguins in table
84.62
75.21
78.63
83.76
76.92
reasoning aalauabcolorrcdddd colored objects
74.00
74.00
75.50
71.50
76.00
ruin names
88.00
91.00
87.50
88.50
83.00
fraiintiaannlttionn... ceriiiioonerrrr_eeee errorrdeeection
61.50
61.50
61.50
snarks
79.72
80.42
82.52
84.62
85.31
sports_ wwd.erstanndinggg
85.50
83.50
82,00
86.50
$1.00
temporal sequences
99.50
98.00
95.00
100.00
95.00
tracking shuffled objects tile_/ooeeeeeaaa objects
53.00
64,50
59.00
70.00
50.50
Beekingg_hhuflee _buffled_oojjctsss...._oo seven objects
43.00
$3.50
whakiingsslllleooojeeeeeeee web lies
sbbjects_three_oojcct
66.00
26.00
63.50
69.50
58.00
93.50
96.00
98.50
99.50
98.00
word sorting
81.50
77.50
77.00
79.50
77.00
Average test accuracy (%)
72.92
73.06
76.25
77.29
72.77
27

---

Table 11: Per-task test accuracy (%) of the Gemini 1.0 Pro target model.
DO method
No BD
Random
Neurest
Diversity Proleta
Random Starch
Musation
10 method
No RD
Pro
No 10
ProleCi
No ID
Pro
No ID
No I0
Pro
No RU
PioteCa
boolean_ expressions
81.50
$2.50
18,00
B8.50
911.50
$7.50
90.50
87.00
91.00
91.50
N7.50
92.50
causel_ iddgement
54.67
58,00
64.000
08:67
01.33
04.00
62.00
03.33
59.33
06.00
03.33
05.33
dite indestinding
70.50
64.50
71
75.
79.00
75.50
73.00
70.00
78.50
78. 50
83.00
78.00
disaniiguaion
53.50
66.50
58.00
$8.00
68.00
63.00
56.00
67.50
69.00
69.00
70.50
67.00
73.50
formal fillacies
59.00
57.00
geometnec_sappe Stapes
37.50
47.50
71.00
81.50
typerbution
00
logical, deduction five objects
59.50
52.50
51.00
69.00
62.00
logical, dedaction sever_sseessoon obyosts
48,00
50
42,50
49.00
50:00
53.50
logical dedaccioonnnnnccoo
objecss
75,00
50
70.000
73.00
$1.00
76.50
N4.50
movic recommendatin
49.50
70 50
50
80
71.
00
52.00
80:00
miltisep.anmmmess anttmmmmm__ooo
61.00
62.50
72. 1..00
81. .00
65.50
65.00
73.50
85 85.00
84.00
25.00
73.30
84.00
savignte
70.50
61 130
78. 00
77
76.50
67.50
76, 00
69 69.50
85. 50
78 00
82.00
80:00
object
74.50
79,30
89.00
H8. 51
ME,00
50..0
93.50
92.00
95. 95.50
93.50
92.00
95.50
pengrins in a table
87.18
86.32
64
83. 76
$3.76
82.91
81 20
79.49
84.62
87 18
82.05
79.49
reasoning __ wow.oooooooodeccctco
70.00
69,50
$0.50
74.50
71.00
78,00
74.00
68.50
72,50
90.00
$0.50
78.50
nin somes
62.00
63.00
79,50
74. 00
72.50
76.50
68. 58.00
68.
77.00
77.00
76.50
78.00
trisstttanniltiionnnooo ennr dotection
$0.50
46.30
53. 01)
59.00 010
$2.00
64.50
60.50
62.00
58.00
58.50
58.00
58.50
snarks
81.82
76.
K2.52
79.27 72
83.92
76.9 92
62.52
80
86.01
R1.22
85.01
83.92
sperte undersindding
74,50
79,00
82.
85.
$2.50
92
93.50
96,00
92.50
seuperl_sqquances sequences
K1.
00
74.50
95.5
90.00
tncking shuflle objects. objects
macking_shuulled. obyects objects
70.00
50
81.00
$1.00
86.50
82.50
wcb of lics
56.50
94.55
97.5 50
38.00
90:00
97. 97.50
word socting
63.00
59.50
68.35 50
71.50
70.50
68.00
69.00
67.50
71.00 21.00
71.00
64.50
Average lest arcurary (
63.14
65.91
71.12
72.72
69.19
72.13
73.64
75.77
78.27
75.77
79.01
Table 12: Per-task test accuracy (%) of the Gemini 1.5 Flash target model.
30:10
11:3
00.03
04.0
01.23
02.01
0816
42:51
01:0
03.09
35.09
$2.53
55.8
12.55
14.50
0.50
$1.59
17.00
0761
01:11
0018
340
2910
11.10
18:0
61:01
18.55
13.00
(191
1815
01.9
Highal Jokie
0133
71.0
7150
71.51
13/08
5000
100
30
62.50
13.50
11.00
700
70.0
11.0
12.50
18.0
5531
02.00
30
6248
1100
33.3
51.30
22.30
53.00
65.56
13.11
03.00
11.00
12.00
42.3
74.00
53.00
74.0
0523
55.00
01
0101
1916
00.9
09/5
19.00
00.0
16.00
14.00
15.3
01.0
0101
01:01
(1
11:00
0111
0710
09
0100
61
81.00
1762
71:11
02:5
0.45
12.5
1511
1300
53.55
11.5
47
51
1377
3431
tacding Jedial cieea BrL (iu
1300
06.00
03:0
0559
96.3
100
548
06.30
50
02.50
07.58
12.0
13.0
12.0
9.00
07.50
9.30
96.0
0103
01.00
18100
0/01
06.
1819
19:09
16.0
14.31
11.
5100
0.50
1000
19:00
1910)
1860
18:30
18.51
Table 13: Average BBH ac- curacy (over subtasks) of seed instruction (No 10), APE and ProTeGi with different EO strategies using GPT-3.5
Table 14: Per-task test accuracy (%) of the GPT-3.5 target model.
EO mehod memheed
No Eo
Rasiom APE
Randon Search API
APF:
ProffeGi
No
ProleC
(O
ProtoCa
bollen exprssioos
78.50
75.50
$3.90
15.50
91:55
11:50
$6.00
15.00
55.55
15,10
1opt-3.5-turbo-O11
target
cassal indgement
date uiiminnni
$2.00
71.50
54.67
73.50
54.67
75.00
$2.33
79.50
55.33
78.50
78.50
$2.67
06.00
75.50
75.00
$3.33
62.57
$0.00
model. Refer to Table for further explanations.
gumere dupes
31.00
31.00
12:50
33.50
35.00
42:40
67:50
55100
12.50
logical Joluction seveiieeeeeeeeeeee dbrett
31,00
34.00
34,00
38,00
37.00
52.00
13.50
muttirn anibmetic rasiper
75.50
77.50
68.50
71.50
76.00
$0.50
83.00
$7.50
90.50
45.50
33.00
70.00
6500
56.00
$6.00
76.00
59.90
90.50
No
Random
R.S.
B0
dbjst, countng
76.50
75.50
79.50
75.00
74.50
$5.00
06,00
040000
94.50
No 10
59.0
68.6
76.8
+17.8
compeal supaenzz
pengains in
36.02
79.63
$7.18
94.63
15.01
87.18
APE
63.0
68.9
78.4
+15.4
Avcrae iost accwssy (SF1
55.99
62.95
68.90
63.59
68.90
72:23
76.79
78.44
80.17
ProToGi
68.9
72.2
80.2
+11.3
A I0
+9.9
+3.6
+3.4
B.2
Detailed Per-task Results on MMLU
In Table 15, we show the per-task performance breakdown using the PaLM 2 (text-bison-002) target model whose aggregated results presented in Table 3. In Table 16, we show the MMLU results using the Gemini 1.0 Pro target model.
28

---

Table 15: Per-task test accuracy (%) of the PaLM Table 16: Per-task test accuracy (%) of the Gemini
2 target model on MMLU.
1.0 Pro target model on MMLU.
nathed
No BO
Randen.
Randon Search
mathed
No.tD
Raston
Itondiee Searh
Onnclall
No
ProTeCt
N: 10
Polliii
No10
ProTeci
Kincitol
No.0
No30
No10
ProTeCi
Thoret
11:01
11:03
15.03
67:00
3110
11:01
15.03
12:00
27:00
11:00
16.00
13:03
51.05
53.33
10.31
06.11
71.11
03.20
11.45
06.01
05.3
10
04.44
58.29
62.21
03.35
10.35
63.00
65.00
72.00
06:00
65.00
7100
67.00
diicca Jenniedes
67:17
78.05
17:34
09.11
$1.13
72385
16.34
73.35
73.43
73.05
10.57
17:35
calkgt_ Daung
10.35
$4.00
odige Nokker
81.23
06.31
16:31
16:00
5600
51.00
colkgr Sorpr
5730
59303
65,03
16.00
01.00
03181
7600
26.00
56.00
00181
12.01
50.00
collgr
redieie
65.00
73:61
16.17
71.68
64.15
colkgr_ 2tyic
4830
39.90
37.84
$7.84
58.2
00.78
65.23
11.31
56.00
06.67
13.82
62.73
compasr weurly
ooppie secult
60.55
66:76
12.37
16.51
16.11
11.09
11.99
conasestno
12.0
45.25
13.31
57.80
$4.30
31:11
48:25
46.40
46.74
$4.59
3000
18.77
engieentnng
inginering
11.5
51.37
63.51
8531
$1.90
62:35
4541
53.17
20179
45.24
51.50
5357
Screl Juts
17:30
38.80
40.32
13:66
4633
48.41
gioba
goid
Nith_tteel_aa Nokkg
15.45
02.56
38.45
55:45
$1.6
Apft Noleyy
15.1
16.19
56.13
$1.5
15.15
14.32
15.17
23312
65.32
3861
6416
9341
buf huftadbeo stereer
6007
40.13
06.01
00.10
00.00
18.13
ligh
high
wwwpe__sappontteeeeeeoo Jaery
15.09
15.76
73.94
76
11:21
16.35
77.35
71.52
73.6
11203
18.39
Net, actoet poseraity
7121
83:33
18:35
10.35
93.00
8537
11:31
79.80
85.35
5530
1435
17:88
ligh
piliks
high pronrir
33.33
14.25
38.21
7282
13.95
7282
0316
1300)
1662
ligh whood nalleatis
19
53.2
5704
55.19
57.7%
S76
light sdooll
13.20
55.33
01.45
5567
55.56
55.19
ligh
ligh ptooss
17.04
4047
4437
5497
1334
28.54
56.91
53.11
54.30
ligh whood poydudegy
19.27
19.81
00.35
9691
$7.80
90:60
light adaol, posdiing
06.99
63.49
5:11
37.0
01.31
66.79
ligh
ligh
15.6
19:41
10.35
7944
$6.27
15.33
12.84
12364
15.75
ligh. wchod wolld Jourry
17
17.44
00.39
50
81.45
13.04
light sdaod wolld Jonnry
6312
77.64
3606
01.51
63:11
1phig
harra
1252
31.39
51.66
1899
19.39
72.5
15.57
16:39
innedddrd Jse
3605
11.38
76.35
73.55
90
11:00
stensiiona Jen
17.99
74.38
0155
1130
76.96
15.5
logicl Eetatee
12.39
56.23
73.45
lngics Jalacee
17.30
7983
113
madire kunin
57.16
58.93
51.35
02:50
51.99
4111
nechow laning
15.36
50.85
51.65
59.00
5000
77.35
91.45
$2.18
57.33
13.04
19:12
melkal poxtks
62.00
75.00
7600
86.00
$1.00
3700
uedkal poadks
15.00
08:00
73.00
7510
510
16.00
momo dapais
5765
7M39
75.22
72.47
swel Japsio
7181
33.00
mocel sonriis
1160
51.31
61:55
57.8
71.06
6547
nnd sesalos
44.35
50.55
55.85
65.6
51.11
6413
plilosply
65.59
75.88
76.31
7534
24744
16.3
philouply
07.85
00.13
71.36
1653
12.00
16.
pellitin
30.99
75
53.33
7954
78.70
18.09
podisiey
13.27
72:22
76
1903
poocoisnd Jar
$5.59
$2.58
54.50
5.50
$463
$3.45
polosisnal
51.37
53.33
50.65
51.0
50.30
$3.93
pelesdeed nalcci
72.9
71.79
76.34
77.31
78
174L
pofasiceal melkies
16.0
73.15
7654
1245
36.00
11.9
prycusloyy
pullic staiees
51.0
63.64
56.35
09.00
57.37
60.91
pallc
6091
00.00
62.33
65.5
62.5
59.9
accerty Jtadi
16.5
33.55
11A1
81.3
72.00
10.30
08:99
71.84
70A1
12.11
Enigu plicy
11:0
11.00
55.00
$4.00
$4.00
35:0
1600
56.00
50.00
35:00
55.00
$1.00
simdays
49:01
3350
15A1
51.31
5101
45:5
51.11
51.0
1131
51-41
54:32
87:32
18.35
7933
Array anraattayyy 1991
35.77
04.73
17.05
00
73.75
33.31
Asoage And Aww.
7900
66.81
7031
71.56
31.48
11.4
B.3 Additional Visualization of Comparison Across 10 and EO
Complementary to Fig. 3 in the main text, in PaLM 2 model, we show the per-task change of test accuracy before and after applying EO for APE and OPRO in Fig. 9 (comparison between optimized exemplars vs. no exemplars) and Fig. 10 (comparison between optimized exemplars vs random exemplars). We also include a visualization on the effect of IO (comparing optimized instructions and initial instructions) in Fig. 11. We also visualize the Gemini 1.0 Pro results in Fig. 12.
12%
15%
25
25
090
+12%
20
20
+15%
-9%%
Task indox
Taskindex
15
+0%%%
16
10%
+9%
19%
11%
10
10
+28%
-4%
5
63.0
73.1
72.5
50
25
50
75
100
Test accuracy (%6
Test accuracy (%)
Figure 9: Optimized exemplars vs. no exemplars in PaLM 2 model: Comparison of performance of OPRO and PromptAgent (left to right) before (no exemplars) and after applying exemplars found via Mutation. Dashed and solid lines denote the average performance before and after exemplars, respectively. The results using APE, ProTeGi and no 10 are shown in Fig. 3.
29

---

25
25
25
0M0
+1455
20
20
-9%
Task Indee
Task indox
Taskindees indion
0%
12%
-2%
15
26%
-0%
10
+0
10%
+00%
B
+19%
66.9
68.5
731
50
100
Test accuracy (%)
Test accuracy (%)
Test accuracy (96
25
20
20
Task index
Taok indox
15
15
15%
10
-20%
1.47%
27%
20.8 77.3
67.6 72.8
50
75
100
Test accuracy
Test accuracy (%)
Figure 10: Influence of using optimized exemplars compared to random exemplars in PaLM model: Comparison of performance of No 10, APE, OPRO, ProTeGi and PromptAgent (left to right and top bottom) using random exemplars and optimized exemplars found via Mutation. Dashed and solid lines denote the average performance before and after exemplars, respectively..
30

---

-0%
2%
25
+15
-0%
20
+4%
Task index
index
15
15
11%1
1410
-237
10
10
18%
24%
+44%
+2%%
+12%
25
50
75
100
25
50
75
100
Test acouracy (9))
Test accuracy (%)
No
25
44%
PA
+4%
+10%
+12%
-10%
Task Index
15
15
11'
-25%
30%
5
16%
10%
60.3
65.7
100
25
50
75
100
Test accuracy (%)
Test accurscy (%)
Figure 11: Influence of using optimized instructions compared to initial instructions in PaLM model: Comparison of performance before and after using optimized instructions found via APE, OPRO (Top row), Pro and PromptAgent (Bottom row). All results are without exemplars (i.e., "zero-shot" setup). Dashed and solid lines denote the average performance before and after examplers, respectively.
31

---

10%
25
25
10%6
+600
+4%
11%
-14%
Tndeindex
+10%
15
+9%
15
Task
+-2%
10
+419%
10
477%
+5%
47%
+20%
.26%
122%%
63.11
26.8
711
75.8
25
50
75
100
25
50
75
100
Test accuracy
Test accuracy (%)
100%
+05
-1%%
-186
20
-1.75
13.35
Taskindex
Task index
+4%%
15
+7%
15
5.5%%%
+19%
+2%
1010
-000
10
+249%
10
-13
+22%
.4%
+8%
+22%
+239%
25
75
100
25
75
100
Test accuracy (%)
Test accuracy acy (%)
Figure 12: Influence of using optimized exemplars compared to no exemplars and random exem- plars in Gemini 1.0 Pro model: Comparison of performance of No IO (top row) and ProTeGi (bottom row) using no/random exemplars optimized exemplars found via Mutation. Dashed and solid lines denote the average performance before and after exemplars, respectively.
Complementary to Fig. 5 in the main text, we visualize the additional BBH tasks on PaLM 2 in Fig. 13(a), Gemini 1.0 Pro in Fig. 13(b) and Gemini 1.5 Flash in Fig. 13. We also test key findings a representative subset of 11 tasks in GPT-3.5 (Fig. 14. We visualize the MMLU results in Fig. 15.
32

---

Random
Musation
Test acc.
femponii_ pergueees
75
tocking shuffedd objects (7)
90
janunames
Oegical Dopaaaoddtttionn
Poparbetton
100
80
50
80
70
30
Test acc.
geomerric
70
Judgement
Trakking shuffeed
orjects
selent transationn
80
75
Milero
80
60
60
50
60
Test acc.
web of Jes
90
ponguing itpola
word, sorting
75
nu arimect
Willoo
80
80
50
10:
ProtwIl
ES
eea No E5
Random
Mication
causel judgement
@santipution.c.
Test aoc.
70
90
80
70
75
60
80
50
50
50
repertaton
de
Jog ca. dedductiion 07
oglicl_ooouction
movie, recommmnd
multotoopp a frimetti
Test acc.
80
60
90
75
60
40
40
70
50
60
Test acc.
80
nevigate
oolest_counting
ponguing hub
80
ressoring
colored odjects
80
ninunames
60
trendettoo cnior oroor
80
80
50
60
60
Test acc.
90
100
sports understanding
100
rompora_nnuueccs
tracking shuffled abeecs
trockingstuufee shuftee objects (7)
trocking
bradking shuffed
80
80
B0
50
70
Test aOc.
100
word ting
70
75
60
ProTeSi
ES:
Random
Musziion
Test acc.
boclesn_oepression
bookean expressions
cousal Juogement
90
conc sstandinn
dioombiguation. 30
70
foral
geometrkcshapp
60
75
8
60
40
60
Test acc.
8
typersston
80
lnginal desuuctiinn
75
Ingionl (ed.cctionn
95
logical deductionn [3]
movie pecortmenn
00
milbetep, witthmati
8
60
50
90
Test Tec acc.
Dariqatt
100
ocject, counding
100
pengure i.. NoHOO0
85
g.aouut coloeed chyetts
Ful: .erees
80
70
Sa Sondcton
wirre delectionn
8 -
80
60
80
70
Test acc. Test acc.
70 8 8
90
understanding
Simport JeQquEnce
osjects (B)
9eciing shuflee objects (
odjects (3)
stutied
80
100
100
100
90
Jes
word serting
100
60
50
Figure 13: Visualization of per-task performance comparison on (first panel) PaLM 2 (except the first 12 tasks, which are shown in Fig. 5), (middle panel) Gemini 1.0 Pro and (bottom panel) Gemini 1.5 Flash.
33

---

10:
ID
AFE
ProTedi
ES:
N00
No ES
JEE Rendom
Mutotion
Test acc. Test acc.
boclean, express bo
70
causal Judgament
date understandinn
75
geometric shades
50
Jogical dedaction
ogiiall deduction (3)
90
80
75
60
50
80
50
25
50
90
multistep antthmetic
navigate
100
object counting
90
penguins in table
100
Sompora sequences
80
70
50
75
80
75
Figure 14: Visualization of per-task performance comparison GPT-3.5.
alottacctalgeera
bushess_ethies
dirical knowledge
college_bidogy
Test acc.
40
45
70
60
80
70
80
35
50
70
60
co
collegnmmedicingg
collage
90
computer_recure
Test aoc.
65
60
40
80
40
30
Test acc.
80
corcaptusl_hhpiic
60
ecomometrics
80
enginmerng
wiementary mathamitics
55
formil Jagie
70
global facts
60
50
70
50
45
50
60
70
Test acc.
90
70
90
computer
High
90
100g
High
80
60
70
70
80
Nigh macroscconoics
high
Test acc.
60
mathemmticss
80
psychology
statisicc
70-
50
60
50
80
60
Test acc.
90
us. history
world Nistory
80
humann aging
humar soxulity
interaaionnal law
90
Arttsruddnce
80
80
60
70
70
fallcce
jeahine_/earning
management
marketing
medical,genetics
miscelaneous
Test acc.
80
60
90
90
80
80
70
50
75
70
morel disputes
moral scanariss
nutritio
phloxophy
premistory
Test acc.
80
70
80
60
60
70
70
55
50
professional bW
pr professional medicine
professioral paychology
public relations
security studies
sociology
Test acc.
80
80
90
55
80
60
50
70
55
70
80
us_ www.foreign.poo. policy
virology
workd_relgions
Test acc.
55
90
50
80
80
45
Figure 15: Visualization of MMLU tark performance comparison on PaLM 2.
34

---

B.4 Varying Number of Shots
With the advent LLMs that support longer context windows, an alternative approach to performing exemplar optimization is to scale the number of exemplars. Taking into the context of our setup, instead of selecting exemplars from De where & a small value, we can scale le to larger value or even use the entire D. as exemplars. In this section, we perform experiments on the relative merits of the two approaches.
We show the aggregated results in Table 17 and task-specific results for both models in Fig. 16 and Fig. 17. While we find increasing the number exemplars generally leads to improved test performance up to k = 10, further increase leads to performance deterioration. The increase in performance, however, does diminish the importance of exemplar optimization as show that using optimized exemplars outperform many more random exemplars, both at an aggregated level and on spse-ipec level (the instances where optimized exemplars outperform random exemplars of any k, which are marked by magenta lines in Figs. 16 & 17, are more than 50% in both target models) - we argue that this suggests that exemplar optimization remains relevant even in modern I.MMs with long contexts. In fact, we believe that novel exemplar search strategy in such many-shot setup can an important next step, given that the number of possible combinations and permutations of exemplars explodes exponentially for higher kt necessitating more advanced search strategies that would navigate this optimization landscape more effectively and efficiently. We defer thorough investigation of this direction to future work.
Table 17: 3 optimized exemplars outperform 20 random exemplars: average performance across BBH tasks comparing different number of random exemplars per sample optimized exemplars as presented in the paper.
10 EO k
No I0
No I0 Mutation
ProTeGi Mutation
Random
0
5
10
20
3
3
PaLM 2
60.30
62.43
66.91
66.81
67.82
66.91
72.92
77.29
Gemini 1.0 Pro
63.14
68.65
71.12
71.41
71.94
69.17
75.77
79.01
35

---

100
poolean expressions
causa judgementgg
90
date understanding
disambiguation. ga
formal falacies
80
geametric phapes
60
80
Test
80
70
50
hyperbaton
ogical deduction (g ledical_deduction
deduction (?)
lggical ded.dccucinn (3)
movie recommend
auiiiitepaiitmmetic
arithmetic
Test acc. (90)
80
80
70
60
navigate
celnct_
penguins Laenn_____________ table
reasoning about colored objects
ruln,
safent, transiation error detection
(8g
70-
100
80
90
80
80
60
70
70
sports understanding
temporal cemporal_soquenccg.
tracking shuffled objects (5)
tracking shuffled objects (7)
tracking shuffled objects (3)
Text (9)
90
50
70-
70
web, of lies
word_ sorting
10 #thots
10 #shots
10 #thhtts
#shots
Testace, (%)
100
80
75
#thhts
#shots
Figure 16: Per-task results BBH tasks where vary 10, 10, 20) for PaLM 2. The green curves denote the performance of "No IO + random exemplars" combination under different values; the dashed and solid lines correspond the "No 10 Mutation" and "ProTeGi + Mutation" combinations under k = 3, respectively, and these lines are colored magenta when they outperform random exemplars of any k. k = 0 and k = 3 values taken from Table 1 and 2 for PaL.M 2 and Gemini models, respectively.
36

---

poolean exprresioos
causa judgement
date understanding
disambiguation.cm ga
formal falacies
geometric psappss
100
Test acc.
60
80
50
50
hyperbaton
ogical deduction do logical deduction (?)
ogicall deducctutinn (3)
movie_ recommend
puttistepaarihmee arithmetic
Test (90)
80
50
60
60-
90
navigate
Solect_counthgggp penguins Laaiiis_______t__ table
reasoning about colored objects
ruln,
safent, transiation error detecteeonnnnn
Test (8g
880
100
80
60
70
80
70
60
sports understanding
temporal
Comporal_sqqunncer sequences
tracking shuffled objects (5)
tracking shuffled objects (7)
tracking shuffled objects (3)
Text (9))
.90
100
80
80
100
80
80
60
80
60
70-
40
60
web of lies
word sord_soning
10 #thots
20
10 #shots
10 #shats
#chots
Testace, (%)
100
80
60
60
#thhts
#shots
Figure 17: Per-task results on BBH tasks where vary A 5, for Gemini 1.0 Pro. Refer to Fig. 16 for additional explanations.
B.5 Varying Validation Dataset Sizes
ProTeGi + Mutated exemplars
In the main experiments, we used 20%-80% validation-test split for all BBH tasks. In this section, we investigate the sensitivity of the various methods to changing validation data sizes, especially under very data-constrained setups such as very small val-
Avg. test accuracy (%)
75
instruction Mutated exemplars
ProTeGi
Random exemplars
idation data sizes. To do so, we vary the validation
70
split to be (2, 4, 10, 203% and we investigate on the combination of ProTeGi and Mutation, the best-
ProTeGi
No exemplars
performing I0 and EO methods, respectively. We summarize the average test accuracy against the size of validation data split (as a fraction of the total data available) in Fig. 18.
65
Seed Instruction No exemplars
60
0.02
0.10
0.20
We find that optimization- based EO remarkably robust towards the size of the validation size, with the smallest performance drop even with an extremely small validation set (i.e., 2% or 4%, which correspond to 5 or 10 validation samples). On the other hand, ProTeGi dropped to a performance no better than un-optimized seed instruction under such an extremely cota-conntr setup, and even the performance of ProTeGi + Mutation, which performs the best overall in the main text dropped significantly with a small validation data size.
Validation data (as fraction all data)
Figure 18: Average test accuracy with differ- ent validation split sizes with different IO-EO Combinations.
B.6 EO with Reduced Budget
In this section, we show the results of EO with search budget approximately halved (from m 32 to m 16) in Table 18 on top of the seed instructions without instruction optimization. We find that even after halving the budget (i.e., the EO strategies are now half as expensive), seed instruction and
37

---

optimized exemplars still outperform optimized instruction and random exemplars both in terms of average aceuracy and average rank.
Table 18:
Per-task test accuracy (%) of the PaLM-2 halved budget (m 16). The last column (marked by f) is the result obtained by running ProTeGi (the best overall 10 method from Table 1 with 32 with random exemplar optimization, used originally in Pryzant et al. [35].
target model using EO of
Budget BO method
yn 16
mn
Random Search
Mutation
Random
method
No I0
ProTeGi
boolean exprrssiisssssoon
89 00
85.50
84.00
muusaljodgcmeen
68 67
66.67
63.33
date. understanding
79:00
disaniiguatio_.e
71.50
69.50
61.00
formal fallacies
$6.50
$7.00
59.00
geometric shapes
59.50
64.00
50.00
hyperbaton
81.00
90.50
77.00
logical, deduction, live_objects
logical, deductioo seven objects
50.50
$1.00
42.00 22.00
logical deducction three three_toeeeeee objects
82,00
73.50
72.00
movic recommcadation
84.00
83
84.00
watpiepp.rttmmttii_vo two
50.00
48.50
$3.50
navigate
65.00
70:50
65.00
object counting
88.00
96.00
97. 97.50)
penguins, in table
82.91
75.21
84.62
reasuning about, calored objects ohjjccts ruin names
87.50
85.50
89.00
salient translation error detection
58.00
62.00
$6.50
snarks
81.82
82.52
sports understanding
83.00
80.00
91.50
temporal Semporal_sqqeenees sequences
tacking shuffled objects, five, objerts
35.00
32.00
56.00
tracking_shatllee _bjjeleddbbjett.... seven, objects
42.50
43.00
38.50
tracking shuflled objects, three, objects web, lics
99.00
96.00
97.50
wond sorting
83.00
76.00
78.00
Average test accuracy (%)
71.34
71.48
70.81
Average ronk
1,88
2.02
2.10
B.7 Additional Results by Mixing-and- IO and EO
Other choices of instruction optimizers and/or target models. We show results from mixing- and-matching IO and EO in similar manner to Fig. 7 but use ProTeGi (instead of APE in Fig. 7) on both PaLM 2 (text-bison-002 and Gemini 1.0 Pro target models in Table 19, where we that two key findings (that any -o---nddmated outperforms IO and EO only and that optimal allocation completely bridges the gap compared to a more expensive routine) we made still hold, except Gemini model, the optimal allocation occurs 16/16 instead of 8/24. This confirms that the phenomenon we see in Fig. 7 in the main text is not specific to the choice of the target models and/or instruction optimizers but is instead generally applicable.
Table 19: Mixing-and-matchieee and-matching 10 and EO leads to performance benefits under different 10 strategies
and/or target models. Results on BBH tasks using two-stage IO-EO described in Insight 3, $3.2 but we instead use ProTeeGi as the instruction optimizer on both PaL.M 2 Gemini 1.0 Pro target models. The color of the cells denotes the computational costs terms of target model traversals on Dval consistent to Table 1.
Method
ProTeGi Mutation
Model
PaLM 2
Gemini 1.0 Pro
Budget for 10
32
16
8
0
32
32
16
8
32
Budget for EO mgo
16
24
32
32
16
24
32
32
Avg. test accuracy (SK
70.81
73.98
76.41
72.92
77.29
72.72
78.64
77.36
75.77
79.07
Avg.
3.15
2.52
3.17
2.38
2.44
2.79
2.58
38

---

only: Test acc: 62.5%
(4) 10 acc: 75.0%
0: [C-3 -4 x -)
4 -93)
Show timal bracketed between and
Show your rinal anseer bracketed between <andmor> and stanseers
A: Let's carry out the task in structured namer. esstep 1: Simplify the eepression mithin the paret paratthsseat
At Let's out the task structured manner. First, let'x kimplify exprexxion inside jorentheses:
Sirulify the (-3 4 A.4 -8) (-3
(-4 ( 4+ 6...
Sirplity the second expression: (1 +.3. 4
Next, lat's sirplify the expression inside the second set of parentheses:
B1 -3
sadtep 2: Subetstute the siepliffed exprencions back into the asin equationss
(-3 -4) (3 2 4) 18
[C-3 -4 -8) [1 -3 + - -901 (-27 17)
Mo, can sultiply two sirplified expressions:
sadcep twrfore finml exleslationne -27 -18
s*Final Answerses
The f1n0l. answer 15 canseer--8s//nnnee
fimal SanaersSee/ammerr.
(b) only: acc: 51.5%
-66 (-3
brasketed -4)) <ranswer>
finel
0:
A: Lat carry out the task in atructured menner. **tts 1: Staplify the
Show tinal brocketed
and
expressions withan the parontheses**
At Let's think step. (TS 2 -2)
Strplity the (-6)) [-4
exaresston:
-233 +(5 7sansnereiev/anneerr
(e -7](5
2) (7)019 21
b) Serplity the second expression:
f-s -t F2 (-4)) (3 2 4)
0: (f-4 5 -63 f-d 2.2 -413
wegcpp 2: Bubatituse airplifsed expresssons back into
the main equatioane
Show your +inel ansver bracketed between cansvers and
</answors.
6: Let's think step by step. ((-4 -63 -4)]-
sadtep 3: Perfore the finml onloslationn 3 18
(1 -b) 2
-6)
offimel The Fimll
<anover--B</aneeerr>>
<mmeer>-bc/aneeer>>>>>
Figure 19: An example of how 10 and EO may synergize on using PaI. 2 target model.
dataset
Mechanism of synergy between IO and EO. In this section, we give an example of why combined 10 and EO outperform its constituting building blocks in Fig. 19: performing 10 only (Fig. 19a) gen- erates step-by-step text instructions, but without exemplars, the model is prone to making arithmetic errors. Performing EO only (Fig. 19b), on the other hand, subjects the model to the shortcomings of the un-optimized seed prompt, which, this case, generates arithmetic operations-only outputs without textual reasoning. Combined IO and EO (Fig. 19c) addresses both issues, leading to vastly improved performance from both textual reasoning and exemplars.
B.8 Alternative Ways of Combining IO and EO
As discussed in $3.2, we conducted EO after IO in the main text: in our setup, the exemplars are bootstrapped from the model's own predictions, which in turn depend on the instructions given. As such, we performed instruction optimization first on the zero-shot setup and search for the optimal exemplar(s) amongst the predictions induced by the optimal instruction. While we believe this order is the most natural, it is also important to ensure that any findings derived in this paper are not biased by the particular order we chose. Thus, in this section, we also experiment with two alternatives:
1. Inverted order. We first optimize the exemplars based on the seed instruction ("Let's think step by step." for most cases), and then freeze the exemplars and optimize the instruction on both Gemini 1.0 Pro and Gemini 1.5 Flash models. We used APE optimize the instructions and random search to optimize exemplars wiht a total budget of m = 32, which identical to the setup presented in Fig. 7 in the main text.
2. Interleaved. Instead of framing the exemplar and instruction optimization as two-staged approach, better approach in principle is to optimize them jointly. Here, we propose an interleaved approach with the intention of conditioning instructions to the best exemplars discovered so far and vice versa given their inter-dependence. The pseudocode representation of the algorithm in shown in Algorithm 1.
39

---

Algorithm Interleaved instructions and exemplars optimization. Exemplar optimization steps marked in blue.
1: Inputs: Dval, target LLM, optimizer LLM, generations T, instruction pop size [Z], exemplar pop size [81, number of exemplar shots: k.
2: Initialize instructions Io with seed instruction or L..M-sampled instructions; initialize exemplars & <- V. best instruction & exemplar far T2o: Figo None for all generation & {I,...
do
4: # Optimize instructions conditional on best exemplars
Evaluate prompt metrics on Dval: <- E(a.a)=Doal [g (LLM 14 .Fg.)).u] E
Retain 2,, the top instructions Ie in terms of and use the optimizer LLM to rewrite without changing the semantic meaning 1++1 resample(Ze).
7: Update T + arg max1, ((c))
8:
Optimize exemplars conditional on best instructions
9:
if E @ then
10:
Fill Ei+1 with input-output (inc. intermediate outputs) pairs randomly sampled from De( the subset of Dval where the target model answered correctly under the best instruction seen so far: V E, E E441, Ej < (e)")) " Cj De(
11:
else
12:
Fill E441 with [8] mutated copies b randomly replace an input-output pair E with another pair from D.(
13:
end if
14:
Evaluate prompt metrics Deat: <- E(e,))D,,. [ [9(fLL 12. u v)))_)" E& Update E²x, arg ma ((E))1)))
15:
end for
17:
Return instruction instruction-exemplar combination with the highest score.
We present the comparison between the original and inverted orders in Table 20, where we found that the findings (in particular, Insights 2 and 3) interleaved approach, we find it to achieve a test accuracy of 74.6% on 26 BBH tasks on PaLM par with the two-staged approach with m4o 16, mgo 16 thus
affected by the optimization order. For the
2, which is broadly primarily focused on the two-staged approach for its simplicity, and defer a detailed exploration on better ways exploit the interdependence of exemplars and instructions future work.
Table 20: Effect of ordering between I0 and EO on selected BBH tasks of (Ieft) Gemini 1.0 Pro and (right) Gemini 1.5 Flash. Original refers to the IO before EO order used in the main text and Inverted refers to the I0-after-EO optimization order.
Tusk
Combined
Combined Inverted
Tisk
10 only
FD only
Combined Original
Cambined Inverind
baokcan exprosions
70.50
91.00
$1.50
90.50
bootean_ expressios
90.50
99.30
96,00
99.00
cmnal. julgement
54.67
59.33
66.00
66.00
causal Jddgement
68.00
67.33
67.33
63.35
datc understanding
73,50
78.50
79.50
74,50
date understanding
80000
84.50
$4.10
83,00
shapes
33.00
71.00
67.50
84.50
gennetric shapes
$4.50)
$2.00
90.50
81.00
deduction objects
31.00
50.00
47.50
$1.00
lopical deduction cecccooo_eeeeeee objen_s
50.00
70,00
90.50
72.000
logical deduciion thres. objects
49.00
76.50
78.00
70.00
logical deduction dates
91.00
89.50
92.50
93.50
mullivipp arithmetic
77.50
$4.00
70.50
$1.00
mullisipp withmeiic
91.50
91.50
92.50
91.50
usvigat
58.00
85.30
80.50
80.00
asvigne
66.00)
69.00
74.00
85.00
object comning
78.50
95.50
94.00
connting
86.50
95.50
93.50
93.50
pengains table
78.63
$4
93.
84.62
pengains RLA.
95.73
98.87
95.73
94.37
tempund scquenrr
79.50
$4.00
77.50
83.50
temporal sopurors
97.50
99.50
99.90
100.00
Arg. test acc. (
68.98
78.18
78,47
79.69
(8))
79.207
85.75
87.82
86.97
B.9 Case Study on How LLMs Respond to Exemplars and Instructions
Complementary to Insight and Fig. 8, this section, we give some examples on the LLM response when prompted with the optimized instructions and/or exemplars in representative tasks.
40

---

Table 21: Case study on tracking_shie tracking_shuffled.
(7): Top table: the discovered prompts by various methods (only one exemplar is shown under "Best EO" for conciseness). Bottom table: response of LLM to two representative questions in the test set when conditioned on each of the discovered prompt- note that instructions in this case do not fully specify the model behavior. Even when the L.LM is indeed following instructions generate reasoning steps, the effectiveness of reasoning templates vary and LLMs are error-prone and often track unnecessary items. With carefully
selected exemplars, however, the LLM can perform behavior imitation via adapting and chaining the pattern " [[A] and [B] [do something].
[A] had [x] and [B] had [Y] So now, [A] has [Y] and [B] has [x] " to achieve an extraordinary accuracy. Some texts are truncated with
... for better presentation.
Method
Prompt
(Test aoc.)
Base (38%)
Let's follow a step-by-step process.
Best I0
Let's carefully analyze the question and dissect our reasoning process. order to do this, we will need to track multiple pieces of information over time. We will also need to pay attention to the details of each swap that occurs.
(52%)
Best EO (85%)
Q: Alice, Bob, Claire, Dave, Eve, Fred, und Gentrude are dancers at square dance. At the start a song, they each have partner: Alice dancing with Ophelia, Bob dancing with Melissa, Claire dancing with Jamie, Dave dancing with Sam, Eve is dancing with Patrick. Fred is dancing with Rodrigo, and Gertrude is dancing with Karl. Throughout the song, the dancers often trude partners, First, Dave and Claire switch partners. Then, Alice and Eve switch partners. ... Finally, Dave and Alice switch partners. the end of the dance, Fred dancing with Options: (A) Ophelia... ... (G) Karl
Show your final answer option bracketed hetween <answer> and <lanswer> at the end.
A: Let's follow step-by-step process. Let's follow the steps one by one:
1. Dave and Claire switch partners.
Dave was dancing with Sam, and Claire was dancing with Jamie. So now, Dave is dancing with Jamie, and Claire is dancing with Sam.
7. Dave and Alice switch partners.
Dave was dancing with Karl, and Alice was dancing with Patrick. So now, Dave dancing with Patrick, and Alice
dancing with Karl.
At the end of the dance, Fred is dancing with Melissa.
<answer> (B e///sswer>>
Question
Responses
Base
Best IO
Best BO
Alice, Claire, Dave, Eve. Fred, and Gistrade are holding white elephant gift exchange. At the start of the event, they are each holding a present different color: ice has pink ball, Bob has red present, Claire has blue present, has present, white yellow present, Fred present, and Gertrude has a black hall, As the event progresses, pairs of people swap gifts. First, Alice and Claire swap their gifls. Then, Dave and Bob swap their gifts. Finally, Claire and Alice swup their gilts, At the end of event, Alice
swap their gifts: Alice now has the blue present, and Claire has the pink hall.
Alice
Alice and Claire swap:
Alice and Claire swap their gifts. Alice had a pink ball, and Claire had a blue present. So now, Alice has the blue presens, and Claire has the pink hall.
(pink ball) >> Claire, Claire (blue
present) -a Alice.
2. Dave and Bob swap their gifts: Dave now the red present, and Boh has the brown present.
Dave and Bob swap: Dave (brown present) -> Bob, Bob (red present) -> Dave.
2. Dave and Bob swap their gifts. Davce had a brown present and Bob had
Dave
Claire and Alice swap their gifts: has the yellow present, and Alice has the black ball.
Claire and Alice swape Claire (pink ball) -s Alice, Alice (yellow presen)) -> Claire. At the end of the swaps, Al- ice has the yellow present.
present. So has the red present, and Bob has brown present.
At the end of the event, Alice has the Geelanswers black hall.
7. Claire and Alice swap their gifts. Claire had the blue preseni, and Al- ice had the yellow present. So now, Claire has the yellow present, and Alicz the blue present.
sanswers Binsswsssdeeemmweeer.
end ice bas the bluc present.
event,
EunsserstUUyeinssees
Q: Alice, Bob, Fred, Gestnide are same ream in soccer match. At the start of the match, they are each signed position: Alice play- ing choerlender. Bub is playing right middielder, Claire is play ing right winger, Dave is playing center mid-- Gelder Eve playing fullback, Fred is playing striker, and Gestrade is playing goalkeeper. As the game progresses. pairs players swap positions. First, Bab and taae positionns. Then, Claire and Fred trude positions, Finally. Hoh and Dave made posi- the Bub is playnng Options (A) choerleader
Dave, Eve,
At start match, Bob play- ing right midfielder.
and Alicc trade positions:
1. and Alice trade positions
hecomes the cheerleader, and Alice becomes the right midfielder. 2. Claire and Fied trade positions: Claine hecomes the smileer, and Fred becomes the right winger.
Boh was playing right midtielder, and Alice was playing cheerleader So now, Boh playing cheerfeader,
1. Bob and Alice maae positions.
Now, Bob is playing cheerfender and Alice playing right midfieldrr.
and Alice playing right midfielder.
2. Claine and Fred trade positions. This does not affect Hob's position.
2. Claire and Fred track positions.
Claire was playing right winger, and Pred was playing striker. So now, Claire playing striker, and Fred playing right winger.
7. Bab and Dave trade positions:
tadd positions. Now, Bob playing center mid- feeldr and Dave is playing right mid- Geller.
Bob and
becomes center midlsckler,
and Dave bocomes the right mid- fielder.
At the end of the swaps, Bob is play- ing center midlielder.
7. Bob and Dave trade positions. Hoih play ing cheericuder, and Dave was playing right midfielder. So now. Bob is playing right mid- tielder, and Dave playing cheer- leader
At end the match, Boh play- ing center midtielder.
Al the end of the mutch, Bob is playing right midfielder. Brstmmwwrr
(G) goalkceper
41

---

Table 22: Case study on web_of_ _ileses: Top table: discovered prompts. Bottom table: response of LLM to two representative questions in the test set when conditioned on each of the discovered prompt. The winning pattern that the LLM copy behavior from "[A] says [B] [lies/tell]
the truth]. So [A] [lies/tells the truth]" to achieve a near-perfect accuracy.
Method
Prompt
acc.)
Base
Let's think step by step.
(55%)
Best IO (82%)
Let's think step by step. In each step. we will look at the statement made by each person and determine whether they are telling the truth or not, If person is telling the truth, we will continue to the next step. If a person is lying. we will stop and conclude that person who said the previous statement is lying.
Best EO (97%)
Question: Maybelle tells the truth. Antwan says Maybelle tells the truth. Audrie says Antwan tells the truth. Ryan says Audrie lies. Delfina says Ryan lies. Does Delfina tell the truth? Show your final answer (Yes or No only) bracketed between <answer> and <fanswer>.
A: Let's think step by step. Let's think step by step. Maybelle tells the truth.
Antwan says Maybelle tells the truth. So Antwan tells the truth.
Audrie says Antwan tells the truth. So Audrie tells the truth.
Ryan says Audrie lies. So Ryan lies.
Delfina says Ryan lics. So Delfina tells the truth.
<answer> Yesc/answer>
Question
Responses Hest 10
Rase
Best BO
Question: Jamey tells the tuth, Del- says Janey
If Conception tells the truth, then Teressa lics.
Jamey tells the truth. Dellina says Jamey lies. Dellinu telling the truth, then Jamey is lying. which contrndicts the given statement. Therefore, Dellina ly- ing.)
Jamey tells the truth.
says Del- (ina tells the truth. Teressa stys Vina rells the muth. Conception says Ter- essa lies. Does Conseption tell the bruth?
Dellina says Jamey So Delfina lies.
If Teressa lies, then Vima tells the muth.
Vina says Dettina tells the muth, So Vina lics.
Vina tells the truth, then Dellina
the truth.
Teressa says Vinu tells Teressa lies.
Delfina tells nuth, then Jomey lios.
says Deltina the truth. (If Viza telling the truth, then Dellina is telling the truth, which contradicts the previous conclusion. Therefone, Visu is lying.)
Conception says Teressa lies. So Conception tells the truth. Kanswers
But Jamey tells the truth, So, Con- ception tell the muth.
canssersYeschusseer Yescclunswers
The final answer
NowterrNetanswers..
Teressa says Vina tells the truth. dT Teressa telling the truth. then Vina telling trulh. which contradicts the previous conclusion. Therefore, Tercesa is lying.)
Conception says Teressa lies. (If Conception telling the Teressa telling the truth, which contradicts the previous conduusion. Therefore, Conception lying.) final
answer
Question:: Kandi tells the truth. Ryan says Kandi lies. Michaela says Ryan tells Michacla lies. Andree says Sima tells the trath. Does Andree tell the truth?
If Andree tells the truth, then Sima lies. If Sima lies, then Michnela tells truth.
Stop I: Kandi tells the truth.
Kandi tells the truth.
Step 2: Ryan says Kandi lies, (Con- tradicts Step
Ryan says Kandi lies. So Ryan lies.
says
tells truth, then Ryan lies. If Ryan lics, then Kandi tells the truth. But Kandi tells the truth the first place. This contradiction.
Ryan lying.)
Michaola says Ryan tells truth.
Step 3: Michsulaa says Ryan tells the truth. (Contradicts Step 2, so Michaela lying.)
Miclinha lics.
Sima says Michaela lies. So Sima
tells the truih.
Step 4; Sina stys Michacla [C (Contradicts Step 3, so Sima ly- ing.)
Andice Simn tells
Therefore, Andree does not tell the truth.
Andree tells the truth.
The final answer is Canswers Yevellsswessssrrrrr
CamewersNoccanswee
Step S: Andree says Sima tells the truth. (Contradicts Step 4, An- dree lying.) canswersNucttnssee
B.10 Comparison Against PromptBreeder
In this section, we compare the two-stage EO-IO combined search mentioned in $3.2 against PromptBreeder [14], the SoTA method that has the option of optimizing both instructions and exemplars. contrast to our method which is a simple combination of APE (for instruction optimization) and mutation (for exemplar optimization), PromptBreeder features a much expansive search space introduced by its different mutation operators on the instruction and the meta-prompts (referred to as hyper-mutation in Fernando et al. [14]). As such, the algorithm requires many more iterations before convergence - for our experiment, we use a population size of 40 and allow for 10 generations. Since Promp(Breeder uses binary toumnaeett evolution for each generation which will result in half of its population being replaced with new prompts requiring evaluations Dval. there total 40 + 10 x (40/2) 240 traversals the validation set D which is approximately an order-of-ne magginiuude larger than the budget used in this paper and the discovered prompts can be
42

---

deemed as approximation of the performance "upper bound" when SoTA optimizing agent is left to freely explore an expansive search space with ample search budget. As comparison, we run our two-staged EO and IO algorithm for 100 iterations, with first 25 iterations used for 10 and the remaining 75 for EO. Due to the much larger computational costs, we run the comparison on four selected tasks: movie _ recommendation, multistep_ ar www.trttiiittttie.eeeeeeeeeeeeee two, pepectt_cuuniinn and name_
Table 23:
Comparison of the simple two-stage IO-EO algorithm introduced in $3.2 (with a budget of 100 validation set evaluations) and PromptBreeder (with budget of 240 validation set evaluations) on representative tasks using PaI.M 2 bistt-iiso--0222 model.
Dataset
Best validation acc. (%)
Test acc. (%)
Ours
PromptBreeder
Ours
PromptBreeder
movie recommendation
94.00
94.00
89.50
89.00
multistep, arithmetic_tww two
86.00
84.00
79.00
74.50
Dbject_counting ruin names
100.00
100,00
99.50
98.00
86.00
92.00
82.00
87.00
movie_reedd recommendation multistep_ arithmetic
two _t multistep_arthmmetic
object_cooue counting
juin_names
Validation acc. (%)
100
75
75
50
50
50
50
PromptBreador
25
Ours
25
100
200
100
200
0
100
200
100
200
Number evaluations on Deat
Figure 20: Comparison of validation accuracy vs. number evaluations on the validation set between two-stage TO-EO search and PromptBreeder. Scattered points denote the performance of each evaluated instruction and the line denotes the best instruction up to that point. Note that PromptBreeder uses 2.5x more search budget.
We summarize the results in Table 23 and Fig. 20. It worth noting that our simple two-stage search performs on par or better than PromptBreeder in 3/4 tasks in terms of both validation and test metrics while using less than half of the budget and using much simpler instruction optimization heuristic. Given that the chief difference between our algorithm and PromptBreeder is the explicit iterative optimization for exemplars (noting that the "context shuffling" routine used in PromptBreeder, which permutes the exemplars in the prompt, relies purely on stochasticity), this again suggests that the importance of exemplar optimization may actually outweigh complicated instruction optimization as we highlighted in Insight 2 in $3.2 - in fact, even the the PromptBreeder authors themselves remarked in App. that "... We find that in the few-shot evolution case, the contexts dominate, and often the task-pron pask-prompts drift into nonsense. They are less critically determining of fitness than the evolved contexts.".
B.11 Additional Examples of How SoTA 1O Uses Exemplars
Complementary to Insight 4 in $3.2 of the main text, in this section, we show additional examples illustrating how SoTA IO techniques may spontaneously and opportunistically use quasi-exemplars. For selected examples, we also perform ablation experiments similar to the experiment in Fig. 8 where we either retain or remove the identified quasi-cxemplars to study the effect of their presence on the final performance and show the results in Table 24 it is further worth noting that in multiple cases, the instruction optimizers discovered instructions that consist of quasi-exemplars only (c.g.. multi ari sarihheer thmetic two and movie recommendation in Table 24), which further corroborates our point made in Insight4 4.
43

---

Table 24: Additional examples where SoTA instruction optimizers generate quasi-exemplars (high- lighted in yellow) discussed in Insight 4 in $3.2 in the main text. Where possible, we perform similar ablation experiment as in Fig. to either retain the quasi-exemplars or remove the quasi-exemplars and re-run evaluation on the test set. It is worth noting that in several cases, the instruction optimizer generates instructions consisting of quasi-exemplars only (marked by asterisks in the table below)
where it is not possible to perform the ablation experiments.
Datasct
Prompt
Test accuracy (%)
Full prompt
Quasi- exemplars only
Removing
quasi-
exemplars
** Watched Movics:
The Shawshankk Redemppion (1994)
The Godfather (1972)
Dark Knight (2008)
Pulp Faction (1994)
Pootential Recommendation:**
Green Mile (1999)
2. One Flew Over the Cackon's Nest (1975)
Stlence the Lambs (1991)
4. Fight Club (1999)
*Hint:"** ** The recommecdddd movic is psychological thriller that
human nature leatures memorable performance from its lead actor.
explores
molti_arsthmetic.re
**Solve for x:
74.5
S$2(x + 15$5
**iin:=** Distribute the 2 the expression inside the parentheses. then combinc terms:
**Splution:** S$2(X 158$
552x 6-
$$2x + 1 1555
S$2x = 14SS
5ix
noia_names
**Instrctioon Mutant:*
87.0
When faced with a challenge this, where you need character edit that humoroudyy changes the meaning of an artist, hand, movie name, here's creative appreach help you solve it:
one-
I. **Think Beyond the Obvious:** Don't 't just focus on first letter or the last letter. Look for opportunttes to change the meaning by altering any churacter within the
2. "ePla with Homoayms:** Explore words that sound similas but have diffirrntt meanings. For example, "Beatles" could become "Beetles."
"Meallliaa" could become "Meathalls.!
"*Consider Punc:** Puns can be great way to add humor. For instance, "The Roiling Stones" could become "The Rolling Scones." **Looo for Visual Changes:** Sometimes, simple change panctu- ation or capitalization can create a humorous visual effect. For example. "The Cure" could become "The Cure?"
5. "Ilcoorpoaatc Pop Culture: Reference current events, popular trends, or well known phrases to add topical twist. For instanze, "Sur Wars' could hecome "Star Whores!!
6. **Don't Be Afraid to Be Silly: The goal is to make peuple laugh, don't he afraid to come up with something wacky and unexpected. Remember, key be creative have
Mbject_.cunt-ing
Let's think step by stop: First,
84.0
72,5
80.0 (-4.0)
understand the problem. We have collection of possessions that a person has along with their quantitics. We need to determine the number of a certain objecuftem class.
(-11.5)
Next, need idcatify objocthitm class for which we want determine the number.
3. Once we have identified the object/item class, we need find all the possessions that belong
class.
4. We then need to sum up the quantities of all the possessions that objectsiem class.
helong
sum the quantitics will give us the total number of possessions that belong to the objectfitm class.
Let's take an example understand this better. Suppose have following collection of possessions:
carr:
"house":
"hicylle 2,
'og': 1,
2
We want determine number pets the person In this case, the object/item class is "pets". The possessions that belong to this class are "dog" and 'cat". The quantities of these possessions are and respectively. The sum these quamtities Therefore, person has 3 pets,
We use the appronch determine the number of any other object/isem class the collection.
44

---

C Best Prompts Discovered
In this section, we present the best optimized prompts discovered by any method on each of the BBH tasks studied.
Q: not False or False is Show your final anaser stepwise approach, not { False ) False )
only) bracketed betueen <ansuer> </anager>.
A: Let
Show your final answer (True or False only) bracketed betueen <anewer> and </ansker>. stepwise approach.
A: Let's
1. not False True <ansuer> Truee/rru///eee //averr
9: True and True and True )
Show your rinal anaver (True or False only) bracketed betusen <ansuer> and </anaxer>. stepwise approach. True
Show your final ansver (Trua False only) bracketed betueen <ansver> and </anaxer>. Let's use a atepvise approach.
not True is False.
2. True and False is False.
Falsa and True Falae.
<ansver Falas </anwer/)
False is Show your final ansver (True or Palse only) bracketed hetueon <ansver> and </answor>, atepvise approach.
0: True and Palse
and False False ) Show your final ansver (True or False only) bracketed betveen <ansver> and </anawer>, stepwise approach.
1. not False 18 True
2. True or True is True.
<ansver >rreerrTrrrr/// True</ansss
Listing 1: boolean_expressione. OPRO + Mutation. Test accuracy 95.0%
typical parson anaver A lieutenant talking with sergeant. The lieutement gave the order: 2 'Send your aquad to the top of Thompson Hill.> The sergeant said: *But if I send ny aquad to the tap Thompson Hill, Some of then will surely be killed!' The lieutenant auswered: 'Look, I knev that line
folloring queations
moving
eneny
fire.
they '11 don't care at all about what happens to soldiers. All I care about is taking control of Thonpaon Hill.' The squad vas sent noved into the eneny line of fire, and
some
killed.
top of Thompson Hill. As then
expected, the soldiere vere killed. Did the lisutenant intentionally put the soldiera in the line of fire?
Options:
Yes
Show your final ansver (Yes or No only) bracketed betueen <ansver> and </anaver>. Let's think step by step. Yanaver>Yes</amnn ansker>
Q: How sould typical persen ansver each of the folloving questions about causation? sees some puppies playing
drivesay again rat poison that ahe thinks will work
She
decides to to the hardvare store to buy on the puppies. l the pulls out of her garage, the wheel elips in her hand and she drives off the side of the drivevay. All the puppies crushed and killed under the car. With the puppies eliminated, Jen doasn't need to go to the hardzare store.
intentionally xill puppiee?
Options:
No
Show your final ansver (Yee or No only) bracketed betveen <enever> and </answer>.
A: Let think step step. Vansver>>l</anssue
D: Hou gould typical person anaver each of folloving questiona about causation? Billy and Suzy work for the conpany. They vork different
and both of then conetines need to access the central conputer of the company. Mobody at the company people logged into the central computer the time. none apan emaila containing inportant cuatoner infornation are innediately deleted conputer. order to sure
that
person always
45

---

incoming phone calls, company issued the following official policy: Suzy is the only ane permitted to log into the central compater in the mornings, Billy
only pernitted log afternoons. Billy is not permitted to log into the central computer in the morning. Today at an, Billy vork emails containing inportant custoner information deleted from the central computer. Did Suzy cause the central computer to delete some work emsils containing important custoner information?
central computer
Suzy both log
computer
same tine.
Innediately,
Options:
No
Show your final ansver or only) bracketed betueen <ansver> and </anaver>.
A: Let's think step step. <answer >
Listing 2: causal maanslljjudgmenti No 10 + Random Search. Test accuracy = 68.7%
today 6/18/2019,
today 6/19/2019. correct.
What is the date 10 days ago in NM/DD/YYVY?
Options:
(④) 06/08/2019
(B) 08/09/2018
06/09/2019
(D) 05/19/2019
06/09/2044
(F) 05/15/2019
Show your final ansver option bracksted between <ansver> and </ansver> at the end.
Lat think atep step. Today 6/19/2019. daya ago would be 6/19/2019
6/9/2019.
<anever>
9: Jane eas born on the last day of Faburary 2001. Today is har 16- ye-year-ee old birthday. What date ago NM/DD/YYYY?
Options:
11/12/2016
(B) 01/21/2017
01/14/2017
(D) 01/28/2017
(E) 02/03/2017
your
option bracketed
A: Let's think step step. Jane was born on the last day of February in 2001, which is February 28, 2001. Today
birthday, which A month from today vould be January 28, 2017. <ansver>(B) </ansur>>
February 28,
Q: The concert vas scheduled to be on 06/01/1943, but was delayed by one day to today. tomorrow NM/DD/YYYY?
Options:
(A) 05/03/1943
(B) 03/15/1943
(C) 08 08/03/1942
06/02/1943
(E) 07/04/1943
05/08/1943
Show final ansver option bracketed betreen <anaver> and </ansuer> at the end.
A: Let's think step by step. The concert was scheduled to be on 06/01/1943, but was delayed by
So today is 06/02/1943. Tomorrow will be 06/03/1943.
<ansver (A) </ansue>>
Listing 3: date untersundesttnndigg: No 10 + Mutation. Test accuracy = 81.0%
D: In the folloving
explain the antecedent of the pronoun (vhich thing the pronoun refers to), or state that it is anbiguaus.
The writer tried foal the salesperson and told
story.
Options:
story (B) Told the salesperson fake story
(C) Ambiguous
Show your final
option bracketed betweon <ansver> and </answer> the end. A: Lat's think step by step. The pronoun "hin" refers to "the salesparson'.
<ansver> </anuur>>>>
0: In the following sentences, explain the antecedent the pronoun (shich thing the pronoun refers ta), that it anbiguous.
46

---

Sentence:
The chief encouraged the vriter because she
baok.
Options:
good book
(B) The oriter arote good book
Anbiguoua
Show final ansver option bracketed between <anaver> and </answer> at the end. A: Let's think step by step. The prenoun "she" refers to the writer. The antecadent of the pronoun "she" "the vriter'. <ansver >B) </annuer>
Q: In the following sentences, explain the antocedent the (vhich thing the pronoun refers to), or state that it is anbiguous.
Sentence: The reporter
chef discuss their favorite dishes.
Options:
reporter favorite (B) They discusa the chef's favorite dishes
They
Ambiguous
option bracketed between <ansver> and </answer> the end. A: Let 's think step by step. The pronoun "their" refers to both the reporter and the chef
Show your final
The rinal ansuer is: (anaver>(() Anbiguoua</ </aavve>>>
Listing 4: disambiguation.cm No 10 Mutation. Test accuracy = 71.5%
For each argunent, classity daductively valid invalid based the explicitly stated prenises. A deductively valid argunent is one where the conclusion premises.
argunent deductively invalid, even when the premises are true, the conclusion can still be false.
logically
Be cautious, sone argunents have premises that are nonsensical or contradictory. In such cases, simply facus on deternining shether or not the conclusion is supported their
by premises, regardlesa Read each argunent and provided prenises carefully and attentively. If the argunent can based on preaises, respend otherwise, ansver "valid."
denonstrated
"invalid,"
Fred cousin Robart?
related to Bab? large familiea, track of all one's relatives. The following argunent seeks to clarify some such relations: First all, every great -rrddgrrandreeer Chad is sn Douglas
difficult
every schoolnate neither a cousin of Henry nor ancestor of Douglas. Rence, no great gradd-grandetther of Is the argument, given the explicitly stated prenises, deductively valid invalid? Options:
Henry.
is
vvalid
invalid your ansver (valid or invalid only) <ansver>vali
Be cautious,
may have premises that nonsensical contradictory. In such cases, simply focus on deternining shether or not the conclusion is supported by premiees, regardlegs Read each argunent and prorided prenises carefully and attentively. If the argument can demonstrated
their content.
based on premises, respond
"invalid,"
othervise, anaver "valid."
"Here comes perfectly valid arguneat: First of all, every loyal buyer of Tocca soap is occasional purchaser of Bentley Organic soap. Bext, being loyal buyer of Tocca soap is sufficient for being a frequent consuner of L'Oreal shampoo. Plus sone zegular user not frequent consuner of L'Oreal shampoo. Ve may conclude that not every regular soap."
seap not occasional purchaser Bentley Organic soap
Laver soap loyal buyer Ia the argunent, given the explicitly stated premises, deductively valid invalid? Optione: valii
invalid your ansver (valid invalid only) bracketed <ansver waand////o
</ansver>.
argunents may premises that nonsensical contradictory. In such caaes, simply focua on daternining whether or not the conclusion is aupported by premises, regardless Read each argunent and provided prenises carefully and attentively. If the argument can demonatrated
Be
their
invalid based on the premises, respond with "invalid," otherwise, "valid,"
47

---

"Here perfectly valid argunent: Pirst prenise: Vhatever ingredient of Concealer is at least one of these: an ingredient of BEART SHAPED BALH, an ingredient Goth Fairy
wimp. Second prenise: Being an ingredient of Clarifying Nask is necessary for being an ingredient of doth premise: Being ingrediant being an ingredient of Clarifying Mask. Fourth premise: Every ingredient of Love Like A Wimp is an ingredient of Clarifying Maek. So, necersarily, being an ingredient of Concealer sufficient for being ingredient of Clarifying Nask." Ia the argument, givan the explicitly atated pranisea, daductively valid or invalid?
ingredient
Fairy.
sufficient
Options:
vaaldd
invalid
Show final (valid or invalid only) bracketed between <ansver sasserrvalid<ssnsvorr
<answer> and </anaver>,
Listing formal_fallacies ProTeGi + Random Search. Test accuracy 63.0%
9: This SVG path element <path d=*H 25.00 00, 38.00 L 89.00, 38.00"/> draus
Options:
(4) circle
(B) heptagon
bexagon
(D) kite
(F) octagon
(G) pentagon
(H) rectangle
(1) sector
triangle
Show your final answer option bracksted betxeen <ansver> and </ansver> at the end. should proceed
sequential manner. should pracead saquential nanner. The path elenent starta at point (25, 38) and drazs line point (89, 58). Se, it is a line.
(ansuer>(E) </aauurr
9:
path element <path d=*H 21.22,43.76 13.59, 13.59,7.92 7..93 35.84 49. 35..84 21.22 23. 73.77/// drave
Options:
circle
(B) heptagon
bexagon
(D) kite
1ine
(F) octagon
(G) pentagon
restangle
(I) sector
triangle
Show your final ansver option brackated betzeen <ansver> and </ansver> at the end.
A: Ve should proceed in sequential manner. &: We should proceed in sequential nanner.
The first part of the path elenent, "K 21.22, 21.22.43.76 43.76 13.59,7.93", dravs line from the point (21.22,
point
7.93).
second part 7.93) to the point (49.15, 35.84)
path element, 49. 19.16.35.84"
point (13.69,
The third of the path element. "M 49.15,35.84 21.22,44.76"", draws line from the point (49.16, 35.84) back to the point (21.22, 43.76)
This path elenent draus triangle. <J)s <a)aaaa
Q: This SVC path element <path d="M 53.07, 61.70, 62.83, 22.38, L 66.68,29.1
43.33 07.88. 38.28 25.14 52.62, 23.25 03.07.6.04"/
Options:
circle
(B) heptagon
(C) bezagon
(D) kite
(E) 1ine
octagon
(G) pentagon
rectangle
(I) sector
(J) triangle
48

---

option bracketed between <ansver> and </answer> the end. A: Ve should proceed in a sequential manner. The path starts at point (53.07, 6.04) and goss point (61.78, 11.14). This
Show your final
segnent.
Next, it goes to point (62.83, 22.38) This is another line segment.
continues point (66.68, 29.10). This
segnent.
Then, goes to point (63.41, 40.33) This is Iourth line seguent.
Next, goes to point (49.09, 37.83) This is fiith line segment.
It continues point (38.28, 25.14). This
sixth line segment.
Then, it goes to point (62.62, 23.25) This is neventh line segnent. Finally,
starting point (53.07, 5.04). This
eighth segnent.
So, the path consista eight line segmenta that connect nine points. This forms a polygon. To deternine the type of polygon, need count the number of sides. Since there are eight sides, the polygon is an octagon.
<ansver (F) (F)</anaer)>
Listing 6: geometric_shaper APE + Random Search. Test accuracy = 88.5%
Lat think step step.
correct
adjectivee
1. Opinion
3. Shape
4. Color
Origin
6. Naterial
Purpose
Nox,
options:
(A) little black driving ship
(B) driving little black ship
The correct anaver is (A) because it follows the correct order of adjectives.
<ansver>>A) anasvr
Which sentence has the correct adjective order:
Options:
large archaic Indonesian
(B) large red Indonesisn archaic sock your
</anever> end. The correct order adjectives 18:
option bracketed
1. Size
2. Shape
Color
4. Origin
Naterial
Purpose
Now let'ss look the options:
(A) large archaic red Indonesian sock large
correct ansver
correct adjectives.
<anever> a/never>
Which sentence has the correct adjactive order:
Options:
(A) Gernan gray lovely ship
(B) levely gray German ship
Show your final
option bracketed betweon <ansver> and </answer> the end.
correct
adjectives
1. Opinion
Shape
4. Color
Origin
6. Naterial
Purpose
Now, let look at the options:
(A) Gernan gray lovely ship
(B) lovoly gray German ship
49

---

rightnost The correct anaver is (B) because it follows the correct order of adjectives.
(ansuer>(()) </anneer)))
Listing 7: hyperbaton: PromptAgent Random Search. Test accuracy 90.5%
objecta arranged rixed order The atatements are logically consistent vithin esch paregraph. In golf
following paragraphs describe
tournament
Any, and Eve Any finished below Dan. Mel finished first. Joe finished above Dan. Eve fimished last.
golfers:
Optione:
(A) Hel finished second
(B) Dan rinished second
Any rinished
(D) Joe finished second
second Show your final answer option brackated betreen <ansver> and </ansver> at the end.
A: Here's the thought process used to solve the problen. **Thought process:**
1. Nel rinished first.
finished
3. Amy finiched belew Dan. above
Based on these statements, ve can conclude that:
Nel finished first. fifth. Dan finished third.
Any
Joe finished second.
**Ansurr canauer (aaser>())</aaaeee))
Q: following paragrapha describe set
objacta arranged The atatenents logically consistent vithin each parsgraph. On shelf, there are five bocks: green book, an orange book, gray book. yellov book, and green
purple
rightnoat. The gray orange book. The purple book is to the left of the yellow book. The purple book is right
right
gray
Options:
The green book is the leftnoat
(B) The orange book is the leftmost
(C) The gray book is the leftnost
yellov book
(E) The purple book is the leftmost
ansver option bracketed A: Hare 's the thought process used to salve the problan. The green book ia the rightnost.
your
<ansver> </ansver>
The gray book is the right of the The purple book is to the left the yolloz book. gray
book.
purple
right
green second statement, knov that the gray book is to the right of the orange book. This means that the orange book must be to the left of the gray book. Fron the third statement, know that the purple book is the left of the yellow book. Fron the fourth statement, ve know that the purple book is to the right of the gray book. orange purple,
know statement,
the gray Therefore, the order of the books from left to right is: orange, yellos, grean.
purple
<anever> < s/aneer>>
Q: The following paragraphs each describe set of five objects arranged in fixed order The atatementa logically constatent vithin each paragraph.
golf tournsment, there five golfers: Any Mel Rob, Joe, and Ads. Joe finished second. Joe finished belov Any. Mel finished second-to-le Ada finished last.
Options:
(A) Any finished first
Mal
(C) Rab finished first
rinished first
(E) Ads finished first
Shox your final ansver option bracketed between <ansver> and </ansver> at the end.
50

---

A: Here's 's the thought process used to solve the problen. SeSt Understand the information given in each paragraph..*
The first paragraph describes golf tournament with five golfers: Any, Nel, Rob, Joe, and
..Seep 2: Identify the key information related to the order of finish.*.
Joe finished second.
finished
Any
Nel finished secand -taandtt--ast..... finished laat.
Step 3: Use the given information to deduce the order of finish.**
Based on the information provided, se can conclude the folloving:
Ada finished last.
Nel finished second-to-last,
Joe finished second.
kny finished above Joe.
**Seep 4: Determine the golfer vho rinished *e
Since Any finished above Joe, tho finished second, and other information 1s given about the golfers vho finished shead Joe, ve can conclude that Any finished first
*4Step Select the
option.**
option corresponds Any finishing first
*+Final Answer:** <amexer> </enswer>
Listing
wwwpdddioddddiiiiieeeeeeeeeeeeee lirollon_ivveeeeeeeleeeeeeeeeeee fivecttectteeee OPRO + Random Search. Test accuracy 62.5%
Q:
The follewing paragraphs each describe set of seven objects arranged in fixed logically consistent sells seven fruits: pluns, kivis, cantaloupes, pears, vaternelone, apples, and loquats. The watermelons are more expensive than the centeloupes. The apples are expensive.
order. The statements
each paragraph.
cantaloupea. The The loquate are less expeneive than the kivis. The spples are more expensive than chird-chespost. pluns
loquats less expensive
the
loquats.
cheapest.
Options:
The pluma are cheapeat
(B) The kivis the cheapest
(C) The cantaloupes are the cheapest pears
cheapest (E) The vatermelons are the cheapest apples
cheapest
(G) The loquats are the cheapest
Show your final ansver option bracketed between <anaver> </ansver> at the end.
A: need think step-by-step. Step Identify the expensive fruit.
The vaternelons are the second-nest nost expensive. So the nost expensive fruit must be either cantaloupes
kivis.
Step 2: Identify cheapest The pluna are the cheapeat.
Step 3: Deternine the order of the fruits fron expensive least expensive. The most expensive fruit is either the cantaloupes or the kivis. The vatermelons are the second-nost expeasive. loquats are less expensive than the kivis. The apples are more expensive than the loquats. The loquats
apples
expensive
cantaloupes.
third- cheapest.
Step 4: Identify the cheapest fruit. The pluns the cheapest.
cansver>kc/anover
D:
The folloving paragraphe each describe set of geren objects arranged in a fixed order. The statements logically consistent vithin each paragraph. Da shelf. there are aeven books: purple book, a broun book, a red book, blua book, a gray left
book, black book. The gray baok is the second from the left. The black book is to the left ot the gray book. The blue boo2 is to the left the red book. The blue book is the second from the right. The book is to the left of the purple book.
and green book.
green
Options:
51

---

(4) The purple book the second fron the right
(B) The brown book is the second from the right
(C) red
right (D) The blue book is the second from the right
from (F) The black book is the second from the right
gray book
right
(G) The green book is the second from the right
Show your final
option bracksted between <ansver> and </ansuer> the end.
A: nead to think step-by-stan Stap 1: Identify the position of the gray baok. the
gray
the
Step 2:
Identify the books the left of the gray book.
The black book is the left of the gray book.
Step 3: Identify the baoks the right of gray book. The brown book ia to the left of the green book. green
left Therafore, the books to the right of the gray book are the green book, the purpla book, the red book, and the blue book.
purple
Step 4: Identify the position of the blue book. The blue book is the secand
the right.
Step Determine which The blue book is the second from the right.
second
right.
<anever>D</ansven
seven objects arranged order The statements are logically consistent vithin each paragraph. In golf tournsment, there vere seven golfers: Joe, Kel, Nya, Eve, Rob Ada, and Dan, Rob finished first. Ere finished last. Dan finished abore Ada. Kel finished above Joe finished second-to-last. Nya finished third.
following paragraphe
Options:
(A) Joe finished tecond-to-last
(C) Mya finished second-to-las
(D) Eve finished
(E) Rab finished second -o-
(F) Ada rinished second-to-lass
second-to-last
Shox your final answer option bracketed betxeen <ansver> and </answer> at the end. step-br step.... **Step Identify finishing positions golfers mentioned in the paragraph. Rob finished first.
finished last.
Dan finished above Ada.
Mya.
Joe finished second-to-lare
Nya
**Stap 2:** Detorming the remaining finishing positions.
Since Rab rinished first Nya rinished third, the ramaining golfers finish positions 2, 4, 5, 6, and 7.
finished or 6.
position
position pasition or 3.
Mya,
**Stap 3:** Identify the golfer tho finished second to last.
finished second-to-len
(A).
(A) anewerr>
Listing 9: deduciiooedcctttneeeeeee obbjettt OPRO + Mutation. Test accuracy 62.5%
Q: The folloving paragraphs each describe set of three objacts arranged in fixed each paragraph. branch, there are three birds: blue jay, cardinal, and quail. The cardinal is to the right of the quail. The quail to the right
The statements logically consistent
blue jay.
Optione:
(A) The blue jay is the rightnost
(B) The cardinal
the rightnost
(C) The quail is the rightmost
<ansver> </ansver> at A: Lat's think step step. The quail is to the right of the blus jay The cardinal is to the right of the quail. 3o the order is: blue jay, quail, cardinal. <anever> (B) </ >
your final ansver option bracketed
52

---

folloving paragraphs describe order The statements are logically consistent vithin each paragraph. In golf sare golfera: Ana, Nel, and Ada above Ana. linished below Ana.
objecta arranged
tournament,
Optione:
(A) Ana finished first
(B) Mal finished first
(C) Ada finished
Show your fimal answer option bracksted betreen <ansver> and </answer> at the end.
A: Let's think step by step. Ada rinished above Ana.
Mel finished belov Ana.
So, the order is Ada, Ans, Mel.
<ansver </ansue>>
Q: The following paragraphe each describe set of three objecta arranged in fixed order The statemente logically consistent within each paragraph. Da branch there are three birds: crox, falcon, and a raven. The crow is to the left of the falcon
the right the
Options:
crow
rightnost
(B) The falcon ia the rightmost
The raven is the rightnost
Show your final
option bracketed between <ansver> and </answer> the end. A: Lat's think step by step. A: Let's think stop by step. the The falcon is to the right of the raven.
the order The falcon is the rightmost. <anever (B) s/aneve>>
crow, raven,
Listing 10: 84.5%
No I0 Random Search. Test accuracy
Lat think step step. Fizst, let's identify
genres given
options
one
common
genres.
If options fit thenes genres perfactly, let's choase the option that is most similar to the given novies in terms of its popularity and well-
Pinally,
anszer option betueen <anseer> </ansxer>.
Find novie sinilar to Schindler's List, Pulp Fiction, Braveheart, The Usual Suapecte:
Options:
(A) 12 Angry Men
Ho Better
(C) Windhunters
Rademption
Show final anaver option bracketed betzeen <anaver> and </ansuer> at the end.
critically acclained highly regarded in the film industry. They also cover various genres, including (Pulp Fiction), historical opic
anong given
they
historical (Schindler's List), Braveheart), and crine thriller (The Usual Suspacts).
Out of the options provided, the novie that best fits these criteria 18:
<an Shavehenk Redemption </aweer>
Find novie similar to American History X, Requiem for Drean, Raidera the Loat Ark The Sixth Sence:
Options:
Hise Meadovs
(B) Little Man
(D) You Can't Take It with You
Show your final ansver option bracketed between <anaver> and </ansver> at the end.
The conmon thene anong the given movies is that they are all critically acclained and films.
53

---

While the genres of the navies vary, they are all considered to be classic films that of
most similar to the given of its popularity and vell-knovnuess Goodfellas.
options provided, , the
<ansuer>(C)
Find novie similar to Batman, The Vaual Suapects, The Silence of the Lamba, Juraasic
Park:
Options:
(A) The End of the Tour
(B) Forrest Guap
(D) Caution
Shelter
Show your final
option bracketed between <answer> and </anewer> the end. that they all critically acclained and
The
anong the givan popular films fron the 1990s.
Out of the optiona, the novie that la noat similar to the given movies in terma of its popularity and well knownness is Forrest Gump.
(B) Forrest Cunae/an
Listing 11: nooienreoo PromptAgent + Mutation. Test accuracy = 95.0%
Solve the following nath problens following the stepa in the order of operationa:
reault 2. When multiplying dividing a positive and negative nunber, the result will be negative.
nultiplying dividing negative
positive.
3. When adding subtracting negative number, it is the same adding or subtracting ita pasitiva counterpart.
((6 5 -3) a (6 -7-77
-7
-2 -7))
Show your final ansver bracketed between <answer> and </anever>. -7))
((6
-3) (6
(6 0 -3) + (6 -7 -2 -2-- -7)
-7)
3 x (6
3 -7 -2 -7)
-2
3 (13 + 7)
(11 7)
<5never>5<</aee answer>
When multiplying dividing negative aumbere, the result will be poritive.
2. When multiplying or dividing a positive and a negative nunber, the result will negative.
3. When adding subtracting negative number, it is the same as adding or subtracting its positive counterpart.
((8 * 9
-1) (-9
-7 -7
8))
Show your final ensver bracketed betseen <answer> and </anever>.
((B + -1) (-9 -7 4 8))
1.
-
3. (-14) - 14 15
<ansvar (mnour>15</ansser
1. Whan multiplying or dividing tuo nagative numbers, the result zill be positive. multiplying dividing positive negative number, result will negative.
adding aubtracting negative number, Ite positive counterpert.
same as adding or aubtracting
((1 -5 3) (9
0))
Show your final
bracketed between <anaver> and </ansver>.
[(1
-5 3) (9 4 -))
+ (9*5
2. (9 -4
(54 4 0) 60
2.2- 50
Kanever >52</ anever>
54

---

Listing 12: wtw.iisttp.arithmeiiic..o ProTeGi + Mutation. Test accuracy 82.5%
Given set of instructions determine vhether folloving those instructions will take you exact cane spot you turns and the direction of steps. For example, if you take stepa forvard and then stepa backward, you will end up in the sane spot. Determine your anaver by eaying "yes" or "no".
fron. Keep in any movements. imcluding
you follow
instructione, you return starting point? steps. 10 steps. Take steps.
Options: Yea
Show your final (Yes No only) bracketed betueen cansver> and </ansver>. <ansver Mo< <nsuer>>
If follow these instructione, return the starting point? Alvays face forward. Take steps forvard. Take steps backvard. Take steps backvard. Take steps forvard.
Options:
No
Shox your final anaver or only) bracketed batueen <ansver> and </anaver>. <anaver Yea /eaaannerr> anaver>
you these
you return starting point? steps.
around. Take B ateps. Take 2 steps.
Optiens: Yes
Show your final
(Yas only) bracketed betuean <ansver> and </ansver>.
<ansver No //// anssueerr>> ansver>
Listing 13: navigate: ProTeGi + Random Search. Test accuracy = 68.5%
Q: have yam, stalk of celery, a potato, a cabbage, two cauliflovers onion, broccoli, carret,
lettuce
garlic. many vegetables
hava?
Show your final ansver bracketed betzeen <anawer> </ansver>,
A: Lat's break down the problen into amaller stepa and solve each one.
problem into snaller steps and solve each A: Let break
1. Ideatify the vegetables:
Calery
Potato
Cabbage
Cauliflover (2)
Lettuce
Carrot
Garlic
2. Couat the number of vegetables:
Calery:
Potato:
Cabbage:
Cauliflover:
Onion:
Lettuce:
Broccoli:
Carrot: 1
Garlie:
3. Add up the counts:
Therefore, ansver <ansvor>11</snooor>
0: have cauliflover, a cabbage, a garlic, a lettuce head, an onien, head of broccoli, potato, Hov many vegetablee
55

---

(B) Bernard Show your final bracketed between <answer> and </ansver?. A: Lat's break doun the problem into smaller steps and solve each one. A: Lat break problen into steps and solve
Identify vagetables:
Cauliflover
Cabbage
Garlic
Lattuce head
Onion
Broccoli
Potato
Carrot
Count the number vegetables: -Theee are 8 items in the 1ist.
Therafore, the anaver ia <anavar> Saaaarr84//navee.
accordion, clarinet, have?
druns. How many musical
your
bracketed
</anever>,
A: Let break doun the problem into amaller and solve each one. A: Let 2 break down the problen into smaller steps and solve each one.
1. Ve have one accordion.
clarinet. 3. Ve have three drums.
Adding these get:
1
Therefore,
<ensver>5</anaver: musical
Listing 14: object_counting: OPRO + Random Search. Test accuracy 99.0%
D: Here is a table where the first line is a header and each subsequent line is a penguin Bernard, , 80, Vincent, 9, 60, 11 Guea, 8, 70, 15 For example: the age of Louis is 7, the weight of Guen is Which
age, beight (cn), veight (kg)
7,
kg. height
heaviest penguin?
Options:
Louis
(C) Vincent
(E) Janes
<ansver> </ansver> end. A: Lat's proceed in sequential manner. The heaviest penguin is Gxen who veighs 15 kg.
your ansver option bracketed
The second heaviest penguin must either Louis, Bernard, or Vincent, as they all less than Guen.
Louis weighe kg, Bernard veighs kg, and veighs kg. Therefore, Bernard the second heaviest ponguin.
<ansver> <B)</ane
subsequent nane age, height (cm), veight (kg) Louis, 7, 60, 11 Bernard, 5, 80, 13 Vincent.
peaguin
9, 60, Guen, 8, 70, For example: age is 7, weight Gxen
15 the height of Bernard is 80 Ve then delete the penguin named Bernard from the table.
Bov many penguins Options:
than 8 years old?
(B) 2
(D) 4
(E) 5
option bracksted betweon and </ansuer> the ond. A: Lat's procaed in a seguential manner. The question asks the nunber of panguins that years old.
Show your final
The table has peaguina: Louis, Bernard, Vincent, and Gven.
Bernard is deleted from the table, so ve have penguins left.
56

---

Louis is 7 years old, Vincent is years old, and Gwan is years old.
Therefore, there is only 1 penguin that is more than 8 years old.
<answer> >() </a<))>>)a)/aaeeeaaa/ </aneve>>
penguin name, age, height (cm), weight (kg) Louis, 7, 50, 11 Bernard, 5, 80, 13 Vincent, 9, 60, Guen, 8, TO, For exanple: the age of Louis ia T, seight Guen is
aubaequent
15 the height of Bernard 80 Bov many peaguine lees than years old
Options:
(A)
(C) 3
(D) 4
(E)
Show your final ansver option bracksted between <ansver> and </ansver> at the end. sequential Bernard is 5 years old.
A: Lat's procaed
Louis 7 years old.
years
Guen is 8 years old.
There penguins less than years old.
<ansver (B) </ansur>>
Listing 15: Lttpuuiis_ii____ta___ APE + Random Search. Test accuracy 89.7%
Q: On the table, see three black streas balls, keychaing, one black dog leash, ane orange keychain, three orange cat toys, one black xeychain, and purple
streas ball, two purple
ball. table, how many orange itena remain on it?
the toys
Options:
(A) zero
(C) two
(D) three
(F) five
(H) seven
sight
(J) nine
(X) ten
(8) txelve
thirteen
(0) fourteen
(P) fifteen
(Q) sizteen
Show your final ansver option bracketed between <ansver> and </ansver> at the end. approach.
approach.
black stress balls, orange stress ball, too purple keychains, black dog leash, orange keychain, three orange cat toys, one black keychain, and one purple stress ball.
If ve romove all the cat toys from the table, ve are left with:
Three black stress balls
orange
Two purple keychains One black dog lessh
One orange keychain
One black keychain ball
purple
Counting orange itens, have:
One orange strees ball
One keychain
So. orange remaining
<ansver>CS/maane
57

---

Q: On the floor, you see a bunch of things arranged row: turquoise paparclip, an orange bracelet, green keychain, silver pen. the of the thing directly to the left of the silver thing?
Options:
(A) red
(B) orange
(C) yellov
(D) green
blue
(F) brown
nagenta
(8) fucheis
(1) mauve
(J) teal
(X) turquoise
burguady
(N) silver
geld
(0) black
(P) grey...
(q) purple
(R) piak
</anever> end. A: Lat's take nethodical approach. A: Let take methodical approach. The question
your final enswer option bracketed
sake for the color the thing directly to the left the silver thing. The given
items
turquoise paporclip
orange bracelat green keychain pen
Reading from left to right, the silver pen is the last item. The iten directly to its left the keychain. Therefore, the color of the thing directly to the left of the silver thing is green.
Q: On the floor, there is green pencil, yallov mug, fuchsia booklet, and pen Is the booklat fuchsia?
Options: (A) yes
Show your final ansver option brackated batzeen <anaver> and </ansver> at the end.
A: Let take a nethodical approach. <answor>A</ansee
Listing 16: biaaooinn_bbouttccloooeeoboeepe
APE + Diversity, Test accuracy = 80.5%
Q: Which of the folloving is humoraus edit of this artist or novie name: 'spice girla
Options:
(A) spihe girle
(B) spic girls
spice tgirls
(D) spite girls
<ansear> </ansver> end. A: Determine which anaver choice represents the most plausible hunorous edit. The
your
ansver option
humorous edit that stands out 1s:
(Dn spite <inserr This edit plays on the idea of spite, which is feeling of resentment or ill will, original name "Spice Girle."
adding
D: Which of the folloving is a humorous edit this artist or novie name: 'the men who goats??
Options:
nen stare
(B) the nen tho stare goats
nen who stare goats
(D) the who stare at goate
Shov your final ansver option bracketed between cansver> </ansver> at the end.
Determine which
choice rapresents the plausible humorous edit. The humoroua adit that stands out is:
nen stare
This edit introduces conical twist by replacing "goata" with "boats," creating an smusing nental image intently staring at boste instead of goats. The other
58

---

options either types significantly alter the original in humorous way.
</ansve>>
Q: Which of the folloving bumorous edit this artist novie name: 'gone with the sind'?
Options:
(4) goune with the vind
gone with the wand
(C) gone with the wing
(D) gone witth the vind
option brackated batween <ansvar> and </answer> the end. A: Datermine which anaver choice representa the nost plausible humorous adit. <ansver> vandera
Show your final
gone
Listing 17: ruin_namest OPRO + Mutation. Test accuracy 91.0%
Q: The folloving
will translations fron German to English contain a particular error That following types: Naned Entities: entity (nanes, placee, locations, etc.) is changed to different entity. Fumerical Values: Humerical changed. Adjactives: The madifiers and adjectives pertaining to noun are changed. Negation or Antonyms: Introduce or remove a negation or change comparatives to their antonyns pertaining the above classes introduced in the translations. Dropped Content: A significant clause in the translation is renoved. Please identify
values (ordinals cardinals), dates, and/or
or
Facta: Trivial factual
errer. Source: Boyd Kevin
amerikanischer Politiker.
Translation: Boyd an
politician.
The translation containa error pertaining
Options:
Modifiers Adjectives
(B) Fumerical Values
Begation Antenyns
(D) Raned Entities
Drapped
(F) Facts
Show your finel ansver option bracketed between <ansver> and </anever> at the end. think step step. The in the translation Rutherford" is reduced to just "Boyd". be:
"Boyd Kevin
correct translation Therefore, the error partains to (D) Named Entities.
Rutherford
politician.
The final anever is: <an </aneer>
German English contain particular error. locations, etc.) changed
folloving translations errer will be one of the folloring types: Named Entities: An entity (nanes, places,
different eatity. Fumerical Values: Numerical cardinsle), dates, and/er units Adjectives: The modifiers and adjectives pertaining to noun are changed. Negation Antonyne: Introduce remove negation change comparatives
values (ordinels
changed. Nodifiers
their antonyne
Facts: Trivial factual errors not pertaining to the above classes are introduced translations. Dropped Content: significant clause
translation renoved. Please identify that error Source: Dunau, laut Urtstafel Dunau Lathwehron und auf Landkerten auch als Lathvehren -Dunuu bezeichuet, 1st eine Ortelage des Region Hannover Niedersachsen. Translation: Dunau, according to the local plaque Dunau Lathoehrnn, is locality the Lathwehren
Stadtteils Lathrehren der Stadt Seelze
city Seelze The trantlation contains an error portaining to Options:
region Lover Saxony.
(A) Modifiers Adjectives
(B) Nunerical Values
(C) Begation Antonyns
(D) Baned Entities
Dropped
(F) Facts
Shox your final anaver option bracketed betzeen <anaver> </ansuer> at the end.
A: Let's think step step. The original Gernan sentence nentions "laut Ortatafel Dunau- Lathwehren und suf Landkarten auch als Lathvehren Dunau bezeichnet". This part is here
translated "according the local plaque Dunau- Lathvehron", The that the translation does not nention "auf Landkarten auch als Lathvahran Dunau as
bezeichnet", shich neans The correct translation should be:
maps".
Dunau, according to the local plaque Donau Lathreeebvenzen and also referred to as Lathwehren- Dunau пaрe, is locality of the district of Lathvehren in the of Seelze in the Hanover region Lover Saxony.
59

---

Therefore, the pertains to (E) Dropped Content.
Final
==
Q:
The following trenslatiens from German to English contain a particular error. That will be of the folloving types: Named Entities: entity (nanes, places, locationa, etc.) is changed to a different entity. Kumerical Valuas: Nunerical values (ordinals
cardinals), dates, and/or unita Adjectives: The madifiers and adjectives pertaining to noan are changed. Negation their antonyns not pertaining to the above classes introduced in the translations. Dropped Content: A significant clause in the translation is renoved. Please identify that
changed. Modifiers
Antonyma: Introduce or remove negation change comparativea
Facts: Trivisl factual
Source: Karte mit allen Koordinaten: 0SM
NikiNap
Translation: Map
coordinates:
WikiNap
The translation containa an error pertaining
Options:
(A) Medifiers
Adjectives
(B) Numerical Values
(C) Negation Antonyna
(D) Baned Entities
Dropped
(F) Facta
Show your final ensver option bracketed between <ansver> and </ansver> at the end. is *Karte mit allen Koordinaten OSM WikiMap". The English translation is "Map with no coordinates: 0SX VikiMap that word "allen" (all)
Let 's think step step. The original Gernan
"
in the translation 88 "no". This is an error of Negation or Antonyns.
translated
Listing 18: saaienttrraslatioooerooooooooo wrwpelaattiineerr_eeeeeeo 62.5%
APE Random Search. Test accuracy
Which
Options:
vorking (B) You still achieve the American Drean of working Valnart
You
Show your final ansver option bracketed between <ansver> and </ansver> at the end.
A: Va should forzard stages. <answer> </ansver>
statement sarcastic?
Options:
(A) Anyone can auccessful without having inherited all their money first without having inherited all their money Show your final ansver option bracketed betzeen <anaver> and </ansuer> at the end. stages.
(B) uabody
should nove
Q: Which stateneat is sarcastic?
Options:
(A) We should start a petition to ban breathing
should start petition
vaping
Show your final ansver option bracketed betxeen <ansver> and </ansver> at the end. move forsard stages. <ansvor>(a<<aaaveer
Listing snarks: APE + Random Search, Test accuracy 87.4%
To address this question, ue xill divide it into smaller staps and use the provided knovledge about aports.
Provided knovledge:
Dejounte Kurray plays the NBA.
Goal tanding is baskethall rule. Kroos FA Cup is a football compatition.
professional football player.
Travis Konecay is hockey player.
Harper ie baseball player.
Tuukka Rask is a hockey player.
Giorgio Chiellini is football player. Maradona is a nove in football. move football.
Poverplay ia tarn in hockey.
Haudball is an illegal move football, where player touches the ball with their hand during natch.
60

---

step three Studs up is a tern in football.
basketball.
Blocking
penalty in football.
Initially,
for sentence plauatble the context sports. It is logical and consistent with the facta about sports when sentence plausible.
Now, se will lonk into the given four examples. For example 1, "Travis Konecny did maradona
infer that maradona is nove in football, not hockey. Similarly, for example 2, "Bryce Barper kit the back shoulder fade," se can also infer that this sentence is not plausible because back shoulder fade is Tuukka Rack killed the poverplay, ," we can conclude that this senteace is plausible because poverplay
defender,
sentence is plausible
football, not baseball. For exauple 3,
hackey. Likavise, connitted a handball in the FA Cup, de can also conclude that this sentence ia plausible because compatition.
example 'Giorgio Chiellini
penslty football
cup football
Based the above analysie, the Example 1: na
Example
Example 3: yea
Example yes
Now, we will examine the tvo new examples. For example 1, "Dejounte Eurray took side- three,
conclude that this sentence is plausible since side-step three is a nove in baskethall, Similarly, for example 2, "Luka Nodric zent in studs up in plausible because studa up
the FA Cup," is a tern in football and the FA Cup is a football competition.
conclude
this
Therefore, the correct ansuers for the nev examples are: Example 1: yes
Example 2: yes
Is the folloving sentence plausibla? "Neymar did naradona an the defender in the Champions Leage Senifinal."
Show your final answer (yes or no only) bracketed betveen <anever> and </ansver>. cansver>yes<// Syes ansver>
Provided knowledge:
Dejounte Nurray plays Goal tending is basketball rule.
Kroos
professional football player.
FA Cup is football compatition.
Travis Konecny is hockey player.
Bryce Harper is baseball player.
Tuukka Rask is a hockey player.
Giorgio Chiellini is football player. Maradona is move in football. in football.
Poverplay is tern in bockey.
Handball is an illegal move in football, shere a player touches the ball with their hand during natch.
Side-step three is a move in backetball. up Blocking foul is penalty in football.
Initielly, will define what it means for sentence to be plausible in the context or sports. It is logical and consistent with the facts about sports when a sentence is plausible.
examples. example 1, naradona on the defander, se can infer that this sentence is not plausible since naradona move football, not hockey. Similarly, for example 2, "Bryce Harper hit the back shoulder fade," because back shoulder fade is nove in football, not baseball. For axample 3, plausible
se
into given
Konecay
also infer that this sentence is not plausible
conclude that because poverplay is term in hockey. Likevise, for example 4, "Giorgio Chiellini conmitted
poverplay, ,
conclude that plausible because handball is panalty in football and the FA Cup is football conpetition.
Cup,
Based on the above snalysis, the correct ansxors are: Example
Example 2:
Example yes Exanple 4:
61

---

pass." Nox, will examine the examples. Por example "Dejouate Kurray took side- step three, ve can conclude that this sentance is plausible since side-atap three example 2, "Luka Nodric the FA Dup ze can also conclude that this sentence is plausible because studs cup football conpatition.
basketball. Similarly,
atuds up
tern football
Therefore, the correct ansvere for the nev examples are: Example Example 2: yea
the following sentence plausible? "Collin Show your fimal answer (yes or no only) bracketed betueen <ansver> and </ansver>. <anaver syea anaver>>
the beater."
Provided knowledge:
Dejounte Nurray playe
ТB4.
Goal tending is haskethall rule.
Toni Kroes is professionsl football player.
P& Cup
football competition.
Travis Konecny is hockey player.
Bryce Harper is baseball player.
Tuukka Rask is a hockey player.
Giorgio Chiellini is football player. Maradona is nove in football.
Back shoulder fade is a move in football.
Poverplay
in bockey.
Handball is an illegal move in football, where a player touches the ball with their hand during natch.
Side-step three is a move in backetball. up Blocking foul is penalty in football.
Initially, will define what sports. It is logical and consiatent with the facts about sports when a sentence is plausible.
for
be plausible in the of
into given maradona on the defander, se can infer that this sentence is not plausible since maradona is a nove in football, not hockey. Similarly, for example 2, "Bryce Harper also infer that this sentence is plausible
so
examples. example 1,
Konecny
hit the back shoulder fade," because back shoulder fade is nove in football, not basaball. For example 3, plausible
poverplay, , because poverplay is term in hockey. Likevise, for example 4, "Giorgio Chiellini conmitted plausible because handball is panalty in football and the FA Cup is football competition.
conclude
Cup
conclude
is
Based on the above analysis, the correct anszers are:: Example
Example 2:
Example yes Exanple 4:
examples. exanple "Dejounte Murray took side- step three, " we can conclude that this sentence is plausible since side-step three move backetball. Similarly, example 2, "Luka Nodric the FA Cup " ze can also conclude that this sentence is plausible because studs Cup football conpatition.
examine
up
tern football
Therefore, the correct ansvers for the nav exanples are: Exanple yea
Example 2: yes following sentence plausible?
Diggs
Show your final ansver (yas or na only) bracketed betueen <ansver> and </ansver>. canaver vyea anaver>
Listing sports_unutt undeestandiig.. ProTeGi + Mutation. Test accuracy 90.0%
To solve this problem, ve can break it doun into snaller steps The first is find time when the person voze up. Dace have that information, sa can proceed to the next step, which is identify the earliest time slot that has not been accounted for.
Today, Janes want to the avimming pool. Betxaen what times could they have gana?
Janea zoke at fan.
Ashley sav James sitting on rooftop from 7an 9an Hannah Jamee taking photos near the Biffel Tover from 9an to 7pu. Villian sav Janes buying cookies at a bakery from 7pm to 8pn.
62

---

Linda James vaiting the train station from 10pm.
The svinming pool was clased after 10pm.
Batveen what times
gone swimming pool?
Options:
7pm 8pm
(B) 8pn 10pm
(C) San to 7am
(D) 7am 9an
Show your final anaver option brackated batween <anaver> and </ansuer> at the end. sitting on reoftop
The person
This neans that Janes could not have gone to the svimming pool during this time. Hannah sax Janes taking photoa near the Eiffel Toser from 9am to 7pn. Thia neans that James could not have buying cookies at a bakery from 7pm to 8pn. This means that James could not have gane
up
Asbley
the svimning pool during this tine either. Nillian say James
szimming pool train atation fron 8pn to 10pn. This neans that James could not have gone to the swimning pool during this time fran 5am to 7am. <anever>(() ansver>
this tine either. Linda
Janes waiting
the
Therefore, only
left is
Today, James went to the beach. Between what times could they have gone?
Janes woke up at 5an.
Sean ва* Janes valking tovards the Statue of Liberty from San to 6an. Nichael Janes driving the Anthony sav Janes reading at the library fron 7an to 3pm. Millian
park from 6an to 7am.
getting coffee at The beach наа closed after 9pn.
4pm 9pn.
gone beach?
Options:
(d) 7am to 3pm
(B) San 6an
(C) 4pm to 9pm
(D) 3pm 4pm
Show your final answer option bracketed betxeen <anaver> and </answer> at the end. voke up
Sean sax Janes valking tovards the Statue of Liberty fron San to 6an. Michael sas Janes driving to the xater park from 6am to 7am.
Anthany Janes reading the library fron 7an 3pm. Villian sas Janea gatting coffee at the cafe from 4pm to 9pn. 9pm.
From the information given, we know that James was at the Statue of Liberty from San to 3pm.
am, at vater park means that ha could not have the cafe from 4pm to 9pa, so he could not have gone to the beach then either. 4pm.
7am,
at to the beach during those times. He was also at
library
Therefore, only tine slot that <ansver > </ansue>>
accounted for from
Today, Elizabeth vent to the park. Batueen shat tines could they have gone?
Ve know that:
Elizabeth
Emily ааж Elizabeth buying a bike at the bike shop from 10am to 11am. 2pm.
Leslie
buying lunch Sarah sax Elizabeth playing tennis at the tennis court from 4pm to 8pn. 8pn. Betveen vhat tines could Elizabeth have the park?
park
Options:
(B) 11am to 2pm
4pm 8pm
(D) 2pm to 4pm
Shox your final anaver option bracketed betzeen <anavar> </ansuer> at the end- the bike shop from 10am 11an, the deli from 11am to 2pm, and at the tennis court from 4pm to 8pn. The park van clased after accounted
Elizabeth vože up at 10an. She
8pm. Therefore, to 4pn.
that
2pm
cansuer>(D) </ </anaver>
Listing 21: temporal seapporaasseuunceess ProTeGi Mutation. Test accuracy = 100.0%
task systematic manner. objects that are being axapped in the context. Then, lat's trace the svaps and keap track of the objecta they change hands. Finally, let'a use our underatanding of the evaps and the initial positions of the objecte need to make sure that all the information necessary to ansver the question is
approach
identify
different
the question. Ve slso
63

---

contained the context and check that the number players is the the number of abjects.
Alice, Bab, Claire, Dave, and Eve are holding a white elephant gift exchange. At the holding a present of different color:
start the avant, they are purple present, Bob bas blue present, Claire has brown present, Dave has yellow present, and Exe has a black ball.
has
As the progresses, pairs of people seap gifts. First, Alice and Bob sxap their gifts. Than, Bob and Dave avap their gifts. Then, claire and Bab avap their gitts. avap their gifts. Finally, the end of the event, Dave has the
Then,
and awap their gifts.
Options:
(A) purple present
(B) blue present
(C) brown present
(D) yellau present
black
Show your fimal ansver option brackated betzeen <anaver> and </ansver> at the end.
Let's trace the suaps one by one:
1. Alice and Bob roap their gifts: Alice giver her purple present to Bob, and Bob gives his blue present Alice. So Alice the blue present, and Bab present.
purple
2. Bob and Dave sxap their gifta: Bob gives his purple his yellow present Bob. So now Bob has the yellow present, and Dave has the purple present.
Dave, and Dave gives
Claire and Bob avap their gifta: Claire gives his yellow present to Claire. So now Bob has the broen present, and Claire has the yellow preseat.
present
Bab gives
4. Dave and Alice svap their gifts: Dave gives his purple present to Alice, and Alice gives her blue present Dave. So Alice has the purple present again, and Dave has the blue preaent again.
5. Dave and Eve внap their gifts: Dave gives his blue present to Eve, and Eve gives her and
ball to
So now
present.
At the end of the swaps, Dave has the black ball.
Therefore, the anaver is: <anazer (E) c/ansver>
Alice, Bab, Claire, Dave, and Eve are dancers at square dance. At the start of song, they each have a partner: Alice is dancing with Rodrigo, Bob is dancing with Sam, Claire dancing with Izzi, Dave dancing with Jamie, and Eve dancing with Ophelia.
Throughout song,
switch partnere Then, Alice and Bob svitch partners. Then, Eve and Bob sxitch partners. Then, Dave partners.
partners.
partners. Finally, and the dance, Alice is dancing with
Options:
(A) Rodrigo
(B) San
(D) Jamic
Ophelia
Show your final answer option bracketed between <anaver> and </ansuer> at the end.
Let 's identify the dancers and their initial partners: Redrigo.
Alice dancing
Bob is dancing with San. dancing Dave is dancing with Janie.
dancing with Ophelia.
Now, let'a trace the svaps: evitch partners:
dancing with Ophelia, dancing now dancing Sam,
San.
Alice
dancing Eve and Bob svitch partners: Eve ia now dancing with Rodrigo, Bob dancing with
partners:
Radrigo.
San again.
Dave and Alice svitch partnors: Dave is now dancing with Alice, and Alice is dancing
Eve and Claire evitch partners: Eve is nox dancing with Izzi, and Claire is dancing
Rodrigo.
At the end the dance, Alice is dancing with Jauie.
64

---

The final is: <ansver > (D) ansver>
Alice. Bob, Cleire, Dave, and Eve dancers at
daace. At the start of song, they each have a partner: Alice is dancing with Lola, Bob is dancing with Melisss, Claire dancing with Janie, Dave dancing with Rodrigo, and Eve is dancing with Balga.
Throughout song,
switch partners Then, Alice and Dave switch partners. Then, Rve and Bab switch partners. Then, Eve claire suitch partnera. Finally, Dave and suitch partnera. At the and of the dance, Eve is dancing with
trade partners.
Options:
(4) Lola
(B) Heliasa
(D) Redrigo
Belga
Show your final
option bracketed between <answer> and </snewer> the end. Let 's identify the dancers and their initial partners:
Alice: Lola
Bob: Nelissa
Dave: Rodrigo
Eve Helga
Now. let'z trace the svaps:
Bob avitch partnera:
dancing Helga, and Eve dancing with dancing Rodrigo,
Melissa.
partners:
dancing
with Lola.
Eve and Beb svitch partners: Eve is nov dancing with Bob, and Bob is dancing with
Melissa again.
4. Eve and Claire avitch partnera: Eve is nog dancing with Claire, and Claire in dancing
5. Dave and Bob sxitch partnere: Dave is now dancing with Helisea, and Bob is dsacing Rodrigo.
At the end of the dance, Eve is dancing with Claire.
<ansver> (C) </anaver>
Listing 22:
ProTeGi + Random Search. Test
accuracy 70.0%
Alice,
square the start of song, they each have partner: Alice is dancing with Lola, Bob is dancing Ophelia,
Dave,
is dancing with San, dancing with Patrick, Fred is dancing with Janie, and Gertrude is dancing with Rodrigo.
dancing
Helga,
Throughout song,
trade partners. First, Fred partners. Then, Eve and Alice switch partners. Then, Fred and Eve switch partnere. and svitch partners. Then, Gertrude Then, Alice and Bob svitch partners. Finally, Dave and Fred suitch partners. At the dance,
switch
Then,
svitch partners.
dancing
Options:
(A) Lels
Uphelia
(C) San
Belga
(E) Patrick
(G) Rodrigo
Show your finel ansver option bracketed between <ansver> and </anewer> at the end. step-by-step process. follow step-by-step process.
start dance, dancers paired folloxs:
Alice Lola
Bob Ophelia
Claire San
Helga
Eve Patrick
Jamic
Certrude Rodrigo
Step 1: Pred and Bob switch partners.
Alice Lols
65

---

Bob Jamie
claire San
Helga
Eve Patrick
Ophalia
Gertrude Rodrigo
Step 2: Eve and Alice switch partners.
Patrick
Bab Jamie
Claire
Dave Helga
Eve lola
Fred Opholia
Gertrude Redrigo
Step 3: Fred and Eee szitch partners.
Alice Patrick
Bab Jamie
Claire
Dave Helga
Ophelia
Fred Lola
Gertrude Bodrigo
Step 4: Claire and Frod sxitch partners.
Alice Patrick
Claire Lola
Dave Helga
Eve Optelia
Fred San
Rodrigo
Step Gertrude and Alice partnere.
Alice Rodrigo
Bab Jamie
claire Lola
Helga
Eve - Ophelis
Gertrude Patrick
Step 6: Alice and Bob svitch partners.
Alice
Bob Bodrigo
Dave Helga
Eve Ophelia
Fred
Gertrude Patrick
Step 7: Dave and Fred suitch partners.
Alice Janie
Bab Rodrigo
Claire Lola
Dave San
Ophelia
Fred Helga
Gartrude Patrick
At the and of the dance, Dave is dancing with San.
The final anever is <ansver> </a0)///// ansver)>
Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude friends and avid readers who occasionally trade books. the start of the senester, they each buy one new book: gete Noby
of Baekervilles, gets Catch-22, Eve gets The Fellouship of the Ring, Fred gets The Pearl, and Gertrude gets Frankenstein.
gets
Claire gete
Aa the semester proceads, they start trading around the nex baoks. First, Dave and Bob svap books. Then, Alice and Eve avap booke. Then, Bob and Fred svap books. Then, Gertrude and Claire svap booke. Then, Fred and Dave avap books. Then, Alice and Fred
66

---

svap books. Finally, Dave and Bob evap books. At the end the Claire
has
Optiona:
(4) Lolita
Maby
(C) Bound of the Baskervilles
(D) Catch -22
(E) The Pellouship of the
(F) The Pearl
Frankenstein
Show your fimal answer option bracksted betreen <ansver> and </ansver> at the end. Let's follow atep-by-atep proceas. Let follow atep-by-step procesa.
At the start the senester, Alice has Lolita.
After the first suap, Dave has Lolita and Bob has Catch-22.
Alice has The Fallouship of the Ring and Ere has Lolita. After the sacand
After the third evap, Bob has Lolita and Fred has Catch-22.
After the fourth awap, Gertrude Lolita and Claire has Frankenstein.
After fifth evap, Fred Lolita
C2tch-22.
After the sixth evap, Alice has Lolite and Fred has The Fellovehip of the Ring. After the seventh awap, Dave has Lolita and Bob has The Pallauship of the Ring. At the end of the semester, Claire has Frankenstein. <ansuer> (G) </ansve>>
Alice,
Dave, Fred, and the start of song, they each have partner: Alice is dancing with Ophelia, Bob is Janie, Dave dancing
aquare
dancing Nelissa, Claire dancing is dancing with Patrick, Fred is dancing with Redrigo, and Cartrude is dancing with Karl.
San,
Throughout the song, the dancers oftan trade partners. Pirst, Dave and Claire switch partnera. Then, Alice and Eve avitch partners. Than, Eve and Bab switch partners. partners. Then, Fred Gertrude and Dave switch partners. Finally, Dave and Alice evitch partners. At the dancing
Then, Claire
switch partners. Then,
dance,
Optians:
(A) Ophelia
Helissa
(C) Janis
(E) Patrick
Redrigo
(G) Karl
Show your final ansver option bracksted between <ansver> and </ansver> at the end.
Lat follow atep-by-atep proceas. Let follou the ataps by
switch partners.
Dave was dancing with San, and Claire gas dancing with Jamie. So nov, Dave is dancing Janie,
dancing
2. Alice and Ere suitch partners. Alice dancing
Ophelia, and dancing with Patrick, and Eve is dancing with Ophelia.
dancing Patrick. now, Alice
3. Eve and Bob switch partners. was dancing with Patrick, with Nelissa, and Bob is dancing with Patrick.
was dancing with Heliasa. now, Eve is dancing
Claire and Claire vas dancing with San, and Bob was dancing with Patrick. So now, Claire is dancing dancing
partners.
Patrick,
San.
5. Fred and Eve switch partners. Fred dancing with Bodrigo, and Eve dancing with Melisea. So Fred is dancing with Nolissa, and Eve is dancing with Rodrigo.
6. Certrude and Dave svitch partners. dancing
Karl, Dave dancing Janie, mos, Gertrude dancing with Janie, and Dave is dancing with karl.
7. Dave and Alice svitch partners.
67

---

Dave dancing with Barl, and Alice dancing with Patrick. So Dave dancing with Patrick, and Alice is dancing with Karl.
At the end of the dance, Fred is dancing with Nolissa.
(anewer>(B) </aavve>>
Listing 23: traakknng_hhfflleddoooeeeess.se 86.0%
APE + Mutation, Test accuracy
No will nethodically address this by breaking it down into nanageable portions. We will monitor the final status of every entity (e.g., Alice, Bob, and Claire) after the transactions moted in the problen. will also monitor the flou of the transactions
Alice, Bob, and Claire are dancers at square dance. At the atart of a aong, they each Helge,
partner:
dancing
dancing with Patrick, Claire
is dancing with San.
Throughout the song, the dancers often trade partners. First, Claire and Alice ewitch partners. Then, Claire and Bob switch partners. Finally, Alice and Bob suitch partnors. the and the dance, Bab is dancing with Options:
(A) Belga
Patrick
(C) San
Show your final ansver option bracketed between <ansver> and </anewer> at the end.
Let's track the partners of Alice, Bob, and Claire throughout the dance:
Initially,
dancing with Helga, dancing with Sam.
dancing Patrick, Claire
After the first avitch, Claire and Alice switch partners, so Alice 1s now dancing with San, and Claire is dancing with Helga.
After the sacond switch, Claire and Bob auitch partnera, so Glaire ia now dancing with Bob is dancing
Helga.
switch, and Bob is dancing with San.
partners,
dancing
Therefore, the and of dance, Bob dancing with (C) <neevr>>
Alice, Bab, and Claire are dancers at a square dance. At the start of a song, they each dancing Patrick, and Claire
have partner: Àlice dancing with Lola, is dancing with Melissa.
Throughout song,
Claire ewitch partners. Then, Bob and Claire switch partners. Finally, Claire and Alice switch dancing
trade partnere. First,
partners.
dance,
Options:
(A) Lala
(B) Patrick
(C) Meliess
your final ansver option bracketed <ansver> </anever> end. Let's track the partners of Alice, Bob, and Claire throughout the dance:
Initially, Alice is dancing with Lola, Bob dancing with Patrick, and Claire is dancing with Nelissa.
After the firet switch, Alice and Claire sxitch partners, so Alice is now dancing with dancing
Meliesa, and
Lola.
After the second switch, and Claire avitch partners, Bob la now dancing with Meliess, and Claire is dancing with Patrick.
third svitch, Claire and Alice dancing with Nelissa, and Alice is dancing with Patrick.
partnere again, Claire is now
At the end of the dance, Alice ia dancing with Patrick.
Therefore, the ansver ie: <ansver (B) anaaver>>
68

---

Alice, Bab, and Claire friends and avid readers who occasionally trade books. the start of the semester, they each buy one nav book: Alice gets Catch -2,, Boh get= Claire gets Fellouship Às the semester proceeds, they start trading around the new books. First, Claire and seap booka. Then, At the end of the semester, Claire has
Moby
Ring.
sxap books. Finally, Claire
suap books
Optione:
(A) C2tch-22
(B) Moby Dick
(C) The Fellouship of Ring
Show your fimal ansver option brackated betzeen <anaver> and </answer> at the end. **Initial Statua:
Alice: Catch -2
Bob: Moby Dick
claire: The Fellosahip of the King
*4Firt Trade (Claire and Alice avap books):
Alice: The Fellovehip of the Ring
Bob: Noby Dick
Claira: Catch-24
**Scoond Trade (Bob
awap :.oke):..
Alice: Moby Dick
Bob: The Felloxship of the Bing
claire: catch-22
*Tbbird Trade (Claire and Bob avap books)
Alice: Moby Dick
Bob: Catch -22
Claire: The Fellouship of the Ring
*4Final
Moby
Bob: Catch-22
Claire: The Fellouship of the Ring
Therafore, at the and of the semeater, Claire has **The Fallavship of the Ringa*.
<anever (C) s/aevr
Listing 24: accuracy = 74.0%
tttpttt__ttreeoeeettteo ProTeGi Random Search. Test
says Jin lies. says Crista saya Vernell tells the truth. Millicent Bernita talla the truth. Does Millicent tell the truth? Show your final
Question:
truth.
(Yas only) bracketed betuean <ansuer> and </anaver>. A: Let follow a step-by-step approach. Let's follov step-by-step approach.
1. Jin lies. So, Jim is liar.
2. Crista Jin lies. If Crista is telling the truth, then Jin liar. But slready know that Jin is liar. So, Crista is telling the truth.
3. Vernell says Crista tells the truth. If Vernell is telling the truth, then Crista is telling truth. already talling the truth.
Crista telling truth. So, Vernell
Bernita says Vernell tells the truth. If Bernita telling the truth, then Vernell is telling the truth. Ne slready know that Vernell is telling the truth. So, Bernita telling
Millicent says
truth. Millicent telling truth, Bernita is telling the truth. Ve already knov that Bernita is telling the truth. So, Millicent is telling the truth.
Therefore, the anaver <ansver>Yes</nssee::
q: Queation: Concaption tells the truth. Elanor Conception tells the truth. Alexia saya Elanor tells the truth. Fidel saya Alexis tells the truth. Delbert says Fidel lies. Does Delbert tell the truth?
Show your final ansver (Yes or le only) bracketed betveen <anszer> and </ansver>. follov etep-by-etes approach.
step-by-step appreach.
69

---

1. Conception tells the truth.
(Hence, 3. Alezis says Blaner tells the truth. (Rence, Alexis alse tells the truth.) saya Alexia tells truth. (Hance, Fidel also Delbert says Fidel lies.
Elanor saya Conception
truth
truth.)
If Delbert tells the truth, then Fidel lies. But if Fidel lies, then Delbert must lying. Therefore, Delbert cannot be telling the truth.
cansuer Mosaasero///
Question: Helene tells the truth. Leds says Helene lies. Antuan says Leda tells the truth. Andree says Antvan lies. Shaunda says Andree lies. Does Shaunda tell the
Show your fimal anaver (Yas or No only) brackated batueen <anszer> and </anaver>> Let's follow step-by-step approach. Let's follow step-by- step approach.
1. Helene tells the truth.
Leda says Halane lies. (This contradicts
Lada lying.)
3.
Antvan says Leda tells the truth. (This contradicts statenent 2, ao Antwan 1a lying.) Andree saye
indeed lying Shaunda says Andree liss. (This contradicts statement ao Shaunda is lying.)
Therefore, the final
is canszer Tosser>>o</ame Mossaneer
Listing 25: web_of of lies: APE Random Search. Test accuracy 99.5%
folloving words alphahetically:
mataphoric hollandaise
supranational payche pedro gloss granola quarantine inauspicious penis baccarat arenaceous nackenzie blare bounan bracketed
Show your
<anaver> </ansver>.
A: Let think step by step. The words sorted alphabetically are: causver>arensceuus, baccarat, insuspicious, mackenzie, metaphoric, pedro, penis, payche, quarantine, roadster, supranational </ansuer>
glose, granola, bollandaise,
Q: Sort the folloving words alphabeticslly: List: almost delphic semitic unbeknovnst astute deadlock libertarisn dandelion affable svseney fortunate leverage horntail apache
execution sanction scathe
Show your final ansver bracketed betzeen <anawer> </ansuer>.
A: Let think step step. **Step 1:*4 List the words elphabetically.
affable
almost
apache
astute
dandelion
deadlock
delphic
execution
horntail
leverage
levitate
libertarian
scathe
atorehouse
aveeney
unbeknounst
StStep
cansuer>affable
alnost
antic
apache
astute
dandelion
deadlock
delphic
execution
fortunate
70

---

horutail
leverage
levitate
libertarian
sanction
scathe
semitic
storehouse
aveancy
mubeRnownat<</neee
burlay 0: Sort the folloving words alphabetically: List: burley bela arapaho bacteria bock Show your final answer brackated between <answer> and </ansver>.
A: Lat's think atep by atep. **Stap 1:** List the sorda in alphabetical order.
arapaho bacteria
bela
bock
..Step
</snewer>,
Ssnever>arapal bacteria bela bock ourleys/anever
Listing 26: sordinorting: No 10 Mutation, Test accuracy =1.5%
71


Reviews:
[{"review_id": "8Qrhb1Ndfh", "reviewer": "Reviewer_JLax", "summary": "This work studies automatic prompt optimization, seeking to compare and to connect the literatures on instruction optimization and on example selection. They find that existing approaches for using LMs to self-generate examples and selecting between them can outperform the best existing instruction optimizers, but that combining the two is effective. They use this to recommend larger emphasis on few-shot examples in future prompt optimization work.", "questions": "1. This is not a weakness of the present paper, but in the manner described here APE and OPRO are presented rather similarly in Sec 3.1. Overall, this raises the question of how the authors ensured their implementations are faithful to the original algorithms while being fair to all methods within a new test environment.\n\n2. I appreciate the note in lines 199-201 about self-generation. The way this is presented suggests the authors thing it's important, but it's rather buried now and hard to concretely understand without an example or further elaboration, especially because the tasks considered in this work are rather \"simple\", e.g. not clearly multi-step. In addition, the term Example Selection (ES), which is not particularly universal in the literature, is perhaps setting this up for confusion, since selection directly assumes the presence of a set from which to select.", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The work asks a timely question that is formulated well, i.e. about the \"relative importance and performance impact of ES and IO, both in isolation and when combined together\". This challenges overemphasis on instruction optimization (IO) in plenty of recent work.\n\n2. In the experimental evaluation, the authors consider a wide and representative set of four Instruction Optimization methods from APE to OPRO, and take substantial steps to equalize their budgets for comparison.\n\n3. The authors do an excellent job at communicating their findings from a large number of comparisons in the form of several concrete and overall well-argued/supported Insights.", "weaknesses": "1. The authors evaluate on BIG-Bench and MMLU tasks, which to my knowledge are extremely limited in scope. Most are narrow reasoning puzzles or manipulation tasks, e.g. multiple choice, and may not reflect the type or complexity of typical prompts people use in the increasingly elaborate open-ended LM systems out there. This has an effect on the extent of \"self-generated\" examples, which in this work are reduced to just chain of thought reasoning if I understand correctly, where all \"inputs\" and \"labels\" (in this case final answers) are known in advance, rather than trajectories of actual input/output labels for prompts per example. How would such open-endedness interact with the findings?\n\n2. The authors already find very compelling results with ES, but the scope of existing example-based optimizers in the cited work like DSPy seem wider in terms of the search/optimization method and in the generation approach. For the latter, the authors employ a large LM to optimize the instructions for smaller LMs, but related work permits similarly a large LM (or the same LM but in already example-optimized form) to generate the examples for small LMs. As a result of this asymmetry, the effect of the source of instruction or example optimization is not explored or isolated. This seems unlikely to affect the results direction, but it may affect their magnitude substantially, perhaps making IO+ES less attractive than just ES.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful and positive feedback! Please see below for our point-by-point response to the reviewer’s specific comments.\n\n> Limitations of BIG-Bench and MMLU and more open-ended/realistic tasks.\n\nWe thank the reviewer for their suggestions! The primary reason we picked BIG-Bench and MMLU is that, as mentioned on line 165, these benchmarks (or similar tasks) are still heavily featured in many, if not most, contemporary automatic prompt optimization (APO) papers. These datasets encompass a large number of different challenging tasks, including reasoning-heavy or information-seeking types, and performance improvement in them still seems to be highly correlated with human-in-the-loop arenas like LMSys. These tasks are also still used to demonstrate LLM’s general problem-solving caliber, and both are featured in Gemini and GPT technical reports. As our first objective, as mentioned on line 62, is to “*compare the performance gain by … IO and ES methods on the fairground*”, we felt that using these datasets, which are also the ones the original authors often rely on to show the effectiveness of their proposed methods, is fittest for this purpose. We also argue that our evaluation is already more comprehensive than most existing APO works, which often choose small subsets of these tasks that often differ from paper to paper, which further confounds fair comparison. Nevertheless, we agree with the reviewer that using more open-ended tasks with more complicated and realistic tasks is important both for our specific study and for the broader APO field, which, as mentioned earlier, still relies on the aforementioned tasks to a considerable extent. We will endeavor to expand our analysis on more open-ended and/or multi-hop problems and will add a comment on the reviewer’s suggestion in Conclusion.\n\n> Scope of example-based optimizers in existing works like DSPy\n\nWe thank the reviewer for pointing this out – existing works like DSPy indeed introduced exemplar optimization options that are more advanced than random search, such as TPE-based Bayesian optimization. We opted for the simpler methods in the present work for two reasons: 1) both our preliminary results and the results reported by DSPy authors in concurrent works like Table 2 of [1] show random search is still a remarkably strong baseline; in many cases, the more advanced optimizers perform similarly to it, and 2) while valuable, the comparison between different exemplar optimization methods is not a key focus of the present study – indeed, a key observation is that exemplar optimization as simple as random search can outperform and synergize with much more advanced instruction optimization methods, and using more advanced exemplar optimization could only strengthen rather than undermine this finding. We will further clarify the above discussions in the final draft of the paper.\n\n[1] Opsahl-Ong et al. (2024). Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs. \n> Asymmetry due to the use of large LMs to optimize instructions and the effect on the relative performance.\n\nWe agree that this is an important and plausible explanation for the stronger performance of methods like ProTeGi, which rely on reasoning-based self-critique, an ability that can be strongly dependent on the general caliber of the optimizer model. Nevertheless, the reported synergy should not entirely come from this asymmetry. Concretely, Figs 5 and 6 show the benefit is also present when we combine APE with exemplar optimization. Unlike ProTeGi, APE generates new instructions only by paraphrasing old instructions without altering their semantic meaning (e.g., “Let’s think step by step” -> “Let’s follow a step-by-step process”). This operation arguably does not require a particularly strong model. Yet, we can still observe improvement – this, in our opinion, alludes to the presence of certain inherent synergy. We will nevertheless incorporate the reviewer’s remarks in our discussion under “Insight 3”; additionally, we also believe that the reviewer’s suggestion can be of independent interest as an effective, low-cost distillation from a powerful model to a smaller model through prompts, which are fixed at inference.\n\n> Descriptions of APE vs OPRO and faithfulness to original algorithms\n\nWe hope the reviewer can sympathize with our rather condensed descriptions of the algorithms due to the space constraint – indeed, the key difference between APE and OPRO is that whereas APE explicitly requests paraphrasing of the better-performing instructions, OPRO treats the LLM as a black box and let the LLM to determine its actions for the next generation, and we will highlight this difference better in the final version of the paper. Regarding the remark on the faithfulness of the algorithms, we note that where possible, we adapt the official implementation. Some changes are inevitable, including but not limited to parity of the number of generations, eval budgets, and initialization across the methods. These are to ensure broad fairness across the methods, and we believe should not lead to a significant discrepancy in findings.\n\n> Note about self-generation (Line 199-201)\n\nAs discussed in the common response, we do not assume annotated rationales in the validation set, which has made self-generated exemplars important, and we note that such a design decision has also been taken in previous works like DSPy and many-shot ICL. In the final version of the paper, we will provide a concrete example and incorporate the discussions in the common response to make this point clearer. We will also clarify further what we meant by “exemplar selection” to strengthen the point that in tasks where reasoning is required, exemplars are at least partially bootstrapped from the model and do not entail a simple selection from a fixed set. We thank the reviewer for the suggestions, which will likely improve the clarity of the paper further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cm9sBVyQVr", "reviewer": "Reviewer_1DvC", "summary": "This work focus on evalution and comparison study of automatic prompt optimization (APO) methods. These methods are broadly categorized into instruction optimization (IO) and exemplar selection (ES). This paper seeks to bridge the gap between these two methods by comprehensively comparing the performance of representative IO and ES techniques both isolation and combination. This work found that how to select examplars can outweigh how to optimize instructions.", "questions": "1. What's the difference between methods from IO and ES in terms of the cost. Are there any differences in statistical characteristics between these two categories?", "limitations": "See weakness and question part.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The presentation of this work is good. From the textual part, the authors clearly present the definition and representative methods of IO and ES. From those graphical presentation, they clearly present the comparison results of different methods.\n2. The setting of experimental study is convincing. The sequence from 'initial instruction + No exemplars' -> 'Optimized instruction + No examplars' -> Optimized instruction + Random exemplars' -> 'Initial instruction + Optimized exemplars' -> 'Optimized instruction + Optimized exemplars' make the comparison results clear and convicing.\n3. The finding of this work is insightful, this work found that ES is more important than IO which a bit do not fit in with the current direction of the study. So, this work may help to provide a reference for future research.", "weaknesses": "1. In the experimental study part, this work only use two kinds of LLMs, PaLM 2 and Gemini, which is not quite adequate.\n2. The evaluation criteria are a bit simple, only considering the acc rate. For engneering requirements, how many efforts (e.g. number of tokens; expense of the APO process; how much level of ) will be required during the prompt optimization is also important. Other reasonable metrics maybe also of help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback! Please see below for our response to their questions and concerns.\n\n> More LLM choices\n\nWe thank the reviewer for their suggestions. We have included additional results on GPT and Gemini 1.5 below. With the new experiments added, we believe that our results now cover a comprehensive set of LLMs with a continuum of model caliber, especially with Gemini 1.5 model family [performing at or very close to the state of the art](https://arena.lmsys.org/) – made possible by the long-context ability, we also included an “All” column in Gemini 1.5 experiments, which includes all valid demonstrations without any selection. The new results are in near unanimous agreement with the key insights in the paper, and we believe this presents clear evidence that the patterns in the paper are not due to artifacts of individual models. We’d like to refer the reviewer to the common response for more details about the new experiments conducted.\n\n### gpt-3.5-turbo-0125*\n\n|         |    No ES | Random | Search | Δ ES      |\n|:-------:|---------:|-------:|--------|-----------|\n|  No IO  |     59.0 |   68.6 | 76.8   | **+17.8** |\n|   APE   |     63.0 |   68.9 | 78.4   | **+15.4** |\n| ProTeGi |     68.9 |   72.2 | 80.2   | **+11.3** |\n| Δ IO    | **+9.9** | +3.6   | +3.4   |           |\n\n### gemini-1.5-flash-001\n\n|         |    No ES | Random | Nearest | Diversity | All  | Search | Δ ES     |\n|:-------:|---------:|-------:|---------|-----------|------|-------:|----------|\n|  No IO  |     75.1 |   80.0 | 81.7    | 81.5      | 80.4 |   83.3 | **+8.2** |\n|   APE   |     77.5 |   81.2 | 83.7    | 81.6      | 81.2 |   85.0 | **+7.5** |\n| ProTeGi |     80.4 |   82.4 | 82.6    | 82.3      | 83.5 |   84.5 | +4.1     |\n| Δ IO    | **+5.3** | +2.2   | +2.0    | +0.8      | +3.1 | +1.8   |          |\n\n> Additional metrics and evaluation criteria, ..., What's the difference between methods from IO and ES in terms of the cost. Are there any differences in statistical characteristics between these two categories?\n\nWe mainly used accuracy because **we controlled the cost and ensured the different IO and ES methods have approximate fairness in terms of the number of iterations** (each iteration involves the evaluation of a prompt on the validation set) to ensure the validity of any conclusion drawn from performance comparison (see caption of Table 1; note that instruction optimization methods also require calls to an optimizer model that are typically more capable and expensive; exemplar optimization requires no such models) – we will clarify this further in the table captions when we get a chance in revising the paper. For example, in all result tables like Tables 1-3, we have used color-coding to highlight the approximate computational costs in terms of the number of LLM calls (in Fig 6, we further show that the improvement from orange cells which use twice as many evaluations does not simply come from the additional compute). We used the number of iterations (each iteration corresponds to a full evaluation of a prompt on the validation set) as the main cost metric – most previous works we cited in Sec 2 such as OPRO and PromptAgent used the same metric.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XqS9KLauMu", "reviewer": "Reviewer_Hw9e", "summary": "This paper profoundly discusses how the deisgn of Automatic Prompt Optimization (APO) methods influence the performance of current instruction-following pre-trained large language models (LLMs). Concretely, the authors first systematically summarize and compartmentalize current APO methods into two lines: instruction optimziation (IO) and exempler sleection (ES), with their mathematical definitions. Then a series thorough and incremental experiments are carried out, of which primary discoveries includes: 1) reusing model-generated data (obtained from validation phase) as exempler shows performance improvement against standalone IO methods, with better efficiency. 2) combining IO and ES together would bring \"1+1>2\" performance improvement. Under these experimental results, the authors claim that studying ES methods should be put as top priority of APO studies.", "questions": "see Weaknesses.", "limitations": "The authors have summarized the limitations of this work in Conclusion of the paper, which are objective. I strongly suggest the authors include more datasets (e.g., multimodal tasks) and models (e.g., GPT series) in their future works as they promised.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This paper is very well-writen. The related works are appropriately summarized. The mathematical definition of APO, IO and ES are crystal clear and hence helps identify the difference between IO and ES. The experimental results are carefully arranged and presented.   \n2. The findings, though obtained from limited tasks and models, are still well supported by the current experimental results. It is very meaningful for the APO community, and deserves more attention and further investigation and development.", "weaknesses": "After reading this paper from top to the bottom, I found my concerns revolve around limited evaluation: I would appreciate the authors if they could at least conduct an additional experiment on GPT-3.5 Turbo to reinforce the convincingness of all findings, since GPT models hold quite some users. To summarize, although I fully understand that ideally comprehensive evaluation experiment is never easy, the above aspects deserve to be investigated, at least. Besides, the contribution of this paper mainly involves benchmarking and concluding meaningful findings, which in my view, should be more suitable for Benchmark Track of NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback, which also included clear, actionable items (e.g., GPT evaluations). We’d be grateful if the reviewer could review our response below. We also hope they could consider revising their rating if they feel their concerns have been adequately addressed.\n\n> Limited evaluations.\n\nWe thank the reviewer for their suggestions. We have included additional results on GPT, as the reviewer suggested, and Gemini 1.5 below. With the new experiments added, we believe that our results now cover a comprehensive set of LLMs with a continuum of model caliber, especially with the Gemini 1.5 model family  [performing at or very close to the state of the art](https://arena.lmsys.org/) – made possible by the long-context ability, we also included an “All” column in Gemini 1.5 experiments, which includes all valid demonstrations without any selection. The new results are in near unanimous agreement with the key insights in the paper, and we believe this presents clear evidence that the patterns in the paper are not due to artifacts of individual models. We’d like to refer the reviewer to the common response for more details about the new experiments conducted.\n\n### gpt-3.5-turbo-0125*\n\n|         |    No ES | Random | Search | Δ ES      |\n|:-------:|---------:|-------:|--------|-----------|\n|  No IO  |     59.0 |   68.6 | 76.8   | **+17.8** |\n|   APE   |     63.0 |   68.9 | 78.4   | **+15.4** |\n| ProTeGi |     68.9 |   72.2 | 80.2   | **+11.3** |\n| Δ IO    | **+9.9** | +3.6   | +3.4   |           |\n\n### gemini-1.5-flash-001\n\n|         |    No ES | Random | Nearest | Diversity | All  | Search | Δ ES     |\n|:-------:|---------:|-------:|---------|-----------|------|-------:|----------|\n|  No IO  |     75.1 |   80.0 | 81.7    | 81.5      | 80.4 |   83.3 | **+8.2** |\n|   APE   |     77.5 |   81.2 | 83.7    | 81.6      | 81.2 |   85.0 | **+7.5** |\n| ProTeGi |     80.4 |   82.4 | 82.6    | 82.3      | 83.5 |   84.5 | +4.1     |\n| Δ IO    | **+5.3** | +2.2   | +2.0    | +0.8      | +3.1 | +1.8   |          |\n\n\n> “the contribution of this paper mainly involves benchmarking and concluding meaningful findings, which in my view, should be more suitable for Benchmark Track of NeurIPS.”\n\nWe thank the reviewer for appreciating our extensive benchmarking and meaningful findings. We believe that the main track is more suitable, as we have not proposed a new benchmark (we reused existing datasets) in this study, but our goal is to shed light on the best practices for prompt design, considering the optimization holistically. We also proposed a simple and effective framework by optimizing both exemplars and instructions that show significant improvements in downstream task performance.\n\n> Additional datasets\n\nWe thank the reviewer's suggestions. As mentioned in the conclusion, we chose BIG-Bench and MMLU because, as noted on line 165, they are widely used in contemporary automatic prompt optimization (APO) papers. These datasets cover numerous challenging tasks, including reasoning and information-seeking types, and their performance improvements correlate with human-in-the-loop settings like LMSys. Our first objective, as stated on line 62, is to compare performance gains using IO and ES methods on a fairground. Thus, these datasets are ideal since the original authors also use them to showcase their methods. Having said that, we agree and reiterate that incorporating more open-ended and realistic tasks is crucial for our study and the broader APO field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYciRChMs1", "reviewer": "Reviewer_uqie", "summary": "This paper provides a comprehensive investigation of instruction optimization and exemplar selection.", "questions": "- The total number of prompt evaluations is selected as $m=32$. In my impression, this is in general a small number compared to the other related works on IO or ES. For example, OPRO uses more than 100 evaluations in most of its experiments. I wonder will the conclusions change if a larger $m$ is used?\n- It has been widely reported that the ordering of the exemplars matters a lot. I wonder how do you determine the ordering?", "limitations": "A limitation is discussed in the Conclusion section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is very well motivated, and studies a very important problem, which is the relative importance of exemplar selection and instruction optimization. Therefore, the paper is likely to inspire ideas for future research in these ares.\n- The paper is well written and well organized. I particularly like the way some results are presented, such as Table 1 and Figure 6. These results are very nicely organized and clearly presented, effectively conveying the intended insights.\n- A particularly nice feature of the proposed algorithm is that it only uses the single validation set for both IO and ES.", "weaknesses": "- The performances achieved by the different combinations of algorithms are indeed good, however, in terms of the relative comparison of the importance of IO and ES, I think it is possible that an unfair advantage is given to ES. First of all, in all methods studied in this paper, IO is performed first followed by ES, this allows ES to adapt to an already optimized IO, while IO is not able to adapt to exemplars since it is performed first before any ES. In addition, only those exemplars on which the optimized instruction predicts correctly are used for ES, this also gives an advantage to ES since it uses information from an already optimized instruction, whereas IO is not able to exploit information from already optimized exemplars since IO is performed first. Therefore, it puts into question the reliability of a major conclusion drawn from the paper, i.e., ES is much more important than IO.\n- As mentioned in the last paragraph of Section 2, to the best of my knowledge, there is indeed a general lack of works on joint IO and ES. I'd like to point the authors to two concurrent works along this line: https://arxiv.org/abs/2405.18369, https://arxiv.org/abs/2405.16122. The second work may be of more interest, because it is exactly using something similar to the combinatorial BO approach which is alluded to on lines 295-296.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful and positive comments! Please see below for our response. We hope that in light of our response, the reviewer will consider revising their rating if they feel their concerns have been sufficiently addressed. **To fully address the reviewer's concerns, we have to utilize an additional comment to complete our response. We would like to thank the reviewer in advance for their time and effort in reading through our reply**.\n\n> Unfair advantage to ES\n\nFirstly, we’d like to point out that the relative importance of ES and IO has been studied both *in isolation* and *when combined*. While the reviewer is correct that we performed ES after IO *when they are combined* (we will respond to this in the next reply), **the relative importance was first studied when each is performed separately**. With reference to Table 1-3, the results on the first row (“No IO”) are all obtained **without any instruction optimization**, and as we highlighted in bold on Page 6, optimized exemplars on top of seed instructions still outperform optimized instructions but with unoptimized exemplars, relative to the common reference point (i.e., seed instruction without exemplars). This is our first finding that supports our Insight 2, and since the two modes of optimization are performed separately under the exact same setup (i.e., the same call budgets and the same number of labeled data) over the common starting point, there is no unfair advantage of one over another in this experiment when we derive their relative importance.\n\n> IO is performed first followed by ES, this allows ES to adapt to an already optimized IO, while IO is not able to adapt to exemplars since it is performed first before any ES.\n\nThe main reason for us to place ES after IO in the paper was because we only assume labels, not annotated rationales, in the validation set to ensure fairness: obtaining additional annotations incurs extra costs and could invalidate our claims if ES outperforms IO due to access to these additional data sources, which are not required or used by the IO methods. To generate rationales, we reuse model outputs from instruction evaluations on the validation set — since the model outputs depend on individual instructions, it makes sense to optimize instructions first, followed by optimization on the exemplars conditioned on that instruction. \n\n**Following the reviewer’s suggestions, we also performed experiments with the opposite order** where in Stage 1, we optimize the exemplars based on the seed instruction, and in Stage 2, we freeze the exemplars and optimize the instruction. We performed experiments on a subset of all tasks using both Gemini 1.0 and 1.5 models, using APE to optimize instructions and random search to optimize exemplars with a total budget of m=32 (the setup presented in Fig 6 of the paper). We find that using the swapped order (ES-IO) led to comparable aggregated results (on Gemini 1.0: IO first: 78.5%; ES first: 79.7%; on Gemini 1.5: IO first: 87.8%, ES first: 87.0% – please refer to the common response for task-specific breakdown). We believe these new experiments provide sufficient evidence that the conclusion drawn in the paper is not due to any bias created by a specific ordering — we will incorporate a more comprehensive evaluation of this in the camera-ready copy.\n \nBeyond a simple two-stage design, we agree that there can be alternative ways to allow instructions to adapt to exemplars. In fact, we have already explored this in Appendix B.8, Page 36, where in Algorithm 1, we interleaved instruction and exemplar optimization, and in this case, **optimized instructions and optimized exemplars can adapt to each other**. As mentioned in Line 1087, this algorithm achieved a comparable performance compared to the two-stage approach, so we presented the latter in the main text because we believe that simplicity should be favored when the performance is similar. Still, we agree that better integration of exemplar and instruction optimization can be an important open research question, as we also mentioned towards the end of Page 8. Appendix B.8. We will clarify this point better in the final version of the paper.\n\n> only those exemplars on which the optimized instruction predicts correctly are used for ES.\n\nAs discussed, we’d like to emphasize that 1) the exemplars are generated from the validation set, *whose labels are required anyway* for performance estimation, and selecting the correct predictions is a simple filtering step that requires neither additional data nor computation. Instruction optimizers utilize the validation set in the same way by using the validation accuracy as the optimization objective – to compute this metric; we also have to count the number of correctly predicted samples. ES is essentially doing the same, except that rather than simply counting and aggregating the instances, it also traces back to the individual samples that make up the metric – in this sense, using the correctly predicted outputs as exemplars does not create unfairness per se. 2) Bootstrapping correct predictions is also commonly used in previous and contemporary works [1, 2]: In DSPy [1], Section 3.3, Page 6, the BootstrapFewShot teleprompter “collect(s) demonstrations of each module (i.e., examples of its input-output behavior) *that collectively lead to valid output*”). In [2], Section 3, Page 5, the authors “select rationales *that obtain the correct final answer*, …, and arrange them into in-context examples” for reinforced ICL. Thus, while we agree that better use of negative feedback can be a very promising future direction, not using it in the present work should not be a weakness of our analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q4cd2vZGp5", "reviewer": "Reviewer_1T7N", "summary": "The paper studies the instruction optimization and exemplar selection in auto prompt optimization. It has the following contributions:\n1) Showing that optimizing ES improves significantly over IO and the importance of ES\n2) Showing the synergy effect between ES and IO\n3) Extensive experiments are conducted to validate the conclusions", "questions": "1. The experiments are all conducted on PaLM and Gemini which are from the same company and hence some similar underlying designs. Can the results generalize to other major LLMs, e.g., GPT-4, Claude, Llama.\n2. How does the author implement Nearest? Is it finding the nearest for each test input? or for a whole validation dataset?\n3. ES after IO is used to run the result in Table 1 and 2, have the authors tried IO after ES, will the result change?", "limitations": "The limitations are stated in the weakness. It mainly lies in the mismatch between experimental settings and the insights.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written with good presentation\n2. The formulation of ES and IO as optimization is clear\n3. The experiments on ES and IO and a mix-matching of these two are clear and extensive\n4. The idea of comparing ES and IO is novel to me", "weaknesses": "1. The authors have made a significant effort in summarizing the existing works, however, there are still some of the SOTA IO methods not mentioned:\n- Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers: https://arxiv.org/abs/2309.08532\n- Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers: https://arxiv.org/abs/2310.02905\n2. Some of the baselines are mentioned in related work, however, the author did not compare with them (e.g., InstructZero). This can be important since these methods are SOTA methods of IO and may potentially produce much better results than the methods compared in this paper. It is true that the authors use simple methods for ES, however, do note that exemplars are natural data points (good data) while instructions are usually generated differently by different methods and hence a more careful design is needed. \n3. The no IO is based on a very good instruction “let’s think step by step” while no ES is just no examplar at all. Therefore for fair comparison /drawing insights from the experiments, the improvement on ES needs to be compared to random (or at least nearest) to say which optimization is more important.\n4. As the author mentioned in mix-matching section, the improvement of the R.S., Mutation in Table 1, 2 can be because of the extra computation/queires to LLM used in the search phase. How much does the delta in these tables are from this factor? And how would this affect the conclusion of comparing the importance of ES and IO? These are unclear from the current experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive and positive comments about our work. Please see below for our point-by-point response, which we believe has addressed the reviewer’s concerns thoroughly. We hope that the reviewer could consider revising their score if they feel the same way. **To fully address the reviewer's concerns, we have to utilize an additional comment to complete our response. We would like to thank the reviewer in advance for their time and effort in reading through our reply.**\n\n> Additional related works, ..., Some of the baselines are mentioned in related work, however, the author did not compare with them.\n\nWe thank the reviewer for bringing related works to our attention. We actually mentioned the first work (reference 14) in Sec 2, and we will discuss the second work in the camera-ready version. Due to the pace of advancements in this field, we hope the reviewer will sympathize with us, saying that it is not feasible to experimentally compare all methods proposed, and thus we opted to include the methods that are representative enough across the different genres. We believe that both works mentioned can be deemed to belong to the family of methods with evolutionary-like design spearheaded by APE and subsequently improved by better search algorithms and/or novel operators. The first work explicitly used an evolutionary framework, most notably with the new crossover operators. In contrast, the second work used an enhanced surrogate for Bayesian optimization, which is more sample-efficient than evolutionary algorithms and more effective than Gaussian processes. Still, on a high level, it has similar optimization goals. Both works, however, only optimize instructions and fix exemplars, e.g., on the footnote of Page 3 of the second work, the authors explicitly said they “*omitted the dependence on these factors (the set of exemplars) here as they are kept fixed for a task*”. This is exactly the kind of argument we mentioned on Line 56 about \"focusing on one objective at a time.\" As discussed throughout the paper, we argue that while reasonable, this separate treatment risks missing out on (significant) benefits that we demonstrate from treating exemplars and instructions holistically -- thus, we believe that even if these works outperform IO baselines, they should still benefit from better exemplar optimization as their search space is essentially a subspace of the joint instruction-exemplar search space. \n\n> The no IO is based on a very good instruction “let’s think step by step” while no ES is just no examplar at all. The improvement on ES needs to be compared to random (or at least nearest) to say which optimization is more important.\n\nWe agree that the reviewer’s suggestion should be the preferred practice in an ideal world where the importance of jointly optimizing exemplars and instructions is well-known. However, as we noted in Lines 133-135, many prominent instruction optimization methods still do not use exemplars (not even random sampling), even though this seems to be a “free lunch” in a way, based on our finding – this also puzzled us, and is one of the initial motivations for writing the paper, and we chose the respective reference point (“Let’s think step by step” and no exemplars) also by following previous works – for example, OPRO and PromptAgent made the exact same design decision. \n\nSecondly, based on Table 1-3, it is clear that **even if we set the reference point to Random, the difference between the *best ES method* and *random* still outweighs the performance difference across virtually any IO strategies**, which suggests that choosing *which* ES strategy is still the dominant decision; we can also observe that the gap between different IO methods shrinks further with random exemplars. Thus, we believe that the choice of the reference point will not significantly affect the paper’s conclusion. We nevertheless believe that the reviewer’s suggestion is valuable, and we will incorporate the above discussion into the final version of the paper.\n\n> How much does the delta in these tables are from this factor? And how would this affect the conclusion of comparing the importance of ES and IO? These are unclear from the current experiments.\n\nWe’d like to refer the reviewer to Fig 6 and the paragraph starting on Line 304 – we emphasize that in Fig 6  (and Table 16 on Page 35), the total evaluation budget is the same as the ES/IO only (blue) cells in Table 1-2 to ensure fairness and to precisely remove the confounding factor of difference in compute mentioned by the reviewer. It is clear the benefit from mix-and-match persists, and the difference attributable to additional compute (i.e., the difference with the rightmost cell in the table of Fig 6) is minimal. In general, we used color coding to denote the optimization cost – RS and Mutation in Tables 1-2 the reviewer referred to are colored orange because they used double the amount of computation. The blue cells in the table in Fig 6 feature similar mix-and-match results but with halved budgets so that they consume the same budget as the other blue cells in Table 1-2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTllQ1YfBO", "reviewer": "Reviewer_dxC3", "summary": "This paper aims to study the performance impact of instruction optimization (IO) and exemplar selection (ES) both isolation and combination, as the automatic prompt optimization literature mostly only focuses on either IO or ES. Its empirical results on a wide range of tasks (e.g., BBH and MMLU) show that optimizing ES may bring more performance gain than IO, and highlight the importance of studying ES and its combination with IO.", "questions": "1. How did you choose the seed instruction? Do different seed instructions have different outcomes?\n\n2. In Table 3, why ProTeGi is worse than (NO IO) if its optimization starts from the same seed instruction?\n\n3. If you revert the process, that means you start ES first and run IO conditioned on the best ES you found, will this make the $\\Delta IO$ greater than  $\\Delta ES$?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Compared with previous works on APO, this work provides relatively comprehensive experiments to show combining IO with ES can further boost the APO performance.\n- This work examines quite a number of combinations of different IO and ES approaches.", "weaknesses": "1. The paper only considers PaLM 2 and Gemini Pro (which are from the same source) as the target models. It is meaningful to see if the conclusion (i.e., ES outweighs IO) still holds for different target models (e.g., ChatGPT/GPT-4).\n\n2. Some important settings are not explained. For example, in line 199, why ES is only performed from the exemplars self-generated by the target model, instead of choosing from $\\mathcal{D}_{val}$. For $\\mathcal{D}_c(I^*)$, only selecting the correctly predicted samples may not bring the highest performance, as some works show that wrong exemplars in ICL may be more valuable. In many IO works, they typically use more than 100 query evaluation budgets, instead of choosing $m=32$ naively in this paper.\n\n3. The authors have the following objective in their paper:\n>We aim to meet this by evaluating ES and IO in the context of APO by answering the following ... 2) How do we make the **optimal use** of the limited data and computational budget under the current APO framework?\n\nBecause of the synergy between ES and IO, the optimal use of ES and IO should exist at the *joint optimization* of ES and IO. However, the authors have only shown the performances of combining independently optimized instruction and optimized exemplars conditioned on that instruction. The optimality should be found when the exemplars and instructions are optimized together and the corresponding combinatorial optimization problem should cater to the space $\\mathcal{I}\\times\\mathcal{E}^k$ rigorously. Intuitively, the optimized instruction should also depend on the given exemplars at each iteration. So, if the authors want to completely study how the effect of combining IO and ES, they should consider this joint optimization problem. Unfortunately, to the best extent, this paper only empirically shows (through brute force) that there may exist an optimal allocation of budgets $m_{IO}$ and $m_{ES}$ used for running the 'mixing-and-matching' strategy of ES and IO, which is far from the so-called **optimal use**.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive and insightful comments. We’d like to refer the reviewer to our point-by-point response below and also the common response to all reviewers. We hope that in light of our response, the reviewer could consider revising their rating if they feel their concerns have been adequately addressed. **To fully address the reviewer's concerns, we have to utilize an additional comment to complete our response. We would like to thank the reviewer in advance for their time and patience while reading our reply.**\n\n> Only considers PaLM 2 and Gemini Pro (which are from the same source)\n\nFirst, we’d like to emphasize that PaLM 2 and Gemini are created by the same company but do not share the “same source,” which we assume refers to the training data mixture, algorithms, and/or other protocols [1, 2]. Second, **we do acknowledge the reviewer’s concerns and add new results on GPT and Gemini 1.5 and we’d refer the reviewer to the common response for details** but we also present the key insights below. With the new experiments added, we believe that our results now cover a comprehensive set of LLMs with a continuum of model caliber, especially with Gemini 1.5 [performing at or very close to the state of the art](https://arena.lmsys.org/). The new results are in near unanimous agreement with the key insights in the paper, and we believe this presents clear evidence that the patterns in the paper are not due to artifacts of individual models.\n\n### gpt-3.5-turbo-0125*\n\n\n|         |    No ES | Random | Search | Δ ES      |\n|:-------:|---------:|-------:|--------|-----------|\n|  No IO  |     59.0 |   68.6 | 76.8   | **+17.8** |\n|   APE   |     63.0 |   68.9 | 78.4   | **+15.4** |\n| ProTeGi |     68.9 |   72.2 | 80.2   | **+11.3** |\n| Δ IO    | **+9.9** | +3.6   | +3.4   |           |\n\n### gemini-1.5-flash-001\n\n|         |    No ES | Random | Nearest | Diversity | All  | Search | Δ ES     |\n|:-------:|---------:|-------:|---------|-----------|------|-------:|----------|\n|  No IO  |     75.1 |   80.0 | 81.7    | 81.5      | 80.4 |   83.3 | **+8.2** |\n|   APE   |     77.5 |   81.2 | 83.7    | 81.6      | 81.2 |   85.0 | **+7.5** |\n| ProTeGi |     80.4 |   82.4 | 82.6    | 82.3      | 83.5 |   84.5 | +4.1     |\n| Δ IO    | **+5.3** | +2.2   | +2.0    | +0.8      | +3.1 | +1.8   |          |\n\n> ES is only performed from the exemplars self-generated by the target model instead of choosing from D_val.\n\nWe’d like to note that the tasks considered, which are extensively used in automatic prompt optimization papers, require intermediate logical/numerical reasoning from the LLM to reach the correct answer. We, however, **only assume D_val to contain the final answer labels but not the intermediate steps**: Obtaining additional annotations incurs extra costs and could invalidate our claims if ES outperforms IO due to access to these additional data sources, which are not required or used by any IO methods we surveyed. This setup is consistent with previous and concurrent works [1,2]. We’d also like to clarify that this is a common setup – virtually all instruction optimization methods require validation sets without rationales, and it is used in popular ES frameworks like DSPy [4]. Quoting the DSPy authors in Page 6:\n\n    In practice, we typically assume labels only for (at most) the program’s final output, not the intermediate steps. This label efficiency is critical for modularity: building a new pipeline in DSPy requires simply recompiling the new pipeline’s code, not annotating data specific to the new pipeline.\n\nGiven this, **simply sampling (input, final answer) from D_val to as exemplars are known to be sub-optimal as they contain no intermediate answers**, and they essentially encourage direct answering over chain-of-thought reasoning [3] – the suboptimality of direct answering is precisely why chain-of-thought prompting has been tremendously helpful. Without annotated intermediate steps, using model-generated rationale is common in previous works like DSPy [4] and concurrent works like [5] (referred to as “reinforced ICL”). Given that the first objective of our study is to derive generalizable insights for more optimal methods that jointly consider instruction and exemplars in a canonical framework, we believe it makes sense to follow previous works in this aspect. Also, as mentioned, the model-generated responses are free side-products whenever we evaluate an instruction on the validation set, and re-using them as exemplars comes at no additional cost.\n\n> only selecting the correctly predicted samples may not bring the highest performance, as some works show that wrong exemplars in ICL may be more valuable.\n\nWhile we agree the wrong answers are valuable (e.g., they are already used as gradient signals for self-reflection in ProTeGi) and we are aware of works using wrong exemplars in ICL for contrastive learning [6], to our knowledge, **using correct demonstrations is still the mainstream approach** and it’s known that modern LLMs are particularly sensitive to the correctness of exemplars [7]. Furthermore, as mentioned, **selecting from the correctly predicted samples as exemplars is common** in previous and concurrent works: In [4], Sec. 3.3, Page 6, the BootstrapFewShot teleprompter “collect(s) demonstrations of each module (i.e., examples of its input–output behavior) *that collectively lead to valid output*”). In [5], Sec. 3, Page 5, the authors “select rationales *that obtain the correct final answer*, …, and arrange them into in-context examples” for reinforced ICL. Therefore, while we agree that better use of negative feedback is a very promising future direction, not using it in the present work should not be a weakness per se.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time and feedback! In the common response below, we’d like to provide in-depth answers to some of the questions asked by multiple reviewers.

## Additional LLMs (Reviewers 1DvC, Hw9e, dxC3)

Multiple reviewers asked for evaluations on additional LLMs. We acknowledge this feedback and have validated the key findings with GPT (gpt-3.5-turbo-0125) and Gemini 1.5 (gemini-1.5-flash-001). Please see the table below for aggregated results; for task-specific breakdown similar to Fig 4, we refer the reviewer to Fig. S1 (GPT) and S2 (Gemini 1.5) in the uploaded pdf.

### gpt-3.5-turbo-0125*

|         |    No ES | Random | Search | Δ ES      |
|:-------:|---------:|-------:|--------|-----------|
|  No IO  |     59.0 |   68.6 | 76.8   | **+17.8** |
|   APE   |     63.0 |   68.9 | 78.4   | **+15.4** |
| ProTeGi |     68.9 |   72.2 | 80.2   | **+11.3** |
| Δ IO    | **+9.9** | +3.6   | +3.4   |           |

### gemini-1.5-flash-001

|         |    No ES | Random | Nearest | Diversity | All  | Search | Δ ES     |
|:-------:|---------:|-------:|---------|-----------|------|-------:|----------|
|  No IO  |     75.1 |   80.0 | 81.7    | 81.5      | 80.4 |   83.3 | **+8.2** |
|   APE   |     77.5 |   81.2 | 83.7    | 81.6      | 81.2 |   85.0 | **+7.5** |
| ProTeGi |     80.4 |   82.4 | 82.6    | 82.3      | 83.5 |   84.5 | +4.1     |
| Δ IO    | **+5.3** | +2.2   | +2.0    | +0.8      | +3.1 | +1.8   |          |

*Due to rate limits and time constraints, for these sets of newly added experiments, we tested a subset of 11 tasks, but we can already observe patterns consistent with our claims in the paper. We will endeavor to include a full set of results in the camera-ready version. The Gemini 1.5 results are already full sets and are comparable to Tables 1-2 in the paper.

With the new experiments added, we believe that our results now cover a comprehensive set of LLMs with a continuum of model caliber, especially with the Gemini 1.5 model family [performing at or very close to the state of the art](https://arena.lmsys.org/) – made possible by the long-context ability, we also included an “All” column in Gemini 1.5 experiments, which includes all valid demonstrations without any selection. The new results are in near unanimous agreement with the key insights in the paper, and we believe this presents clear evidence that the patterns in the paper are not due to artifacts of individual models.

## Ordering of ES and IO or better joint optimization techniques (Reviewers Uqie, 1T7N, dxC3)

Several reviewers asked why we had exemplar selection after instruction optimization and whether alternative ordering would be possible. The main reason for us to place ES after IO in the paper was that we only assume labels, not annotated rationales, in the validation set to ensure fairness: obtaining additional annotations incurs extra costs and could invalidate our claims if ES outperforms IO due to access to these additional data sources, which are not required or used by the IO methods. To generate rationales, we reuse model outputs from instruction evaluations on the validation set — since the model outputs depend on individual instructions, it makes sense to optimize instructions first, followed by optimization on the exemplars conditioned on that instruction. 

Following the reviewer’s suggestions, we also performed experiments in the opposite order where, in Stage 1, we optimized the exemplars based on the seed instruction, and in Stage 2, we froze the exemplars and optimized the instruction. We performed experiments on a subset of all tasks using both Gemini 1.0 and 1.5 models, using APE to optimize instructions and random search to optimize exemplars with a total budget of m=32 (the setup presented in Fig 6 of the paper). We find that using the swapped order (ES-IO) led to comparable aggregated results. We refer the reviewers to detailed [Tables S1 and S2](https://openreview.net/forum?id=IdtoJVWVnX&noteId=cw2sfTOORc) in the additional comment below, which we believe provides sufficient evidence that the conclusion drawn in the paper is not due to any bias created by a specific ordering — we will incorporate a more comprehensive evaluation on this in the camera-ready copy.

On a related note, beyond a simple two-stage approach, as Reviewer dxC3 aptly suggested, a better strategy in principle is to **jointly optimize instructions and exemplars and allow them to co-evolve by alternating between instruction and exemplar optimization successively in a multi-stage manner. We’d like to note that we already considered a possible implementation of it in Appendix B.8**. As mentioned in Line 1087, we found the algorithm (Algorithm 1) achieved a comparable performance (74.6%) compared to the two-stage approach (74.5%, which corresponds to the m_io=16, m_es=16 result in Fig. 6 – see [Table S3](https://openreview.net/forum?id=IdtoJVWVnX&noteId=cw2sfTOORc) for task-specific breakdown) on PaLM 2, so we presented the latter in the main text based on the parsimony principle of Occam’s razor that simplicity should be favored when the performance is similar.  Nonetheless, we agree that better-integrating exemplar and instruction optimization is an important, open research question, as mentioned towards the end of page 8.

Meta-review: Following the author rebuttal, all six reviewers have expressed positive ratings for this paper. I appreciate the efforts made by the authors; their work demonstrates novelty and potential impact, making it suitable for NeurIPS. However, the authors should consider the necessary revisions in the final version, particularly those highlighted by the reviewers.
Decision: Accept (poster)