Prompt Engineering a Prompt Engineer (2024)

Abstract: Prompt engineering is a challenging yet crucial task for optimizing the performance of large language models on customized tasks. It requires complex reasoning to examine the model’s errors, hypothesize what is missing or misleading in the current prompt, and communicate the task with clarity. While recent works indicate that large language models can be meta-prompted to perform automatic prompt engineering, we argue that their potential is limited due to insufficient guidance for complex reasoning in the meta-prompt. We fill this gap by infusing into the meta-prompt three key components: detailed descriptions, context specification, and a step-by-step reasoning template. The resulting method, named PE2, showcases remarkable versatility across diverse language tasks. It finds prompts that outperform “let’s think step by step” by 6.3% on MultiArith and 3.1% on GSM8K, and outperforms competitive baselines on counterfactual tasks by 6.9%. Further, we show that PE2 can make targeted prompt edits, rectify erroneous prompts, and induce multi-step plans for complex tasks.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 355–385 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
Large language models (LLMs) are powerful tools for many natural language processing tasks when given the right prompts.1 However, due to the model’s sensitivity (Jiang et al., 2020; Zhao et al., 2021; Reynolds and McDonell, 2021; Lu et al., 2022), finding the optimal prompt can be challenging, often necessitating extensive manual trial-anderror efforts. Moreover, once an initial prompt is deployed into production, unforeseeable edge cases may emerge, demanding more rounds of manual efforts to further refine the prompt. These challenges give rise to an emerging research field of
†Work done while interning at Microsoft. 1In this paper, we limit the scope to textual prompts of task description (e.g., “Translate English to French”) or instruction (e.g., “Let’s think step by step”).
methods typically use a meta-prompt that guides an LLM to inspect the current prompt , provide feedback (sometimes refered to as textual “gradients”) and then generate an updated prompt . In this paper, we design and investigate meta-prompt variants to guide LLMs to perform automatic prompt engineering more effectively.
automatic prompt engineering. Within this field, a notable line of methods involves leveraging the capabilities of LLMs themselves (Zhou et al., 2023b; Pryzant et al., 2023). This entails meta-prompting LLMs with instructions such as “inspect the current prompt and a batch of examples, provide feedback, then propose a new prompt.” (See Figure 1)
While these methods achieve impressive performance, a subsequent question arises: What makes a good meta-prompt for automatic prompt engineering? To answer this question, we connect two key observations: (1) Prompt engineering, at its core, is a natural language generation task that requires complex reasoning: it involves closely examining the model’s errors, hypothesizing what is missing or misleading in the current prompt, and communicating the task more clearly to the LLM. (2) Complex reasoning capabilities in LLMs can be elicited by prompting the model to “think step by step” (Wei et al., 2022a; Kojima et al., 2022) and can be further improved by instructing them to
1 355
reflect on their outputs (Madaan et al., 2023; Chen et al., 2023).
Bridging these two observations, in this work, we prompt engineer a prompt engineer—we aim to construct a meta-prompt that guide LLMs to perform prompt engineering more effectively. We argue that prior works do not provide sufficient guidance in the meta-prompt, thereby limiting the potential of LLMs for automatic prompt engineering. To address this, we introduce new meta-prompt components such as detailed two-step task descriptions, context specification, and a step-by-step reasoning template, to better equip LLMs throughout the process (§3; Fig. 3).
The resulting method, named PE2, achieves strong empirical performance (§5.1). When using text-davinci-003 as the task model, the prompts produced by PE2 surpass the zero-shot chain-ofthought prompt “let’s think step by step" (Kojima et al., 2022) by 6.3% on MultiArith and 3.1% on GSM8K. Moreover, PE2 matches or outperforms two automatic prompt engineering baselines, Iterative APE (Zhou et al., 2023b) and APO (Pryzant et al., 2023) in multiple settings (Fig. 2). PE2 is most effective on counterfactual tasks (Wu et al., 2023), where the automatic prompt engineer is anticipated to reason about non-standard situations (e.g., do addition in base-8 instead of base-10) and explain such situation to the task model through the prompt. Beyond academic datasets, we show that PE2 can improve an expert-written production prompt consisting of over 5,000 tokens, resulting in an 8.0% increase in F1 score.
Upon examining the prompt edit history of PE2 (§5.3), we find that PE2 consistently offers meaningful prompt edits (Table 4). It is able to amend erroneous or incomplete prompts and enrich the prompts with additional details, which leads to improved final performance. It is also able to devise multi-step plans for complex tasks. For example,
in the task of movie recommendation, PE2 makes the plan to “consider factors such as genre, plot and style” in the prompt. Interestingly, when uninformed about performing addition in base-8, PE2 formulates its own arithmetic rules from the examples: “If both numbers are less than 50, add 2 to the sum. If either number is 50 or greater, add 22 to the sum.” This demonstrates PE2’s non-trivial ability to reason and adapt in counterfactual situations, while also raises concerns of “shortcut learning” in prompt optimization.
2 Background
In this section, we provide a formal formulation of the prompt engineering problem (§2.1), and describe a general framework of automatic prompt engineering using LLMs and meta-prompts (§2.2). Building on this foundation, we introduce new meta-prompt components used in PE2 in §3.
2.1 Prompt Engineering
The goal of prompt engineering is to find the textual prompt p∗ that achieves the best performance on a given dataset D when using a given LLM Mtask as the task model. More specifically, we assume all datasets can be formatted as textual input-output pairs, i.e., D = {(x, y)}. We are given a training set Dtrain for optimizing the prompt, Ddev for validation, and Dtest for final evaluation. Following the notations in Zhou et al. (2023b), the prompt engineering problem can be described as:
p∗ = argmax p
∑
(x,y)∈Ddev f(Mtask(x; p), y) (1)
where Mtask(x; p) is the output generated by the model when conditioning on the prompt p, and f is a per-example evaluation function. For example, if the evaluation metric is exact match, f(Mtask(x; p), y) = 1[Mtask(x; p) = y].
2
2.2 Automatic Prompt Engineering with LLMs
To alleviate the intensive efforts of human prompt engineering, recent works explore automating this process by meta-prompting LLMs to paraphrase the prompt (Zhou et al., 2023b) or refine the prompt by inspecting failure examples (Pryzant et al., 2023). In the following, we describe a framework that encapsulates these prior works and is employed in our investigation in later sections. It has three parts: prompt initialization, new prompt proposal, and the search procedure.
Prompt Initialization. To start the prompt engineering process, a set of initial prompts P (0) is needed. We consider two initialization methods: (1) Manual initialization is applicable for tasks that has pre-existing prompts written by humans experts. For example, “Let’s think step by step” is effective on mathematical reasoning tasks and can be used as the initialization for prompt optimization. In (2) Induction Initialization, we follow Zhou et al. (2023b) by using a batch of examples {(x, y)} from Dtrain and a prompt pinit (“Here are the input-output pairs. What is the instruction?”) to generate a set of initial prompts P (0).
New Prompt Proposal. Given a set of initial prompts, the automatic prompt engineer will continuously propose new and potentially better prompts. At timestamp t, the prompt engineer is given a prompt p(t) and expected to write a new prompt p(t+1). Optionally, a batch of examples B = {(x, y, y′)} may be inspected in the new prompt proposal process. Here y′ = Mtask(x; p) represents model-generated output and y represents the ground-truth label. We use pmeta to denote a meta-prompt that is used to instruct the LLM Mproposal to propose new prompts. Therefore,
p(t+1) = Mproposal(p(t), B; pmeta) (2) Constructing a better meta-prompt pmeta to improve the quality of the proposed prompt p(t+1) is the main focus of this study. We will describe in more details in §3.
Search Procedure. As LLMs are sensitive to trivial prompt variations, it is possible that the newly proposed prompt p(t+1) under-performs the original prompt p(t). Therefore, automatic prompt engineering is typically combined with a back-tracking enabled search procedure. At timestamp t, we select n best-performing prompts from all prompt
candidates obtained in previous timestamps (i.e., P (0)∪P (1)∪...∪P (t)). For each of these n prompts, we sample m different batches B of model errors, and run the meta-prompt in Eq. 2 to produce m new prompts. This results in m×n new prompts, which we denote as P (t+1) collectively and are used at the next timestamp t+ 1. The search algorithm is described more formally in Algorithm 1.
3 Prompt Engineering a Prompt Engineer
Much like how the prompt plays an important role for the end task performance, the meta-prompt pmeta introduced in Eq. 2 plays an important role in the quality of newly proposed prompts, and the overall quality of automatic prompt engineering. In this work, we focus on prompt engineering the meta-prompt pmeta—we develop metaprompt components that can potentially help improve LLMs’ prompt engineering quality.
In the following, we reflect on the limitations of prior works and subsequently introduce three metaprompt components targeting these limitations. We visualize these components in Fig. 3. We name our method using these three components as PE2, a prompt engineered prompt engineer.
(a) Two-step Task Description. In APO (Pryzant et al., 2023) the task of prompt engineering is decomposed into two steps (Fig. 1): In step 1, the model is expected to inspect the current prompt and a batch. In step 2, the model is expected to generate an improved prompt. However, in APO, each step is explained briefly and on the fly. In contrast, we consider clarifying the steps and expectations upfront in the meta-prompt.
(b) Context Specification. In practice, how the prompt and the input text are formatted together is flexible. It may appear before the input text to describe the task, e.g., “Translate English to French.” It may appear after the input text, e.g., “Let’s think step by step” to elicit reasoning capabilities. Recognizing these varying contexts, we explicitly specify the layout of the prompt and the input.
(c) Step-by-step Reasoning Template. To encourage the model to examine each example in the batch B closely and reflect on the limitations in the current prompt, we guide the prompt proposal model Mproposal with a list of questions. For example: Is the prompt correctly describing the task? Is it necessary to edit the prompt? If yes, provide actionable suggestions on prompt editing.
3
Other Meta-prompt Components We Tried. Inspired by optimization concepts such as batch size, step size and momentum, we considered adding their verbalized counterparts to the metaprompt and investigate their effects. We also considered adding a prompt engineering tutorial in the meta-prompt to help the LLM better understand the task. Our observations on these components are mixed. We report these results in Appendix B.
4 Experiment Setting
4.1 Tasks
We use the following five groups of tasks to evaluate the effectiveness of PE2. More details (e.g., dataset sizes, license) are deferred in Appendix D.3.
(1) Math Reasoning. We use MultiArith (Roy and Roth, 2015) and GSM8K (Cobbe et al., 2021), which contain grade school math problems that requires multiple steps of arithmetic operations.
(2) Instruction Induction. Instruction Induction (Honovich et al., 2023) is a benchmark for inferring the underlying instruction from few-shot examples. We use 14 selected tasks that cover a wide range of use cases, e.g., “Formality” is a task that aims at rephrasing a sentence to be more formal.
(3) BIG-bench Hard. BIG-bench Hard (Suzgun et al., 2023) is a collection of 23 tasks (27 subtasks) that are challenging to LLMs but the performance may be improved with advanced prompting techniques (Wei et al., 2022b). Some of BIG-bench Hard tasks are closely related to real-world applications (e.g., movie recommendation).
(4) Counterfactual Evaluation. We use the arithmetic, chess, and syntax tasks and their counterfactual variants introduced in Wu et al. (2023). For arithmetic, the original task is base-10 addition, and the counterfactual tasks are base-8/9/11/16 addition. We use this set of tasks to observe whether PE2 can reason about counterfactual situations and communicate them to the task model.
(5) Production Prompt. Lastly, we optimize an internal production prompt for a hierarchical, multilabel classification task. The task is to classify a user query into domains, intents and slots, and then output a nested dictionary as the result. The initialization prompt is carefully designed by experienced engineers and has more than 5k tokens.
4.2 Compared Methods We compare PE2 with the following automatic prompt engineering methods. (a) APE (Zhou et al., 2023b): The base version of APE is an
4
initialization-only method and does not involve new prompt proposal steps. It uses an initialization prompt pinit to generate multiple prompt candidates from a few examples, and select the best one among them based on Ddev performance. (b) Iterative APE (Zhou et al., 2023b): After initialization, pmeta instructs the model to produce a paraphrase of p(t) and use it as p(t+1). (c) APO (Pryzant et al., 2023): pmeta contains short instructions on inspecting the batch B, generating textual “gradients” (feedback), and producing a new prompt p(t+1). We include the pinit and pmeta used in these baseline methods in Appendix E.
4.3 Expeirment Details
LLMs. By default, we use gpt-4 (OpenAI, 2023) as prompt proposal model Mproposal and use text-davinci-003 (Ouyang et al., 2022) as the task model Mtask performing the underlying task. Experiments on BIG-bench Hard are conducted at a later stage, and we use gpt-4-turbo and gpt-3.5-turbo-instruct to save costs and demonstrate the compatibility of our methods. For fair comparison, we use the same set of LLMs when running all compared methods and PE2.
Prompt Initialization. For Math Reasoning and BIG-bench Hard, we use “Let’s think step by step.” (Kojima et al., 2022) as the initialization prompt, which can elicit multi-step reasoning in LLMs to perform these tasks. For Instruction Induction, we follow the setting in prior works and use induction initialization. For Counterfactual Eval, we experiment with both. For the production task, we use the prompt written by an experienced engineer.
Search Budget. To ensure fair comparison, we use the same search budget for all prompt optimization methods. For experiments using induction initialization, 30 prompts are generated by pinit and form the initial candidate set P (0). The number of optimization steps T is set to be 3. At each timestamp, we select n = 4 best-performing prompts, and propose m = 4 prompts from each of them.
5 Results and Analysis
5.1 Main Results
Improved baselines with more recent LLMs. In Zero-shot CoT (Kojima et al., 2022) and APE (Zhou et al., 2023b), the results were obtained with a earlier text-davinci-002 model. We first rerun the prompts in these two works with
text-davinci-003, an upgraded model. In Table 1, we observe a significant performance boost by using text-davinci-003, suggesting that it is more capable of solving math reasoning problems with Zero-shot CoT. Moreover, the gaps between the two prompts are narrowed (MultiArith: 3.3% → 1.0%, GSM8K: 2.3% → 0.6%), indicating text-davinci-003 has a reduced sensitivity to prompt paraphrasing. Given this, methods that rely on simple paraphrasing, such as Iterative APE, may not enhance the final outcome as effectively. More precise and targeted edits are necessary to improve the performance.
PE2 outperforms Iterative APE and APO on various tasks. PE2 is able to find a prompt that achieves 92.3% accuracy on MultiArith (+6.3% compared to Zero-shot CoT) and 64.0% on GSM8K (+3.1%). Additionally, we demonstrate the wide applicablity of PE2 on a wide range of language tasks. In Fig. 2 we summarize the results and show that PE2 outperforms Iterative APE (Zhou et al., 2023b) and APO (Pryzant et al., 2023) in multiple cases. Most notably, when induction initialization is used, PE2 outperforms APO on 11 out of 12 counterfactual tasks (Fig. 11), demonstrating that PE2 is capable of reasoning about contradictions and counterfactual situations. We defer experiment details and performance breakdown for these benchmarks in Appendix F.
PE2 generates targeted prompt edits and highquality prompts. In Fig. 4 we plot the quality of prompt proposals over the course of prompt optimization. We observe very distinct patterns for the three prompt optimization methods: Iterative APE is based on paraphrasing, so the newly generated prompts have smaller variance. APO makes drasti-
5
cally large prompt edits and thus the performance drops in the first step. PE2 is the most stable one among the three methods. In Table 2, we list the optimal prompts found by these methods. Both APO and PE2 are able to provide instructions on “considering all parts / details”. In addition, PE2 is designed to inspect the batch closely, enabling it make very specific prompt edits such as “remember to add or subtract as needed”.
5.2 Ablation Study To demonstrate the effectiveness of the three metaprompt components introduced in PE2, we run ablation experiments by removing these components during prompt optimization on MultiArith and GSM8K. In these experiments, we make sure that the meta-prompt still contains sufficient information about the task of prompt engineering after the removal. From the results in Table 3, we observe that these three components contribute significantly to prompt engineering quality. As shown in Fig. 5, the exclusion of any one of these components results in a higher variance in the quality distribution of newly-proposed prompts. Moreover, without these components, the proposal model more frequently suggests low-quality prompts.
We also conduct an ablation study on back-
tracking (i.e., at timestamp t, select top-performing prompts from ∪ti=0P (i) versus only P (t)) and hard negative sampling (i.e., the batch B is sampled from the model’s errors, versus the batch is randomly sampled from Dtrain). We conclude that these design choices are important to PE2.
5.3 Case Study
PE2 amends erroneous or incomplete instructions, and devises multi-step plans for complex tasks. In Table 4 and Table 14, we present notable prompt edits made by PE2. In the task of rhymes (finding a word that rhymes with the input word), induction initialization mistakenly suggests the task is about changing the first letter of a word. PE2 successfully correct this after one optimization step. On the task of movie recommendation (Table 4), PE2 is able to decompose the complex task into concrete criteria, such as genre, plot and actor, when determining movie similarities. On the task of date understanding (Table 14), PE2 identifies the key step of referencing information about “today”. We believe these behaviors on summarizing key steps from failure examples and incorporating them into prompts demonstrate the intelligence and capabilities of PE2.
Limitations on following the meta-prompt and hallucination. Despite the successes made by
6
Task t Prompt Dev Acc.
Correct wrong or incomplete task instructions
Rhymes 0 Remove the first letter from each input word and then replace that first letter with a similar sounding letter or group of letters to form a new word. 0.35
1 Generate a word that rhymes with the input word. 0.45
Lay out tailored multi-step plans for complex problems
Movie Recommendation
0 Let’s think step by step. 0.58 1 Consider the genre, plot, and style of the input movies. Using this information, think step by
step to identify which of the following options is most similar to the given movies. 0.74
2 Considering factors such as genre, director, actors, release period, audience target, animation style, and humor, analyze the similarities among the given movies and identify the movie from the options that shares the most similarities.
0.82
PE2, we note several factors that’s limiting its performance when applied to challenging counterfactual tasks. We provide representative cases in Table 5. Occasionally, PE2 insists that “the prompt is correct, but the label is incorrect” and refuses to propose a new prompt. We also attempt to guide PE2 with hints (e.g., “the calculation may be performed in a different numerical base.”). Regrettably, this sometimes guides the model to generate incorrect solutions (e.g., base-80), and even create rationales to verify this imagined solution. These oberservations highlight the critical need to improve instruction following abilities and address hallucination issues in LLMs.
Discussion on “shortcut learning.” We find interesting yet concerning prompt edits on the counterfactual task of base-8 addition. When induction initialization is used (i.e., PE2 is uninformed with the information of base-8 and must infer it from the examples), PE2 is able to devise its own arithmetic rules that is partially correct (“... if either number is 50 or greater, add 22 to the sum”; see Table 4). On the positive side, it demonstrates PE2’s ability to engage in sophisticated counterfactual reasoning. However, it is concerning that models prompted with these self-induced shortcut solutions achieve a test accuracy of 37% (average over 5 runs), which outperform models explicitly prompted to perform base-8 addition (test acc: 17%/28% before/after PE2 optimization). Shortcut learning (Geirhos et al., 2020) has been studied extensively for gradient-based optimization. Our experiments suggest that similar failure modes may be present in textual prompt optimization as well.
5.4 Additional Analysis
Due to space limit, we summarize our other findings below and defer the details to Appendix A.
Effect of Initialization. (§A.1) (1) PE2 is able to recover from misleading or irrelevant prompt initializations, however the final prompt after optimization is worse than when using an instructive initialization. (2) We experiment with induction initialization. In this case, PE2 is able to discover a high quality prompt from scratch that matches with “Let’s think step by step” on MultiArith.
Effect of Task Format/Difficulty. (§A.2) We experiment with using a generative format (i.e., generating the answer string) and a multi-choice format (i.e., selecting from given choices A/B/C/D) on the Date Understanding task in BIG-bench Hard. We observe that automatic prompt engineering methods has limited effect on the multi-choice format, but bring significant gains on the generative format.
Do optimized prompts generalize to other LLMs? (§A.3) We evaluate prompts optimized for text-davinci-003 on other models such as mpt-7b-instruct and mistral-7b-instruct. Our results do not exhibit consistent generalization trends. This suggests that, at least in our current experiment setting, automatic prompt engineering methods tend to find model-specific prompts.
6 Related Work
Automatic Prompt Engineering. To alleviate the intensive trial-and-error efforts in manual prompt engineering, the research community has
7
developed various strategies to automate this process with techniques such as incremental editing (Prasad et al., 2023), reinforcement learning (Deng et al., 2022; Zhang et al., 2022), algorithmic search (Xu et al., 2022), generating in-context demonstrations adaptively (Wan et al., 2023a,b), among others. A line of work focus on meta-prompting LLMs themselves for automatic prompt engineering (Honovich et al., 2023; Zhou et al., 2023b; Pryzant et al., 2023). In our work, we discuss potential limitations in these methods and subsequently introduce new meta-prompt components for improvement.
Prompting LLMs for Complex Reasoning Tasks. Recent research works suggest that LLMs can perform complex reasoning tasks, e.g., grade-school math problems (Cobbe et al., 2021). There are two major techniques to boost LLMs’ performance on this: (1) prompting methods that guide the model to produce intermediate reasoning steps, either with few-shot demonstrations (Nye et al., 2021; Wei et al., 2022a; Yao et al., 2023) or with zero-shot prompts (Kojima et al., 2022); (2) self-reflection methods that progressively guide the model to inspect its current output and refine it (Chen et al., 2023; Madaan et al., 2023; Paul et al., 2023; Kim et al., 2023a). At its core, prompt engineering is a complex language generation task. Human prompt engineers usually examine the failure cases produced by the current prompt closely, make hypotheses, and compose a new prompt. In this work, we explore various prompting strategies when building an LLM-powered automatic prompt engineer.
Self-training and Self-improving for LLMs. Self-training refers to the technique of using a weak model to annotate input-label pairs and using them for further training (Rosenberg et al., 2005). In
the context of LLMs, STaR (Zelikman et al., 2022) and Self-Improve (Huang et al., 2023) show that employing LLMs to generate high-quality reasoning chains, followed by model fine-tuning on these chains, can significantly improve the model’s reasoning capabilities. In this work, we consider textual prompts as the “parameters” of LLMs, and we optimize these “parameters” with LLMs. This may be categorized as a case of self-improving (Goodman, 2023). More discussion in Appendix C.1.
7 Conclusion
In this paper, we introduced three meta-prompt components that lead to improved performance on automatic prompt engineering. The resulting method PE2 refines prompts written by human experts and surpasses established automatic prompt engineering baselines across various scenarios, notably on counterfactual tasks and production applications. Through comprehensive analysis and case studies, we illustrate PE2’s ability to make targeted prompt edits and generate high-quality prompts.
The challenge of prompt engineering a prompt engineer remains ongoing. As highlighted in our case study, we believe improving the LLM’s instruction following abilities and mitigating hallucination issues will be crucial for improving automatic prompt engineering. As the capabilities of LLM continue to evolve, their potential involvement in optimization or feedback loops necessitates a deeper empirical understanding of their failure modes, including shortcut learning discovered in this study. Looking ahead, we are also excited about applying PE2 to optimize its own meta-prompt in a self-referential way, in line with Metz et al. (2020); Irie et al. (2022); Fernando et al. (2023); Zelikman et al. (2023).
8
Limitations
Firstly, we opt for a relatively small prompt search budget (T = 3, m = 4, n = 4; see §4.3) due to cost considerations. In most of our experiments, the performance tends to plateau after T = 3 optimization steps. However, it’s important to consider that the use of the initialization prompt "let’s think step by step" in many cases might introduce a potential confounding factor. This prompt could be already near-optimal, leading to fast convergence during prompt optimization. Given the stochastic nature of natural language generation sampling and prompt optimization dynamics, it is possible that a larger prompt search budget or different experimental settings could yield new insights and conclusions.
Secondly, our study uses proprietary models such as gpt-4 and text-davinci-003. (1) It raises reproducibility concerns as proprietary models may undergo upgrades or discontinuation over time. However, we believe the core concepts introduced in this paper is model-agnostic. This is supported by our experiments where we use two different sets of (Mproposal, Mtask) (see §4.3). Consequently, we anticipate that PE2 could be adapted for use with open models once their capabilities catch on. (2) It also raises concerns on data contamination, as the tasks and prompts included in our study may or may not have been part of the model’s training data.
Lastly, apart from the three translation tasks in the Instruction Induction benchmark, our study predominantly focuses on tasks in English. We recognize the importance of inclusivity in language technology and acknowledge the need to extend our research to a multilingual setting in the future.
References
Howard Chen, Huihan Li, Danqi Chen, and Karthik Narasimhan. 2022. Controllable text generation with language constraints. arXiv preprint arXiv:2212.10466.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. 2022. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3369–3391, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673.
Noah Goodman. 2023. Meta-prompt: A simple selfimproving language agent.
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2023. Instruction induction: From few examples to natural language task descriptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1935–1952, Toronto, Canada. Association for Computational Linguistics.
Xinyu Hu, Pengfei Tang, Simiao Zuo, Zihan Wang, Bowen Song, Qiang Lou, Jian Jiao, and Denis X Charles. 2024. Evoke: Evoking critical thinking abilities in LLMs via reviewer-author prompt editing. In The Twelfth International Conference on Learning Representations.
Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051–1068, Singapore. Association for Computational Linguistics.
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. 2022. A modern self-referential weight matrix that learns to modify itself. In Proc. Int. Conf. on Machine Learning (ICML), Baltimore, MD, USA.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023a. Language models can solve computer tasks. arXiv preprint arXiv:2303.17491.
9
Seungone Kim, Se Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. 2023b. The CoT collection: Improving zero-shot and few-shot learning of language models via chainof-thought fine-tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12685–12708, Singapore. Association for Computational Linguistics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
Moxin Li, Wenjie Wang, Fuli Feng, Yixin Cao, Jizhi Zhang, and Tat-Seng Chua. 2023. Robust prompt optimization for large language models against distribution shifts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1539–1554, Singapore. Association for Computational Linguistics.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming fewshot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651.
Luke Metz, Niru Maheswaranathan, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. 2020. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint arXiv:2009.11243.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114.
OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023. GrIPS: Gradient-free, edit-based instruction search for prompting large language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3845–3864, Dubrovnik, Croatia. Association for Computational Linguistics.
Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7957–7968, Singapore. Association for Computational Linguistics.
Ning Qian. 1999. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1):145–151.
Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. 2005. Semi-supervised self-training of object detection models. 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION’05) - Volume 1, 1:29–36.
Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752, Lisbon, Portugal. Association for Computational Linguistics.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366.
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, and Xuezhe Ma. 2023. Evaluating large language models on controlled generation tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3155–3168, Singapore. Association for Computational Linguistics.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada. Association for Computational Linguistics.
10
Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. 2023a. Better zero-shot reasoning with self-adaptive prompting. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3493–3514, Toronto, Canada. Association for Computational Linguistics.
Xingchen Wan, Ruoxi Sun, Hootan Nakhost, Hanjun Dai, Julian Eisenschlos, Sercan Arik, and Tomas Pfister. 2023b. Universal self-adaptive prompting. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7437–7462, Singapore. Association for Computational Linguistics.
Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric Xing, and Zhiting Hu. 2024. Promptagent: Strategic planning with language models enables expert-level prompt optimization. In The Twelfth International Conference on Learning Representations.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022a. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022b. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2023. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. 2022. Gps: Genetic prompt search for efficient few-shot learning. arXiv preprint arXiv:2210.17041.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers. arXiv preprint arXiv:2309.03409.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. 2023. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint arXiv:2310.02304.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, volume 35, pages 15476–15488. Curran Associates, Inc.
Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. 2022. Tempera: Test-time prompting via reinforcement learning. arXiv preprint arXiv:2211.11890.
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning.
Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan. 2023a. Controlled text generation with natural language instructions. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 42602–42613. PMLR.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023b. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations.
11
A Additional Results and Analysis
A.1 Effect of Initialization
Previously, we use “Let’s think step by step.” as the initialization for math reasoning tasks. We further experiment with using a misleading prompt, an irrelevant prompt and induction initialization (inducing from a few examples). The results are presented in Table 6 and the optimization dynamics are visualized in Fig. 6.
In general, performance drops when alternative initialization methods are used, which highlights the importance of high-quality initialization. Still, PE2 is able to override the irrelevant or misleading prompts and gradually improve the performance (Fig. 6). Remarkably, PE2 is able to discover a high quality prompt2 by itself using induction initialization (84% on MultiArith-Dev) that almost matches with “Let’s think step by step” (85%) designed by highly-experienced human prompt engineers. This demonstrates the impressive prompt engineering capability of PE2 and suggests its potential for finding even better prompts when given additional computational resources.
2MultiArith prompt found by PE2 using induction initialization: “Analyze the problem and perform the calculations. Consider addition, subtraction, division, multiplication and perform them in the order they appear. If required, round up results to the nearest whole number. Subtract done tasks from total when necessary.”
A.2 Effect of Task Format For Date Understanding from BIG-bench Hard, we experiment with both a generative format (i.e., generating the answer string; used in Gao et al. (2023)) and a discriminative/multi-choice format (i.e., selecting from given choices A/B/C/D; used in Suzgun et al. (2023)). For Movie Recommendation, we experiment with two different multi-choice formats. See Table 7 for the formats that we used.
We report the results in Fig. 7. For Date Understanding, the multi-choice format narrows the output space and thus lower the difficulty of the task. We hypothesize that in combination with Zero-shot CoT, the task performance is close to saturation and automatic prompt engineering method does not provide extra benefit. The task is more challenging in the generative format and the detailed instructions in the optimized prompt bring significant performance gains.
For Movie Recommendation, we found that prompt optimization methods bring significant performance gains in both cases. The optimized prompts contain multi-step plans (e.g., consider
12
genre, director, ...), which boost the task performance. Minor formatting decisions such as outputting the option letter or the option string can still mildly affect the final accuracy.
Overall, the question on “when is automatic prompt optimization most effective” is dependent on many factors, including task format, task difficulty, output space size, task model capability, etc.
A.3 Do optimized prompts generalize to other LLMs?
We evaluate 5 GSM8K prompts for our prompt generalization study (see Table 8). Note that the APO and PE2 prompts are optimized for text-davinci-003. The two prompts reported in OPRO (Yang et al., 2023) are optimized for PaLM 2-L.
We investigate the generalization of optimized prompts by evaluating them on four models: gpt-3.5-turbo-instruct, mistral-7b-instruct-v0.2, yi-6b, and mpt-7b-instruct. We report the results in Fig. 8.
Our results do not exhibit consistent generalization trends. Optimized prompts generally outperforms the original zeroshot CoT prompt on text-davinci-003, mistral-7b-instruct-v0.2, yi-6b. However, with gpt-3.5-turbo-instruct, the original CoT prompt outperforms all optimized prompts. Our hypothesis is that “Let’s think step by
step.” is included in public instruction tuning collections (Kim et al., 2023b) and may be used during the training of gpt-3.5-turbo-instruct. As the instruction tuning mixture used for gpt-3.5-turbo-instruct are not disclosed, we cannot further investigate this.
Overall, our results suggest that current automatic prompt optimization methods tend to find model-specific prompts that do not reliably generalize to alternative models. This conclusion contrasts with the findings in PromptAgent (Wang et al., 2024), which we attribute to discrepancies in experimental setup. To maintain consistency with prior research (Zhou et al., 2023b), we limit the prompt length to be 50 or 200 tokens.
Future work may develop robust prompt optimization methods that operate across multiple task models, in a way similar to Li et al. (2023) which operates across domains. This may help identify high-quality prompts invariant to the underlying task model, so that when new and more powerful models (e.g., GPT-5) are released, the optimized prompt may be used directly.
B Other Meta-Prompt Components We Tried
In addition to the meta-prompt components studies in the main paper, we also tried other components. As the results are mixed and inconclusive on these components, we report them here in the appendix. We illustrate these components in Fig. 9.
Providing Detailed Instructions and Context.
(d) Prompt Engineering Tutorial. To help the LLM better understand the task of prompt engineering, we provide an online tutorial of prompt engineering in the meta-prompt.3
Incorporating Common Optimizer Concepts. The prompt engineering problem described in Eq. 1 is essentially an optimization problem, and the prompt proposal in Eq. 2 can be considered as doing one optimization step. Thus, we consider the following concepts commonly used in gradientbased optimization and develop their verbalized counterparts to be used in our meta-prompt.
(e) Batch Size. Batch size is the number of (failure) examples that is used in each prompt proposal step (Eq. 2). By default PE2 uses a batch
3https://www.promptingguide.ai/introduction. Published under MIT license.
13
14
size of 2. We experiment with batch sizes of {1, 4, 8} additionally in this section.
(f) Step Size. In gradient-based optimization, the step size determines the extent to which the model’s weights are updated. In prompt engineering, the counterpart would be the number of words (tokens) that can be modified. We directly specify that “You are allowed to change up to s words in the original prompt” in the meta-prompt, where s ∈ {5, 10, 15,None}.4
(g) Optimization History and Momentum. Momentum (Qian, 1999) is a technique to accelerate optimization and avoid oscillations by maintaining the moving average of past gradients. To develop the verbalized counterpart of momentum, we include all past prompts (at timestamp 0, 1, ..., t − 1), their performance on the dev set, and a summary of prompt edits.
Results and disucssion. We do not observe significant improvement by incorporating prompt engineering tutorial. As the tutorial is excessively long (2500+ tokens) and slows down the runtime, we do not include it in the final version of PE2. The optimizer-inspired concepts can improve the performance occasionally, but the current experiments do not give a consistent conclusion regarding their utilities.
Similar to the challenges encountered in gradient-based optimization, the process of hyperparameter selection is inherently noisy and often varies depending on the task at hand. For discrete prompt optimization, this complexity is further compounded by factors such as the task model’s sensitivity to prompts and the proposal model’s capability to follow instructions in the meta-prompt. For example, Sun et al. (2023) point out that LLMs
4Chen et al. (2022) and Zhou et al. (2023a) showed that LLMs could follow text generation constraints specified in natural language.
struggled at meeting fine-grained constraints such as “generate exactly 5 words,” which could potentially diminish the effectiveness of the (f) step size component. Additionally, (g) momentum requires multiple optimization steps to accumulate, yet our experiments are restricted to three steps due to cost constraints.
The potential of these meta-prompt components warrants re-examination in the future, particularly as models become more capable, and the efficiency and scalability of automatic prompt engineering methods improves.
C Additional Discussion
C.1 Recent Works
A recent work (Yang et al., 2023) introduced the concept of large language models as optimizers and proposed optimization by prompting (OPRO). In the following, we discuss the differences and connections between OPRO and our work. (1) Focus of the work. Both OPRO and our work conduct experiments on prompt optimization; the focus of the two works differ. OPRO can be applied to general optimization problems, including linear regression and traveling salesman problem. In our work we limit the scope to prompt optimization, with a specific focus on proposing and investigating different components in the meta-prompt. (2) Optimization strategy. The optimization strategies of the two works are different. PE2 is largely inspired by the concepts in APO (Pryzant et al., 2023), instructing the model to produce textual feedback (“gradient”) explicitly. It is more analogous to gradient descent. OPRO uses the execution accuracy as rewards to guide the optimization indirectly, which, in our understanding, is more analogous to in-context RL methods (Shinn et al., 2023). For future work, it would be interesting to compare the effectiveness and efficiency of both methods in a controlled setup. (3) Challenges in making direct comparison. Yang et al. (2023) mainly uses PaLM 2-L model and text-bison model as the task model (scorer), and optimizes the prompt for up to 200 steps. In our work, we mainly use text-davinci-003 and GPT-4, and optimize the prompt for 3 steps by default. Due to access and budget constraints, we are unable to make direct comparison with OPRO.
In addition to OPRO, several recent works have explored automatic prompt engineering using diverse strategies. PromptBreeder (Fernando et al.,
15
Algorithm 1 Search Procedure 1: P (0) = Pinit or P (0) = Minit(x1, y1, ..., xn, yn; pinit) ▷ Manual init. or induction init. 2: for t = 0, ..., T − 1 do 3: P (t+1) = ∅ 4: for p(t) ∈ Select-Best(∪ti=0P (i), n) do ▷ Select best n prompts based on Ddev 5: for j = 1...m do 6: B = Sample(Dtrain) ▷ Sample a batch 7: p(t+1) = Moptim(p(t), B; pmeta) ▷ New prompt proposal 8: P (t+1) = P (t+1) ∪ {p(t+1)} 9: end for 10: end for 11: end for 12: return Select-Best(∪Ti=0P (i), 1) ▷ Return the final best prompt based on Ddev
2023) adopts a self-referential prompt evolution framework, employing mutation prompts (similar to the concept of “meta-prompts” discussed in this paper) to edit task prompts, and hyper-mutation prompts to edit mutation prompts. PromptAgent (Wang et al., 2024) adopts the Monte Carlo Tree Search algorithm that iteratively performs selection, expansion, simulation and back-propagation for strategic prompt editing. Evoke (Hu et al., 2024) introduces a collaborative approach where an LLMReviewer and an LLM-author work together to refine the prompt using critical thinking. In parallel to these works, we focus on the design and evaluation of the meta-prompt in LLM-powered automatic prompt engineering in this paper.
C.2 Discussion on using PE2 to optimize its own meta-prompt
Conceptually, we can replace p(t) and p(t+1) in Eq. 2 with the meta-prompt pmeta directly to enable PE2 to optimize the meta-prompt. However, three challenges (and broader questions) arise if we pursue this method, and we look forward to addressing these challenges in the future: 1. How to collect data for such a study? To ensure
this meta-prompt is general we may need a large collection of tasks along with prompt optimization history associated with them. Creating a resource like this will be a large effort. 2. How to automatically optimize the meta-prompt when there are no ground truth labels for prompt engineering? Math problems have ground-truth answers so that PE2 can reason on them and provide feedback on prompt refinement. The task of prompt engineering does not have ground truth labels, and this potentially makes the metaprompt optimization process more noisy. 3. It would be very costly to run and even evaluate
a system like this. To evaluate one meta-prompt candidate, we will need to use it for prompt optimization on various tasks. We would expect the optimization process of the meta-prompt to be a magnitude more costly.
D Additional Experiment Details
D.1 Prompt Search Algorithm See Algorithm 1.
D.2 Controlling Prompt Length By default the max length of prompts is set to be 50 tokens, following Zhou et al. (2023b). For counterfactual tasks, to allow more space to explain the counterfactual situations, the max length is set to be 200 tokens.
D.3 Tasks and Data We summarize the dataset size and data split information in Table 10. We summarize the source and license information of the datasets in Table 11. To the best of our knowledge, our usage of these datasets are consistent with their intended use; the data we use do not contain personal or sensitive information. Most of the datasets are in English and not domain-specific.
(1) Mathematical Reasoning. The MultiArith dataset (Roy and Roth, 2015) contains 600 examples. As our prompt optimization method requires a training set, we randomly split into 100/100/400 for train/dev/test. This creates a slight discrepancy when comparing the results with past reported results. We ensure our reproduction is fair across different methods by using this fixed split. The GSM8K dataset (Cobbe et al., 2021) has a provided test split (1319 examples). We randomly selected 200 examples for the original train split, and use 100 as Dtrain and 100 as Ddev.
16
(2) Instruction Induction. We closely follow the settings in Zhou et al. (2023b). For each subtask, we randomly sample 5 different Dtrain/Ddev/Dtest of size 100/20/100. We list the sub-tasks in Instruction Induction benchmark in Table 12. We removed 8 tasks (active to passive, diff, first word letter, letters list, num to verbal, singular to plural, sum), because our baseline method APE (Zhou et al., 2023b) already achieves near perfect accuracies (95%+) on these tasks. We also removed 2 tasks (cause and effect, common concept) because they have less than 50 examples in total, and it is challenging to create train/dev/test split from these examples.
(3) BIG-bench Hard Tasks. We mainly experiment with the BBH task format used in Suzgun et al. (2023). As the public BBH repository have 250 examples per task, we randomly split them into 100/100/50 for Dtrain/Ddev/Dtest. For Date Understanding and Movie Recommendation, we consider using alternative tasks formats to study the their effect (see §A.2). We obtain the data from the original BIG-bench repository which contains more examples per task. Hence we randomly sample 100/100/500 examples for Dtrain/Ddev/Dtest in these two experiments.
(4) Counterfactual Evaluation. We use three subtasks in this evaluation suite: arithmetic, chess and syntax. For each subtask, we randomly sample 5 different Dtrain/Ddev/Dtest of size 100/20/100.
(5) Production Prompt. We use a randomly sampled sub-set of human annotated queries and labels (> 150), which are derived from user reported er-
rors. The data is divided between training (50%), validation (25%) and testing (25%). We use the F1-score for evaluating model outputs and report the absolute change in score with the initialization prompt.
D.4 Infrastructure and Runtime
Infrastructure. We use OpenAI API5 to access gpt-4,gpt-4-turbo, gpt-3.5-turbo-instruct, text-davinici-003. For prompt generalization experiments using mistral-7b-instruct, mpt-7b-instruct and yi-6b, we run experiments locally using one Nvidia RTX A6000 GPU and the vLLM toolkit (Kwon et al., 2023).
Runtime. One prompt optimization experiment using gpt-4/text-davinici-003 as task/proposal model takes about 90 minutes. This is also subject to API rate limits.
Costs. When using gpt-4/text-davinci-003 it costs about $25 USD for one prompt optimization experiment. In the later stage of this project, we use gpt-4-turbo/gpt-3.5-turbo-instruct which are newer and cheaper, and the cost is reduced to about $3 USD per experiment.
E Meta-prompts
We implement the meta-prompts using the guidance toolkit 6, which enables multi-round conversations and supports basic handlebars-style syntax to control the workflow.
5https://openai.com/blog/openai-api 6https://github.com/guidance-ai/guidance
17
Task Instruction Demonstration
Subtasks used in this work (14)
Second Letter Extract the first letter of the input word. cat → a Starting With Extract the words starting with a given letter from
the input sentence. The man whose car I hit last week sued me. [m] → man, me
Negation Negate the input sentence. Time is finite → Time is not finite. Antonyms Write a word that means the opposite of the input
word. won → lost
Synonyms Write a word with a similar meaning to the input word.
alleged → supposed
Membership Write all the animals that appear in the given list. cat, helicopter, cook, whale, frog, lion → frog, cat, lion, whale Rhymes Write a word that rhymes with the input word. sing → ring Informal to Formal Rephrase the sentence in formal language. Please call once you get there → Please call upon your arrival. Translation EN-DE Translate the word into German. game → spiel Translation EN-ES Translate the word into Spanish. game → jeugo Translation EN-FR Translate the word into French. game → jeu Sentiment Determine whether a movie review is positive or
negative. The film is small in scope, yet perfectly formed. → positive
Sentence Similarity Rate the semantic similarity of two sentences on a scale of 0 to 5 Sentence 1: A man is smoking. Sentence 2: A man is skating. → 0 - definitely not Word in Context Determine whether an input word has the same meaning in the two sentences. Sentence 1: Approach a task. Sentence 2: To approach the city. Word: approach → not the same
Subtasks removed due to near-perfect accuracy (95%+) with baseline method (8)
First Letter Extract the first letter of the input word. cat → c List Letters Break the input word into letters, separated by spaces. cat → c a t Singular to Plural Convert the input word to its plural form. cat → cats Active to Passive Write the input sentence in passive form. The artist introduced the scientist. → The scientist was introduced by the artist. Larger Animal Write the larger of the two given animals. koala, snail → koala Sum Sum the two given numbers. 22 10 → 32 Diff Subtract the second number from the first. 32 22 → 10 Number to Word Write the number in English words. 26 → twenty-six Subtask removed due to small dataset size (2)
Cause and Effect Find which of the two given cause and effect sentences is the cause. Sentence 1: The soda went flat. Sentence 2: The bottle was left open. → The bottle was left open. Common Concept Find a common characteristic for the given objects. guitars, pendulums, neutrinos → involve oscillations.
The initialization prompt is originally from APE (Zhou et al., 2023b). In this paper, it is shared by all methods (Iterative APE, APO and PE2).
1 {{#system~}} 2 You are a helpful assistant. 3 {{~/ system}} 4 5 {{#user~}} 6 I gave a friend an instruction and {{n_demo}}
inputs. The friend read the instruction and wrote an output for every one of the
inputs. 7 Here are the input -output pairs: 8 9 {{demos}}
10 11 What was the instruction? It has to be less than {{max_tokens}} tokens. 12 {{~/user}} 13 14 {{#assistant~}} 15 The instruction was {{gen 'instruction ' [[ GENERATION_CONFIG]]}} 16 {{~/ assistant}}
1 {{#system~}} 2 You are a helpful assistant. 3 {{~/ system}} 4 5 {{#user~}} 6 Generate a variation of the following
instruction while keeping the semantic meaning.
7 8 {{prompt}} 9
10 The new instruction has to be less than {{ max_tokens}} words. 11 Reply with the new instruction. Do not include other text. 12 {{~/user}} 13 14 {{#assistant~}} 15 {{gen 'new_prompt ' [[GENERATION_CONFIG]]}} 16 {{~/ assistant}}
E.3 APO
Part 1 - Generating “gradients” 1 {{#system~}}
18
Task Category Demonstration
Arithmetic - Two-digit addition
Base-10 Original 22+10 → 32 Base-8 Counterfactual 76+76 → 174 Base-9 Counterfactual 76+14 → 101 Base-11 Counterfactual 76+14 → 8A Base-16 Counterfactual EC+DD → 1C9 Chess - Legality of a 4-move opening
Normal Rules Original 1. g3 Ng6 2. b3 Kf8 * → illegal Swapping bishops and knights Counterfactual 1. g3 Ng6 2. b3 Kf8 * → legal Syntax - Identify the main subject and the main verb of a sentence
SVO Original he has good control . → he has SOV Counterfactual he good control has . → he has VSO Counterfactual has he good control . → he has VOS Counterfactual has good control he . → he has OVS Counterfactual good control has he . → he has OSV Counterfactual good control he has . → he has
26 Please provide me with the prompt engineering history, the current prompt, and the examples you have. 27 {{~/ assistant}} 28 29 {{#user~}} 30 ## Prompt 31 {{prompt}} 32 33 ## Full Template 34 This describes how the prompt of interested is concatenated with the input text. 35 The prompt may appear before the input text, or after the input text. 36 Optionally the full template may contain other template information. 37 ``` 38 {{full_prompt}} 39 ``` 40 41 ## Examples 42 {{examples}} 43 44 ## Instructions 45 For some of these examples, the output does not
match with the label. This may be due to the prompt being misleading or not describing the task precisely.
46 47 Please examine the example(s) carefully. Note
that the ground -truth labels are __absolutely correct__, but the prompts ( task descriptions) may be incorrect and need modification. For each example, provide reasoning according to the following template:
48 49 ### Example <id > 50 Input: <input > 51 Output: <output > 52 Label: <label > 53 Is the output correct compared to the label: < yes or no, and your reasoning > 54 Is the output correctly following the given prompt: <yes or no, and your reasoning > 55 Is the prompt correctly describing the task
shown by the input -label pair: <yes or no, and your reasoning >
56 To output the correct label, is it necessary to edit the prompt: <yes or no, and your reasoning > 57 If yes, provide detailed analysis and actionable suggestions to edit the prompt: <analysis and suggestions > 58 {{~/user}} 59 60 {{#assistant~}} 61 {{gen 'reasoning ' temperature=0}} 62 {{~/ assistant}} 63 64 {{#user~}} 65 Now please carefully review your reasoning in
Step 1 and help with Step 2: refining the prompt.
66 67 {{#if history}} 68 ## Prompt Refinement History from the Past 69 Note that higher accuracy means better. If some
edits are useful in the past, it may be a good idea to make edits along the same direction.
70 {{history}} 71 {{/if}} 72 73 ## Current Prompt 74 {{prompt}} 75 76 ## Instructions 77 {{#if step_size}} 78 * You are allowed to change up to {{step_size}} words in the original prompt. 79 {{/if}} 80 {{#if max_tokens}} 81 * The total length of the prompt should be less than {{max_tokens}} words. 82 {{/if}}
83 * Please help edit the prompt so that the updated prompt will not fail on these examples anymore. 84 * Reply with the prompt. Do not include other text. 85 {{~/user}} 86 87 {{#assistant~}} 88 {{gen 'new_prompt ' temperature=0.7 max_tokens=30 0}} 89 {{~/ assistant}} 90 91 {{#if history}} 92 {{#user~}} 93 Now please summarize what changes you 've made to
the prompt, in the following format. Make sure the summariy is concise and contains no more than 200 words.
94 95 " * At step {{timestamp}}, the prompt has
limitations such as <summary of limitations >. Changes to the prompt include <summary of changes >."
96 97 Reply with the summarization. Do not include other text. 98 {{~/user}} 99
100 {{#assistant~}} 101 {{gen 'new_history ' temperature=0.7 max_tokens=2 00}} 102 {{~/ assistant}} 103 {{/if}}
F Additional Result Figures and Tables
Notable Prompt Edits. Additional examples on notable prompt edits made by PE2 are in Table 14.
Results Breakdown. We report the results on each subtask in Instruction Induction in Fig. 10. For counterfactual tasks, results using induction initialization are in Fig. 11 and results using manual initialization are in Fig. 12. For BIG-bench Hard tasks, we report the results in Fig. 13. We report the results on Date Understanding and Movie Recommendation when alternative task formats are used in Table 19.
20
21
22
G Prompt Optimization Results
See Table 15-19.
23
Task Method Prompt
syntax_svo
APO Your task is to identify the subject and the main verb of the primary clause in an input sentence. Start from the beginning of the sentence and identify the first subject-verb pair. Ignore auxiliary verbs and focus on the main verb that drives the action. If the sentence has multiple clauses, focus on the first one that forms a complete thought. Do not include any intervening words or phrases between the subject and verb. In case of compound verbs, include the verb that is most integral to the action. Ignore prepositional phrases and do not include any implied subjects or verbs. Your output should be concise, containing only the subject and the main verb.
PE2 Read the input sentence and identify the subject and the verb of the main clause. Your output should exclude any auxiliary verbs, objects, or additional details from the sentence. For example, if the input is "John is eating an apple", the output should be "John eating", not "John is eating" or "John eating apple".
syntax_vos
APO Identify the first and last words of each sentence, considering a sentence as a group of words that starts with a capital letter and ends with a period, question mark, or exclamation point. Ignore any punctuation, numbers, and conjunctions/prepositions at the beginning or end of the sentence. Write these two words in reverse order. If the sentence begins and ends with the same word, write it once. Treat compound words or phrases as single words. For example, ’uniroyal’ and ’has’ should be treated as ’uniroyal has’.
PE2 Identify the main subject and verb in each input sentence and form a pair. The subject is usually a noun or pronoun that the verb refers to. The verb should be the main verb of the sentence, not an auxiliary verb. For example, if the input is "The cat chased the mouse.", the output should be "cat chased". If the input is "She has eaten the cake.", the output should be "She eaten", not "She has".
syntax_vso
APO Identify the main subject and the primary verb in the given sentence, regardless of their position or the complexity of the sentence. Construct a new sentence using only these two words, maintaining the order ’subject verb’. Ignore additional information, context, or implied subjects/verbs. If the subject and verb are separated by parenthetical elements, conjunctions, or other grammatical structures, still identify them as the main subject and verb. Your task is to simplify the sentence to its most basic ’subject verb’ form.
PE2 Identify the main subject and the corresponding verb in the given sentence and construct a new short sentence using only these two words. The order should be ’subject verb’. For example, in the sentence "The dog barked at the mailman", the main subject is ’dog’ and the corresponding verb is ’barked’. So, the new sentence would be "Dog barked".
Task Method Prompt
PE2 You are a mathematician. Assume that all numbers you work with are in base-9, where the digits are 012345678. Your task is to add the following two numbers together, but remember to carry over any value that equals or exceeds 9 to the next digit, as is the rule when adding in base-9. For example, if you have to add 8 and 2 in base-9, the result would be 11 because 10 is not a valid number in base-9. Now, compute the sum of the following two numbers.
chess_cf
Manual Init. You are a chess player. You are playing a chess variant where the starting positions for knights and bishops are swapped. For each color, the knights are at placed that where bishops used to be and the bishops are now placed at where knights used to be. Given an opening, determine whether the opening is legal. The opening doesn’t need to be a good opening. Answer "legal" if all moves are legal. Answer "illegal" if the opening violates any rules of chess.
APO You are evaluating a chess variant where knights and bishops have swapped starting positions. Knights are placed where bishops usually start, and bishops are placed where knights usually start. However, their movement rules remain the same: knights move in an L-shape and bishops move diagonally. Your task is to determine the legality of a given opening. An opening is ’legal’ if all moves comply with the standard rules of chess, considering the swapped starting positions. If all moves are legal, answer ’legal’. If any move violates the chess rules, answer ’illegal’. The opening doesn’t need to be a good strategy, it just needs to be legal.
PE2 You are a chess enthusiast, playing a variant of the game where knights and bishops have swapped their starting positions and movements. The knights, now placed where the bishops were, move as bishops. The bishops, positioned where knights were, move as knights. Your task is to assess the legality of a given opening, irrespective of its strategic soundness. Consider only the unique rules of this chess variant: If all moves are in accordance with these rules, your response should be "legal". However, if any move contravenes these rules, respond with "illegal". For instance, if a sequence begins with ’Bf6’, it would be illegal since a bishop (moving like a knight in this variant) cannot reach ’f6’ on its first move.
chess_original
Manual Init. You are a chess player. Given an opening, determine whether the opening is legal. The opening doesn’t need to be a good opening. Answer "legal" if all moves are legal. Answer "illegal" if the opening violates any rules of chess.
APO You are a chess expert. Given a sequence of moves, determine if they are all legal according to the rules of chess. Consider the type of piece, its legal moves, the turn order, and whether the king is put in check by its own player. If all moves are legal, answer "legal". If any move violates the rules of chess, answer "illegal". Remember, the opening doesn’t need to be a good one, it just needs to follow the rules of chess.
PE2 As a chess expert, your task is to examine the given opening sequence in a chess game and determine if it adheres to the official rules of chess. Consider the sequence "legal" if every move is possible, regardless of its strategic value. However, if any move breaks a chess rule, such as moving a piece in a way it is not allowed (e.g., a knight moving like a bishop), classify the sequence as "illegal". Your response should be one of two words: "legal" or "illegal".
syntax_osv
Manual Init. You are an expert in linguistics. Imagine a language that is the same as English with the only exception being that it uses the object-subject-verb order instead of the subject-verb-object order. Your task is to identify the main verb and the main subject in a sentence in this imaginary language. Show the main verb (a single word) and its subject (also a single word).
APO You are a linguistics expert. Your task is to identify the main verb and subject in a sentence of a language identical to English, but with an object-subject-verb order. The main verb is the primary action word, excluding auxiliary verbs. The main subject is the primary entity performing the action. In complex sentences, focus on the main clause. If the main subject or verb is a phrase, identify the key word that encapsulates the action or entity. If the main subject or verb is a proper noun, treat it as a single word. Your output should be a phrase consisting of the main subject and verb. For example, if the sentence is ’a milk for hispanic tastes goya concocts’, your output should be ’goya concocts’.
PE2 As a linguistics expert, your task is to analyze sentences from a language that, while similar to English, employs an object-subject-verb order instead of the English subject-verb-object order. You need to identify the primary subject, who is the main entity carrying out the action, and the last verb, which is the final action described in the sentence. Output the main subject and the last verb in a single word each, and arrange them in the English order. For instance, for "apple the eats boy", your output should be "boy eats". Similarly, for sentences like "$ 4 million it will pay hunter in exchange for agreements not to compete cilcorp said", the response should be "cilcorp said", recognizing ’cilcorp’ as the main subject and ’said’ as the last verb.
syntax_ovs
Manual Init. You are an expert in linguistics. Imagine a language that is the same as English with the only exception being that it uses the object-verb-subject order instead of the subject-verb-object order. Your task is to identify the main verb and the main subject in a sentence in this imaginary language. Show the main verb (a single word) and its subject (also a single word).
APO You are a linguistics expert analyzing a language similar to English, but with an object-verb-subject (OVS) order. Your task is to identify the main verb and the main subject in a sentence. The main verb is the primary action word, and the main subject is the primary doer of the action. They may not always be adjacent. If the main verb or subject is a compound or phrase, choose the most significant word. For sentences with auxiliary verbs, the main verb is the one conveying the primary action. After identifying, reverse the order to subject-verb for your output. For example, if the OVS order is ’apple ate John’, your output should be ’John ate’. Remember, your output should always be in subject-verb order.
PE2 You are an expert in linguistics. Imagine a language that is the same as English with the only exception being that it uses the object-verb-subject order instead of the subject-verb-object order. Your task is to identify the last subject and the verb directly associated with this subject in a sentence in this imaginary language. Show the subject first (a single word) and then the verb (also a single word). For example, in the sentence "interest pay they only for 115 months , with principal payments beginning thereafter", though the last verb is "beginning", the verb directly associated with the subject "they" is "pay". Therefore, the answer is "they pay".
syntax_sov
Manual Init. You are an expert in linguistics. Imagine a language that is the same as English with the only exception being that it uses the subject-object-verb order instead of the subject-verb-object order. Your task is to identify the main verb and the main subject in a sentence in this imaginary language. Show the main verb (a single word) and its subject (also a single word).
Continued on next page
27
Task Method Prompt
BIG-bench Hard (Manual Initialization - “Let’s think step by step.”)
boolean_expressions
Iterative APE Consider approaching this methodically, one step at a time.
APO First, resolve ’not’ immediately before it. Then evaluate expressions within parentheses. Next, perform ’and’ operations, and finally ’or’. Determine the truth value step by step.
PE2 Evaluate the logical expression step by step following precedence: ’not’, ’and’, ’or’. Conclude by affirming the truth of the full statement.
causal_judgement Iterative APE Approach each stage sequentially. Continued on next page
28
Task Method Prompt
APO Identify the direct cause of the outcome: was it the immediate action or condition without which the event wouldn’t have occurred?
PE2 Determine if the action was intentional and a contributing factor to the outcome. Answer ’Yes’ if intentional and causative, ’No’ otherwise.
disambiguation_qa
Iterative APE Consider each phase one by one.
APO In each sentence, determine the pronoun’s antecedent based on typical roles and actions. If unclear, mark as ambiguous."
PE2 Let’s think step by step, using grammatical structure and context to identify the antecedent of the pronoun or determine if it is ambiguous.
date_understanding
Iterative APE Consider the process in a sequential manner.
APO Let’s think step by step.
PE2 Determine the date referred to as ’today’ from the context given, then calculate the specified date step by step in MM/DD/YYYY format.
dyck_languages
Iterative APE Let’s think step by step.
APO Output the sequence of closing brackets needed to properly close all open brackets in the given order: parentheses ’()’, square ’[]’, curly ’’, and angle ’<>’. Ignore all other characters.
PE2 Given an incomplete sequence of brackets, output only the missing closing brackets required to complete the sequence. Exclude any part of the original sequence.
formal_fallacies
Iterative APE Consider each phase one at a time.
APO Determine if the conclusion necessarily follows from the premises using deductive logic. Identify any logical fallacies or overlooked counterexamples. Is the argument valid or invalid?
PE2 To assess if an argument is deductively valid, examine if the conclusion logically follows from the premises without exception.
geometric_shapes
Iterative APE Consider our actions in a sequential manner.
APO Classify the shape formed by the SVG path’s ’d’ attribute, analyzing vertices, line segments, and closure to distinguish polygons and figures.
PE2 Analyze the SVG path commands to determine the shape they draw. Consider each command and visualize the path step by step.
hyperbaton
Iterative APE Approach this gradually, step by step.
APO Choose the sentence with adjectives in the correct order: opinion, size, age, shape, color, origin, material, purpose, noun."
PE2 Let’s think step by step, considering the standard order of adjectives in English: opinion, size, age, shape, color, origin, material, purpose.
logical_deduction _five_objects
Iterative APE Let’s proceed with a methodical, step-by-step approach.
APO Let’s think step by step.
PE2 Let’s think step by step.
logical_deduction _seven_objects
Iterative APE Consider each phase one at a time.
APO Arrange seven objects in order using the clues provided. Identify the correct position for each object from the options listed, ensuring logical consistency."
PE2 Let’s analyze the information provided step by step, ensuring each deduction follows logically from the statements given to arrive at the correct answer.
logical_deduction _three_objects
Iterative APE Let’s proceed with a methodical, stepwise approach.
APO Let’s think step by step.
PE2 Let’s carefully analyze the information to determine the accurate ranking of objects based on their attributes.
multistep_arithmetic _two
Iterative APE Approach this step-by-step, tackling each phase sequentially.
APO Let’s think step by step.
PE2 Let’s meticulously solve the math problem by simplifying each part of the equation, verifying our calculations at every step before moving forward.
movie _recommendation
Iterative APE Consider each phase methodically.
APO Select the movie that best matches the reference in terms of genre and cultural significance, ignoring minor details. Choose the closest match:"
PE2 Evaluate the given movies for genres, themes, narrative styles, tone, and characters to select the most similar one from the options.
navigate Iterative APE Let’s carefully assess each stage of our plan in sequence.
APO Calculate the net distance from the starting point after following these step-by-step instructions, considering ’forward’ as positive and ’backward’ as negative movement.
PE2 Let’s calculate each movement’s effect on our position. Start at zero and add or subtract steps as instructed, considering the direction each time to ensure accuracy.
Continued on next page
29
Task Method Prompt
object_counting
Iterative APE Let’s continue by taking systematic, sequential steps.
APO Let’s think step by step.
PE2 Let’s identify and count the instances of the specified category of items mentioned, tallying multiples, to determine their total quantity.
penguins_in_a_table
Iterative APE Consider the procedure progressively.
APO Let’s think step by step.
PE2 Let’s think step by step.
reasoning_about _colored_objects
Iterative APE Consider each phase methodically.
APO Let’s think step by step.
PE2 Let’s think step by step and pay close attention to details such as colors and quantities.
ruin_names
Iterative APE Let’s proceed with our tasks one by one.
APO Identify the funniest edit of the given name by selecting the option that best incorporates a pun or playful twist related to the original.
PE2 Identify the most humorous edit by considering only puns or clever wordplay that creates a witty variation of the original name, excluding misspellings or simple pluralizations.
salient_translation _error_detection
Iterative APE Let’s take this gradually, step by step.
APO Let’s think step by step.
PE2 Let’s analyze the source and translation for errors. Check if there are any changes or inaccuracies in Named Entities, Numerical Values, Modifiers, Negation, Facts, or missing details. Identify which type of error occurs.
snarks
Iterative APE Evaluate each stage sequentially.
APO Identify the sarcastic statement by considering the reversal of expectations and societal norms. Look for irony and implied meanings contrary to the literal words.
PE2 To identify which statement is sarcastic, consider that sarcasm often means saying the opposite of what’s true in a mocking way.
sports_understanding
Iterative APE Let’s think step by step.
APO Determine if the sentence is plausible: Match the athlete’s known sport, current activity status, and sport-specific terms to assess accuracy in context.
PE2 Let’s think step by step.
temporal_sequences
Iterative APE Consider each stage methodically.
APO Identify the time period when the person was not seen and the location was open. Exclude times when the person was observed elsewhere or the location was closed.
PE2 Analyze the timeline to pinpoint time slots when the individual was not seen, which indicate when events could have occurred.
tracking_shuffled _objects_five_objects
Iterative APE We’ll tackle this systematically, one stage at a time.
APO Track ball swaps and position changes separately. List each swap, update positions and ball ownership after each, and determine final states for both.
PE2 Let’s carefully track each player’s position swaps step by step to determine their final positions.
tracking_shuffled _objects_seven_objects
Iterative APE Approach each stage with systematic thought.
APO Let’s think step by step.
PE2 Let’s carefully track each book exchange step by step to determine the final owner of each book.
tracking_shuffled _objects_three_objects
Iterative APE Reflect on each stage individually.
APO Let’s think step by step.
PE2 Let’s analyze each position swap in sequence to determine the final positions. Confirm the last known positions of all players before concluding.
web_of_lies
Iterative APE Let’s proceed with a methodical, stepwise approach.
APO Given a sequence of people’s statements about others’ truthfulness, determine the truth status of the final person. Assume the first statement’s truth is known. Apply logical negation for each liar’s statement.
PE2 To determine who is truthful, invert the claim of a liar and trust a truthful person’s claim. Apply this logic until the last claim.
word_sorting
Iterative APE Progress through each stage sequentially.
APO Sort words alphabetically, ignoring case. Exclude ’List:’ label. For same first letters, sort remaining letters. Output in lowercase.
PE2 Let’s analyze and sort words. Ignore words that are part of instructions, like "List:", and then arrange the remaining words in alphabetical order.
30

heading: Future Work
text: We recognize the importance of inclusivity in language technology and acknowledge the need to extend our research to a multilingual setting in the future. References
Howard Chen, Huihan Li, Danqi Chen, and Karthik Narasimhan. 2022. Controllable text generation with language constraints. arXiv preprint arXiv:2212. 10466. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304. 05128. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168. Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. 2022. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3369–3391, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309. 16797. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673. Noah Goodman. 2023. Meta-prompt: A simple selfimproving language agent. Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2023. Instruction induction: From few examples to natural language task descriptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1935–1952, Toronto, Canada. Association for Computational Linguistics. Xinyu Hu, Pengfei Tang, Simiao Zuo, Zihan Wang, Bowen Song, Qiang Lou, Jian Jiao, and Denis X Charles. 2024. Evoke: Evoking critical thinking abilities in LLMs via reviewer-author prompt editing. In The Twelfth International Conference on Learning Representations. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051–1068, Singapore. Association for Computational Linguistics. Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. 2022. A modern self-referential weight matrix that learns to modify itself. In Proc. Int. Conf. on Machine Learning (ICML), Baltimore, MD, USA. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023a. Language models can solve computer tasks. arXiv preprint arXiv:2303. 17491. 9
Seungone Kim, Se Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. 2023b. The CoT collection: Improving zero-shot and few-shot learning of language models via chainof-thought fine-tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12685–12708, Singapore. Association for Computational Linguistics. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. Moxin Li, Wenjie Wang, Fuli Feng, Yixin Cao, Jizhi Zhang, and Tat-Seng Chua. 2023. Robust prompt optimization for large language models against distribution shifts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1539–1554, Singapore. Association for Computational Linguistics. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming fewshot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 17651. Luke Metz, Niru Maheswaranathan, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. 2020. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint arXiv:2009. 11243. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112. 00114. OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303. 08774. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304. 01904. Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023. GrIPS: Gradient-free, edit-based instruction search for prompting large language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3845–3864, Dubrovnik, Croatia. Association for Computational Linguistics. Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7957–7968, Singapore. Association for Computational Linguistics. Ning Qian. 1999. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1):145–151. Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7. Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. 2005. Semi-supervised self-training of object detection models. 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION’05) - Volume 1, 1:29–36. Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752, Lisbon, Portugal. Association for Computational Linguistics. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303. 11366. Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, and Xuezhe Ma. 2023. Evaluating large language models on controlled generation tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3155–3168, Singapore. Association for Computational Linguistics. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada. Association for Computational Linguistics. 10
Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. 2023a. Better zero-shot reasoning with self-adaptive prompting. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3493–3514, Toronto, Canada. Association for Computational Linguistics. Xingchen Wan, Ruoxi Sun, Hootan Nakhost, Hanjun Dai, Julian Eisenschlos, Sercan Arik, and Tomas Pfister. 2023b. Universal self-adaptive prompting. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7437–7462, Singapore. Association for Computational Linguistics. Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric Xing, and Zhiting Hu. 2024. Promptagent: Strategic planning with language models enables expert-level prompt optimization. In The Twelfth International Conference on Learning Representations. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022a. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022b. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2023. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307. 02477. Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. 2022. Gps: Genetic prompt search for efficient few-shot learning. arXiv preprint arXiv:2210. 17041. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers. arXiv preprint arXiv:2309. 03409. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations. Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. 2023. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint arXiv:2310. 02304. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, volume 35, pages 15476–15488. Curran Associates, Inc. Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. 2022. Tempera: Test-time prompting via reinforcement learning. arXiv preprint arXiv:2211. 11890. Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning. Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan. 2023a. Controlled text generation with natural language instructions. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 42602–42613. PMLR. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023b. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations. 11
A Additional Results and Analysis
A. 1 Effect of Initialization
Previously, we use “Let’s think step by step. ” as the initialization for math reasoning tasks. We further experiment with using a misleading prompt, an irrelevant prompt and induction initialization (inducing from a few examples). The results are presented in Table 6 and the optimization dynamics are visualized in Fig. 6. In general, performance drops when alternative initialization methods are used, which highlights the importance of high-quality initialization. Still, PE2 is able to override the irrelevant or misleading prompts and gradually improve the performance (Fig. 6). Remarkably, PE2 is able to discover a high quality prompt2 by itself using induction initialization (84% on MultiArith-Dev) that almost matches with “Let’s think step by step” (85%) designed by highly-experienced human prompt engineers. This demonstrates the impressive prompt engineering capability of PE2 and suggests its potential for finding even better prompts when given additional computational resources. 2MultiArith prompt found by PE2 using induction initialization: “Analyze the problem and perform the calculations. Consider addition, subtraction, division, multiplication and perform them in the order they appear. If required, round up results to the nearest whole number. Subtract done tasks from total when necessary. ”
A. 2 Effect of Task Format For Date Understanding from BIG-bench Hard, we experiment with both a generative format (i. e. , generating the answer string; used in Gao et al. (2023)) and a discriminative/multi-choice format (i. e. , selecting from given choices A/B/C/D; used in Suzgun et al. (2023)). For Movie Recommendation, we experiment with two different multi-choice formats. See Table 7 for the formats that we used. We report the results in Fig. 7. For Date Understanding, the multi-choice format narrows the output space and thus lower the difficulty of the task. We hypothesize that in combination with Zero-shot CoT, the task performance is close to saturation and automatic prompt engineering method does not provide extra benefit. The task is more challenging in the generative format and the detailed instructions in the optimized prompt bring significant performance gains. For Movie Recommendation, we found that prompt optimization methods bring significant performance gains in both cases. The optimized prompts contain multi-step plans (e. g. , consider
12
genre, director,. ), which boost the task performance. Minor formatting decisions such as outputting the option letter or the option string can still mildly affect the final accuracy. Overall, the question on “when is automatic prompt optimization most effective” is dependent on many factors, including task format, task difficulty, output space size, task model capability, etc. A. 3 Do optimized prompts generalize to other LLMs?
We evaluate 5 GSM8K prompts for our prompt generalization study (see Table 8). Note that the APO and PE2 prompts are optimized for text-davinci-003. The two prompts reported in OPRO (Yang et al. , 2023) are optimized for PaLM 2-L. We investigate the generalization of optimized prompts by evaluating them on four models: gpt-3. 5-turbo-instruct, mistral-7b-instruct-v0. 2, yi-6b, and mpt-7b-instruct. We report the results in Fig. 8. Our results do not exhibit consistent generalization trends. Optimized prompts generally outperforms the original zeroshot CoT prompt on text-davinci-003, mistral-7b-instruct-v0. 2, yi-6b. However, with gpt-3. 5-turbo-instruct, the original CoT prompt outperforms all optimized prompts. Our hypothesis is that “Let’s think step by
step. ” is included in public instruction tuning collections (Kim et al. , 2023b) and may be used during the training of gpt-3. 5-turbo-instruct. As the instruction tuning mixture used for gpt-3. 5-turbo-instruct are not disclosed, we cannot further investigate this. Overall, our results suggest that current automatic prompt optimization methods tend to find model-specific prompts that do not reliably generalize to alternative models