Large Language Models are Human-Level Prompt Engineers
Authors: Yongchao Zhou
Andrei Ioan Muresanu
Ziwen Han
Keiran Paster
Silviu Pitis
Harris Chan
Jimmy Ba
Venue: ICLR 2023 poster (2023)
Keywords: few-shot learning
automated reasoning
large language models

Abstract: By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the "program," optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. To evaluate the quality of the selected instruction, we evaluate the zero-shot performance of another LLM following the selected instruction. Experiments on 24 NLP tasks show that our automatically generated instructions outperform the prior LLM baseline by a large margin and achieve better or comparable performance to the instructions generated by human annotators on 21/24 tasks. We conduct extensive qualitative and quantitative analyses to explore the performance of APE. We show that APE-engineered prompts can be applied to steer models toward truthfulness and/or informativeness, as well as to improve few-shot learning performance by simply prepending them to standard in-context learning prompts.

Published as a conference paper at ICLR 2023
LARGE LANGUAGE MODELS ARE HUMAN-LEVEL PROMPT ENGINEERS
Yongchao Zhou Andrei Ioan Muresanu Ziwen Han 12,,, Silviu Pitis!.2, Harris Chan 12, Jimmy Ba..2
12.,, Keiran Paster
'University Toronto Vector Institute UUnivesitty of Waterloo "Equal contribution (yczhou, hanziwene keirp, spitis, hchan, jba) êcs toronto....oo... edu (andrei muresanuj @uwaterloo.ca
ABSTRACT
By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purper purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handerafted by humans. Inspired by classical program synthesis and propose Automatic Prompt Enginee
the human approach to prompt engineering, (APE) for automatic instruction generation and selection. our method, we treat the instruction as the "program," optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. To evaluate the quality of the selected instruction, we evaluate the zero-shot performance of another LLM following the selected instruction. Extensive experiments show that our automatically generated instructions outperform the prior LLM baseline by a large margin and achieve better or comparable performance to the instructions generated hy human annotators on 24/24 Instruction Induction tasks and 17/21 curated BIG-Bench tasks. We conduct extensive qualitative and quantitative analyses explore the performance of APE. We show that APE-engineered prompts are able to improve few-shot learning performance (by simply prepending them to standard in-context learning prompts), find better zero-shot chain-of- thought prompts, as well as steer models toward truthfulness and/or informativeness.
INTRODUCTION
The combination of scale and attention based architectures has resulted in language models possessing an unprecedented level of generality (Kaplan et al., 2020 Vaswani et al. 2017). These so-called "large language models" (LLMs) have shown remarkable, often superhuman, capabilities across a diverse range of tasks, including both zero-shot and few-shot setups (Brown et al. 2020 Srivastaval
et al. 2022). With generality, however, there comes a question of control: how can we make LLMs do what want them do?
To answer this question and steer LLMs toward desired behaviors, recent work has considered fine-tuning (Ouyang et al., [202]] ZZegler et al. 2019), in-context learning (Brown et al.) 2020), and several forms of prompt generation (Gao (021), including both differentiable tuning of soft softssooooonn prompts QQin & Eisner |02 Lester et al. 2021) and natural language prompt engincering (Reynolds &
McDonell 2021). I.. The latter is of particular interest, as it provides a natural interface for humans to communicate with machines and may be of great relevance not only LLMs but other generalist
models such as prompted image synthesizers (Rombach et al., 2022 Ramesh et al. 2022). for which public interest in prompt design and generation has also emerged (see Appendix A for examples).
Behind this interest is the fact that plain language prompts do not always produce the desired results, even when those results are possible to produce with alternative instructions. Thus, human users must experiment with wide range of prompts to elicit desired behaviors, as they have little knowledge of how compatible instructions are with a particular model. We can understand this by viewing LLMs as black-box computers that execute programs specified by natural language instructions: while they
We define "prompt engineering" optimizing the language in prompt in order to elicit the hest possible performance. Notably, this does not include prompts that chain multiple I.I. queries together or give the L.M access to external tools. Our code available wttpt:tt//////t http:////gg..aaoooccco//a//aaaaa http:///eeaeiaaaaaatooooooo.eaep neer

---

Published as a conference paper at ICLR 2023
X
Intoruuarte Moan Zaro Shar PorforPfrraaaa Mormang 8
0.73
Peeineer
Back aclin mataaya
Sclwing burradicu -INSLETT>
Erghher
0.11
Chapet deerre
lepe Oheee
wrthe the naker wn vent
4.2
Dessatsiio Did.
phe ahapp peddoedial
24-
(LTMivus Becorpping Moddet
2.2-
curke lil keeple
*.3 ti oppartcc wendd gives.
4:38
lepus
5.1 901
car
Ouapat
Intanageo voat
Graady (2P9.)
Grandy (Iatrus
APE (PPT-3)
API (mmmaaP7))
(a) Automatic Prompt Engineer (APE) workflow
(b) Interquartile mean across 24 tasks
Figure I: (a) Our method, Automatic Prompt Engineer (APE), automatically generates instructions for task that is specified via output demonstrations: it generates several instruction candidates, either via direct inference or recursive process based on semantic similarity, executes them using the target model, and selects the most appropriate instruction based on computed evaluation scores. (b) As measured by the interquartile mean across the 24 NLP tasks introduced by |Honovich et al. (2022). APE is able to surpass human performance when using the InstructGPT model (Ouyang et al.
2022
can execute a broad range of natural language programs, the way these programs are processed may be intuitive for humans, and the quality of instruction can only be measured when executing these instructions on a downstream task (Sanh et al. 2022 Wei et al. 2021).
To reduce the human effort involved in creating and validating effective instructions, we propose novel algorithm using LLMs to generate and select instructions automatically. We call this problem natural language program synthesis and propose to address it as a black-box optimization problem using LLMs to generate and search over heuristically viable candidate solutions. In doing so, we leverage the generalist capabilities of L.MSs in three ways. First, we use an L.M as an inference model (Ellis et al. 2021 Honovich et al. 2022) to generate instruction candidates based on a small set of demonstrations in the form of input-output pairs. Next, we guide the search process by computing score for each instruction under the LLM we seek to control. Finally, we propose an iterative Monte Carlo search method where LLMS improve the best candidates by proposing semantically similar instruction variants. Intuitively, our algorithm asks LLMs generate set of instruction candidates based on demonstrations and then asks them to assess which instructions are more promising. We call our algorithm Automatic Prompt Engineer (APE). Our main contributions are:
We frame instruction generation natural language program synthesis, formulate it as black-box optimization problem guided by LLMs, and propose both a naive and an iterative Monte Carlo search methods to approximate the solution.
Our proposed method, APE, achieves human-level performance on zero-shot learning with model-generated instructions on 24/24 Instruction Induction and 17/21 Big- Big-Bench tasks.
We provide extensive qualitative and quantitative analyses exploring various facets of APE, and demonstrate applications of APE for improving few-shot learning, finding better zero-shot chain of thought prompts, and steering LLMs toward desired behaviors such as truthfulness and/or informativeness.
2 RELATED WORK
Large Language Models Scaling up transformer-based language models in terms of model size, training data, and training compute has been shown to predictably improve performance
wide range of downstream NLP tasks Vaswaani et al.) 2017 Devlin et al. 2018 Brown et al. 2020). Many emergent abilities (Wei et al. 2022a) of LLMs Ms have been discovered as result of this scaling, including few-shot in-context learning. zero-shot problem solving, chain of thought reasoning.
instruction following, and instruction induction (Cobbe et al. 2021 Wei et al. 2022b Kojima et al.
2

---

Published as a conference paper at ICLR 2023
2022 Sanh al. 2022 Wei et al. 2021: Ouyang et al., 2022 Honovich al. 2022). In this paper, view LLMs as black-box computers that exccute programs specified by natural language instructions and investigate how to control an LLM's behavior using model-generatee instructions.
Prompt Engineering Prompting offers a natural and intuitive interface for humans to interact with and use generalist models such as LLMs. Due its flexibility, prompting has been widely
used as a generic method for NLP tasks (Schick & Schütze [202]] Brown et al.] 2020 Sanh et al.
2022). However, LLMs require careful prompt engineering, either manually (Reynolds & McDonell 202T] Shin et al. 2020), as models do not seem to understand the
2021) or automatically (Gao et al. prompts in the same way a human would (Webson & Pavlick 2021 Lu al. 202I). Though many
successful prompt tuning methods perform optimization over continuous space using gradient-
methods (Liu et al., 2021 Qin Fisner 2021; Lester et al. 202T), this becomes less practical with scale, as computing gradients becomes increasingly expensive and access to models shifts to APIs that may not provide gradient access. In our paper, we borrow components from discrete prompt search methods, such prompt generation (Gao et al. |021|
et al. (202)), prompt scoring
(Davison et al.] 2019) and prompt paraphrasing (Jiang et al., 20220 Yuan et al. (202T) to optimize instructions by searching directly in the natural language hypothesis space. As compared to this past work, which uses specialized models for each component and leans heavily on human templates, we
show that the entire search can be conducted by single LLM.
Program Synthesis Program synthesis involves the automatic search over a "program space" to find a program satisfying a particular specification (Gulwani et al. 2017). Modern program synthesis admits a wide variety of specifications, including input-output examples (Ellis et al., |021 Wong al. T02I) and natural language (Jain al. 2022). The range of feasible program spaces to search over has also grown, from historically restrictive domain-specific languages to general-purpose programming languages (Austin et al.) 2021). In contrast to prior approaches that require a suitable structured hypothesis space and library of components (Liang et al. 2010; Ellis et al.] 20188) we leverage the structure provided by I.IMS to search over the space Using inference models is a standard practice to speed up the search by restricting the search space to a limited space of possible expressions (Menon et al. 2013 Lee et al. 2018 Devlin et al. 2017 Ellis et al. 2021). Inspired by this, we use LLMs as approximate inference models to generate program candidates based on a small set of demonstrations. Unlike classical program synthesis, our inference models do not require any training and generalize well to various tasks.
natural language programs.
NATURAL LANGUAGE PROGRAM SYNTHESIS USING LLMS
We consider a task specified by a dataset Dimin = {(Q, A)} of input/output demonstrations sampled from population X, and a prompted model M. The goal of natural language program synthesis is to find a single instruction P such that, when M is prompted with the concatenation [p: Q] of instruction and given input, .M produces the corresponding output A. More formally, we frame this as an optimization problem, where we seek instruction P that maximizes the expectation of some per-sample score {p, Q, A) over possible (Q, A):
p² arg max f(p) arg max E(Q.A) [f Q.
(I)
Note that in general, Q may be the empty string, such that we are optimizing P as a prompt that directly produces outputs {A). While this task has been widely attempted by humans, we have little knowledge of how compatible any particular instruction with model M. Thus, we propose to treat this human-intractan intractable question as a black-box optimization process guided by LLMs. Our algorithm. APE, uses LLMs in each of two key components, proposal and scoring. As shown in Figure and summarized in Algorithm D APE first proposes few candidate prompts, and then filters/refines the candidate set according to a chosen score function, ultimately choosing the instruction with the highest score. We discuss options for proposal and scoring next.
3.1 INITIAL PROPOSAL DISTRIBUTIONS
Due to the infinitely large search space, finding the right instruction can be extremely difficult, which has rendered natural language program synthesis historically intractable. Recent progress in NLP has shown language models are very good at generating diverse natural language text. Therefore,
3

---

Published as a conference paper at ICLR 2023
Algorithm 1 Automatic Prompt Engineer (APE)
Require: Duni {(Q. A)).: training examples, px D + R: score function
1: Use LLM to sample instruction proposals U 6 {pI.... Pro ). (See Section 3.1
2: while not converged do
3: Choose a random training subset Drain C Drain-
4: for all in 2/ do
5: Evaluate score on the subset 3 t- f(, Duam) (See Section 3.2
6: end for
7: Filter the top k% of instructions with high scores Us C U using (51...., ...,..,,. 3m
8: Update instructions L/ U2 or use LLM resample < resample(14.) (See Section 3.3
9:
end while
Return instruction with the highest score p¹ + arg maxpat,, {p, Duain)
consider leveraging a pretrained LLM to propose a good set U of candidate solutions that will guide our search procedure. While random samples from LLMs are unlikely to produce the desired (Q, A) pairs, we can instead ask the LLM to approximately infer the most likely instructions with a high score, given the input/output demonstrations; i.e., to approximately sample from P(p Duain f(p) is high).
Forward Mode Generation We consider two approaches to gen- crate high-quality candidates from P(pl Preinn f(p) is high). First, we adopt an approach based on "forward" mode generation by trans- lating this distribution P(p] (Duain: {p) high) into words. For example, in our instruction induction experiments (Subsection 4.1). we follow Honovich et al. (2022) and prompt the LLM using Figure
Forward Generation Template
gave inputs, The friend read the instruction and output for every of the imputs. Here are the inpul-ouiput pairs:
insmuction
2 (Top).
Input: [Q:]
Output: [A1]
Input: [Qg)
Output: [Ay]
Reverse Mode Generation Although the "forward" model works out of the box for most of the pretrained LLMs, translating P(p|Dinin fp) is high) into words requires custom engineering across different tasks. This is because while instructions are typi- cally found in the beginning of passages, the "forward" model only generates text from left to right, which requires the instruction be predicted at the end of the prompt. Therefore, we desire a more flex- ible approach such that the instruction can be anywhere in the text. To address this, we consider "reverse" mode generation, which uses an LLM with infilling capabilities-c..8 e.g. T5 (Raffel al.) (020), GLM (Du et al. 2022), and InsertGPT (Bavarian et al. (002)) to infer the missing instructions. Our "reverse" model directly samples from P(pl Drain: (p) is high) by filling in the blank. We show an example the such template in Figure (Middle).
The instruction was -COMPLETE>
Reverse Goneration Template
instructed my friend to <I NISERT>>
The friend read the instruction and wrote an ourput for every one of the
Here the inpul-oulpul pairs:
Imput: [Q1]
Oulput: [Ag]
Imput: (Q2)
Output: (Ag)
Template TruthfulQA
Protéssoor Smith was given
following instuctions: <INSERT>
Customized Prompts Note that depending on the score function being used, there may exist more appropriate prompts than the sam- start
Here Professoo's resporses:
Input: (Qt]
Output: [44]] Output: [Ag]
Input: (Qal
ples above. For example, in TruthfulQA experiments, with the human-designed instructions from the original dataset (Lin et al. 2022) and ask the the "reverse" model propose initial in- struction samples that fit the missing context (Figure (Bottom).
Figure 2: Prompts for LLMs
3.2 SCORE FUNCTIONS
To cast our problem as black-box optimization, we choose score function that accurately measures the alignment between the dataset and the data the model generates. In our instruction induction experiments, we consider two potential score functions, described below. In the TruthfulQA ex- periments, we focused primarily on automated metries proposed in [in et al. (2022), similar to the execution accuracy. In each case, we evaluate the quality of a generated instruction using Equation (I) and take the expectation
held-out test dataset D
Execution accuracy First, we consider evaluating the quality of an instruction using the execution accuracy metric proposed by (Hnovicch al. (2022), which we denote fexee- In most cases,
4

---

Published as a conference paper at ICLR 2023
execution accuracy is simply defined the 0-1 loss, ((p.Q.A) 1 [M[[p: A]. On some tasks, execution accuracy takes into account invariants; e.g., it may be an order invariant set matching loss, described in Appendix A Honovich et al. (2022).
Log probability We further consider a softer probabilistic score function, which we hypothesize might improve optimization by providing more fine-grained signal when searching low-quality instruction candidates. In particular, we consider the log probability of the desired answer given the instruction and question under the target model M, which per sample basis, is log P(A [p; Q]).
Efficient score estimation Estimating the score by computing the score over the entire training dataset for all instruction candidates can be expensive. To reduce the computation cost, adopt a filtering scheme where a promising candidate receives more computation resources while low- be achieved by using multi-stage computation
quality candidate receives less computation. strategy on lines 2-9 Algorithm| I We first evaluate all candidates with small subset of the training dataset. For the candidates with score greater than certain threshold, we sample and evaluate new non-overlapping subset from the training dataset to update the moving average of the score. Then, we repeat this process until a small set candidates is left, which are evaluated on the entire training dataset. This adaptive filtering scheme significantly improves the computation efficiency by keeping the exact computation costs for the high-quality samples and drastically reducing the computation costs for low-quality candidates. We note that similar score estimation scheme has been used in previous works (Li et al. 2022 Maclaurin & Adams 2015).
3.3 ITERATIVE PROPOSAL DISTRIBUTIONS
Despite our attempt to directly sample high-quality initial instruction candidates, it could be the case that the method described in Subsection 3.1 fails to produce good proposal set 2,, either because it lacks of diversity or does not contain any candidates with suitably high score. In case of such challenges, we explore an iterative process for resampling U.
Iterative Monte Carlo Search Instead of only sampling from the initial proposal, we consider exploring the search space locally around the current best candidates. This allows us to generate new instructions that are more likely to be successful. We call this variant iterative APE. At each stage, evaluate set of instructions and filter out candidates with low scores. Then, an LLM asked to generate new instructions similar to those with high scores. We provide the prompt used for resampling in Figure Figure 6 (Right) shows that although this approach improves the overall quality of the proposal set Ut the highest scoring instruction tends to remain the same with more stages. We conclude iterative generation provides marginal improvement over the relative simplicity and effectiveness of the generative process described in Subsection 3.1 Therefore, we use APE without iterative search as default unless otherwise stated.
Prompt for Resampling
Generate variation the following instruction while
keeping the semmantc meaning.
Input: [INSTRUCTION]
Output: <COMPLETE>
Figure 3: Resampling
4 LARGE LANGUAGE MODELS ARE HUMAN-LEVEL, PROMPT ENGINEERS
This section examines how APE can guide LLMs to desired behaviors. We investigate from four perspectives: zero-shot performance, few-shot in-context learning performance, zero-shot chain-of- thought reasoning, and truthfulness. Our experiments show that APE can find prompts that improve task performance, performing equal to or even better than those authored by humans. APE also often produces insightful tricks for how to best prompt language models that can be successfully transferred to new tasks (see Section 4.3
4.1 INSTRUCTION INDUCTION
We assess the effectiveness of zero-shot and few-shot in-context learning 24 instruction induction tasks proposed in Honovich et al. (202). The tasks span many facets of language understanding, from simple phrase structure similarity and causality identification. We provide detailed descriptions of each task in Appendix B. For each task, we sample five input-output pairs from the training data and select the best instruction using algorithm I Then, evaluate the quality of the instruction
5

---

Published as a conference paper at ICLR 2023
Gready
Execution Accuracy
Selerdon
Formaiiyy
Liste
Menceencpp
Negition
Number Word
Preeeccation
Plume Lewtio
Rhymee
Swannd Leter
Swntenoo Simberty
Sentifient
Starting With
Sum
Synonyms
Transiatio cn
Transiation
Translaton en
Word In Context
Figure 4:
Zero-shot test accuracy on 24 Instruction Induction tasks. APE achieves human-level or better performance all 24 out of 24 tasks.
by executing the instruction on InstructGPT We repeat our experiments five times with different random seeds to report the mean and standard deviation. The exact templates for our experiments can be found in Appendix (Table
Zero-shot Learning We compare our method against two baselines: human prompt engineers (Human 1 and the model-generated instruction algorithm proposed by Honovich al. (2022). This algorithm can be thought of as greedy version of APE, without search and selection process; thus, we refer to it as "Greedy". Figure [ shows the zero-shot performance of InstructGPT using human instructions and model generated instructions. Our algorithm outperforms "Greedy" on every task and achieves equal or better than human performance on 24 of 24 tasks. Morcover, the Interquartile Mean (IQM) (Agarwal al. 2021 across all 24 tasks in Figure T suggests that APE with InstructGPT outperforms human-engineered prompts, obtaining an IQM of 0.810 vs humans' 0.749.
We summarize the instruction selected by APE for each task in Appendix (Table T2].
Few-shot In-context Learning We evaluated APE-generated instructions in few-shot in-context learning, where insert the instruction before the in-context demonstrations. Those instructions are selected based on zero-shot execution accuracy, and we denote this setting as "Instruction + In-context" in Figure As shown in Figure g adding an instruction achieves comparable or better test performance than the standard in-context learning performance on 21 of 24 tasks. Counter- intuitively, adding in-context examples for Rhymes, Large Animal, and Second Letters hurts model performance. We conjecture that it may be because the selected instructions overfit the zero-shot learning scenario and thus do not perform well on few-shot case. Therefore, we experiment using few-shot execution accuracy as the selection metric. Figure [hows that the few-shot metric achieves comparable or slightly better than the zero-shot metric except for Rhymes. have an intuitive understanding of what is happening, provide qualitative analysis in Appendix C.I
4.2 BIGBENCH
To see whether APE can be applied to more challenging tasks, we propose and curate BIG-Bench Instruction Induction (BBII), clean and tractable subset of 21 tasks that have clear, human--ritter instruction that can be applied to all examples in the dataset. The selected tasks cover many facets of language understanding and includes all nine such problems from the BigBench- Harrdd Subset (Suzgun et al. 2022). In particular, it includes emotional understanding, context-free question answering, reading comprehension, summarization, algorithms, and various reasoning tasks (e.g., arithmetic, commonsense, symbolic, and other logical reasoning tasks). We provide detailed description of the task and our selection criteria in Appendix
We use the text-davinci-002 the OpenAl API (https:: /bttabbaaac//// a opena i com/) Though not stated explicitly in the API, we assume the models are those reported by Ouyang et al, (2022).
(2.22)
We the gold annotations from Honovich et al. (022 which were manually verified for correctness.

---

Published as a conference paper at ICLR 2023
For each task, used the reverse mode generation of InstructGPT to generate a set of instruction candidates and ranked the instructions based on their execution accuracy. Then, we executed the selected instruction
InstructGPT to compute the zero-shot performance on the test set and compared it with the default human prompt. As shown in Table APE achieves comparable or better performance than the default human prompt 17 out of 21 tasks.
4.3 ZHRO-SHOT CHAIN OF THOUGHT
Ohain-of-though reasoning has been shown to dramatically improve the ability of LLMs to complete complex reasoning tasks, such solving math problems that require multiple steps. Early works (Nye) ct al. 2021 Betz et al., 2021 Wei et al. 2022b) on thain-of-thoughe used fine-tuning or in-context learning to get LLMs to show their work for such problems. One of the most influential recent works of prompt engincering was the discovery (Kojima et al. 2022) that LLMs could be made to give chain-of-thoughts simply by prepending "Let's think step by step." to the begimning of the LLM's response. Known as Zero-Shot-CoT,, this prompting strategy improves the zero-shot performance of InstructGPT MultiArith (Roy & Roth (016) from 17.7 to 78.7 and improves performance on GSM8K(Cobbe et al.) 2021) from 10.4 to 40.7. As shown in Table 7 Kojima et al. (2022) found their prompt was the best performing out of at least nine human-designed prompts.
We used APE to automatically search for the best answer-prefix across the suite of tasks used in Kojima et al. (022). Our approach to optimizing this prompt was inspired by Zelikman et al. (2022). First, we generate dataset of questions and reasoning steps generated using InstructGPT with "Let's think step by step." Then, we remove any data points that had incorrect answers. Finally, we use APE to find prompt starting with "Let's" that maximizes the likelihood of these correct reasoning steps. See 55 for the template used for prompt generation and evaluation. APE produces the prompt "Let's work this out in step by step way to be sure we have the right answer." This generated prompt further improves performance from 78.7 to 82.0 on MultiArith and from 40.7 to 43.0 on GSM8K. We believe this general workflow represents common use-case for APE where prompt engineers use APE to optimize parts of their exiting templates to improve performance. See Figure for details on the performance of this prompt on other reasoning tasks.
4.4 TRUTHFULQA
We apply our method on TruthfulQA (Lin et al., 2022) see how APE PrE-geeerated instructions can steer an L.M to generate answers with different styles, and study the trade-off between truthfulness and informativeness. Borrowing the metrics from the original paper, we use APE to the learn instructions that maximize three metrics: truthfulness (% True), informativeness (% Info), and combination of both (TTrue+ + Info). [in et al. (2022) used human evaluation to assess the model performance, but they found their automated metrics align with human prediction over 90% of the time. In our experiments, we rely on their fine-tuned GPT-judge and GPT-info to evaluate the scores.
Prompt Engineering in TruthfulQA We want to stress that the TruthfulQA dataset is intended to test pretrained models in zero-shot settings. Our results are not in any way compatible with the original benchmarks. Because we have optimized the instructions using small portion of the question and answer pairs as training demonstrations, our results are not "true few-shot learning" (Perez et al., 202I). We randomly sampled 100 out of 817 questions for the actual experiments to form training demonstrations Drrain- To sample the proposal set 24 we ask a "reverse" model to generate instructions based on six randomly chosen demonstration pairs, similar to our previous experiments. Unlike in Instruction Induction, in TruthfulQA, we aim to find single best instruction prompt that works well across all 38 categories of questions spanning health, law, politics, and fiction. It is worth noting all our generated instructions are very generic, e.g., "You will be asked a series of questions. For each question, you must either answer the question or decline to answer, which case you must state that you have comment", and do not contain any examples from the dataset.
Truthfulness ys Informativeness Trade-off We found that APE outperforms the human-engineered prompt with only 200 candidates proposed by InstructGPT (175B), as seen in Figure We compared our generated prompt with the "help" prompt from Lin et al. 2022). The training and test performance shown in Figure Figure(ja)-(te (b). We found that choosing the top 10 of 200 candidates on the training set generalizes well to the test set. We report the average performance across the top 10 instructions for the three metrics. This result by itself is not surprising as the human baseline is not carefully
7

---

Published as a conference paper at ICLR 2023
Homan
Hime
up Katan
mat
(a) Average performance Train
(b) Average performance
(c) % TTue-& IInfo trade-off (d) % True- IInfo trade-off
Test
Training
Test
Figure 5: Comparison of APE and "help" (human) prompt on the TruthfulQA task. (a) Percentage of answers that were either true (% True), informative (% Info), or both (% True + % Info) on the 100 training examples. (b) Same data on the 717 test examples. (c) % True-%Info frontier computed on training data with top 10 instructions from each metric. (d) % T'True-%lll IIno frontier on the test data.
chosen, as pointed out by Askell et al. (2021). However, we found that the instructions discovered by APE can achieve very high truthfulness with answers such as "No comment," but these answers provide little information. We used our top candidates further investigate the trade-off between truthfulness and informativeness. We visualize the top 10 proposed samples across the three metrics on the truthfulness-infonl informative plots shown Figure (ic) and Figure While APE achieves over 40% accuracy in providing both true and informative answers (v.s. 30% by the "help" prompt from humans), the instructions discovered tend to target the two ends of this Strue-%inn Pareto frontier.
QUANTITATIVE ANALYSIS
In this section, we conduct quantitative analyses to better understand the three main components of our method: proposal distribution, score functions, and iterative search. Moreover, we conduct cost analysis in the Appendix to understand the most cost-efficient way find the best prompt. We observe the larger and more powerful language models are more cost-effective for generating the best prompt despite a higher per-token cost.
5.1 LLMS FOR PROPOSAL DISTRIBUTION
How does the proposal quality change model size affects the quality of the initial proposal distribution, we examine eight different model available via the OpenAl API. To assess the quality of the proposal distribution, we generate 250 instructions per model and compute the execution accuracy on 50 test data points. We visualize the survival function (percentage of instructions with test accuracy greater than certain threshold)
increase the model size? To understand how the
and the histogram of test accuracy for simple task (i.e., Pluralization) in Figure (a) and include a similar plot for more challenging task (Start With) in the Appendix (Figure 28) As shown in both figures (and unsurprisingly), larger models tend to produce better proposal distributions than smaller ones, as do the models that were fine-tuned to follow human instructions. On the simple task, all instructions generated by the best model, InstructGPT (175B), have reasonable test accuracy. In contrast, half of the instructions are off-topic and perform poorly on the more challenging task.
5.2 LLMS FOR SELECTION
Does proposal quality matter under selection? we sample more instructions from the LLMs. then it becomes more likely for us to find better instructions. To verify this hypothesis, we increase the sample size from to 128 and evaluate the test accuracy change. Figure 7 (Left) shows a monotonically increasing trend with a diminishing return, as human-leve performance is achieved with 64 instruction samples. Thus, we choose 50 as our default sample size. Under this configuration, we investigate how the proposal distribution affects the test accuracy of the best instruction selected by our algorithm. Figure [bb) shows that though the small models may be less likely to generate good instructions, they nonetheless generate some good ones if we sample enough candidates. Therefore, we still find promising instructions with a small model by running our selection algorithm, explaining why our method outperforms the greedy approach Honovich et al. 2002 across all eight models.
We use ada, babbage, curie, davinci, text-ada-001, text-hahbage-001, text-curic-001, text-davanci-002
8

---

Published as a conference paper at ICLR 2023
GPT-3120M) 211-111
5P13 20
InstueS 10.00
Inaamctions with accunbsy
04-
Count
Colnt
000
06
02
04
06
80
66
Twir accurery (1)
Town acoumey (1)
Trei WEEIENEy [t)
Figure 6: (Left) Quality of the proposal distribution of models with different size assessed by test execution accuracy. (Right) Iterative Monte Carlo search improves the quality of the instruction candidates at each round.
10
0.8
Biete Nevito
Spatmar Conditer
06
Exexoution
0.6
Secandl Leter
Pankiiantor
Transisioon
0.5
APE (Tra APT (teel)
00-
Ligf
Hunar
15 Inder
Posterer
Senkmamt
Aetaryna
Cauac Betckkn
Figure 7: (Left) Test execution of the best instruction as we increase the number of instruction candidates. We report the mean and standard deviation across different tasks. (Middle) Spearman Correlation between the test accuracy and two metrics on 24 tasks, (Right) Test execution accuracy of the best instruction selected using APE and iterative APE (APE (IT)).
Which scoring function is better? We compute the correlation between the test accuracy and two metrics on 24 instruction induction tasks to study how good our proposed metrics are. We generate 250 instructions per task using InstructGPT (175B) in "forward" mode and compute the metric score and test accuracy on 10 test data points. We visualize the Spearman correlation between the test accuracy and two metrics. Figure (Middle) shows that the execution accuracy aligns better with the test performance across the tasks. Thus, we choose it as our default metric unless otherwise stated.
5.3 ITERATIVE MONTE CARLO SEARCH
Does Iterative Search improve the instruction quality? We visualize the survival function and histogram of test accuracy on the "Passivization" task Figure (Right) and include five more tasks in the Appendix. The survival plot shows that the curves increase the round goes up, which suggests that iterative search does result diminishing returns to further selection rounds the quality seems to stabilize after three rounds.
higher-quality proposal set. However, we observe
Do we need Iterative Search? We compare APE and iterative APE on six task As shown in Figure 7 the iterative search marginally improves performance tasks where APE underperforms humans but achieves similar performance on the other tasks. This is consistent with our hypothesis that iterative search would be most useful tasks where generating good initial o4 is challenging.
CONCLUSION
Large language models can be seen as general-purpor computers that execute programs specified by natural language prompts. We automate the prompt engineering process by formulating it as black-box optimization problem, which we propose solve using efficient search algorithms guided by LLMs. Our method achieves human-level performance on various tasks with minimum human inputs. As recent LLMs demonstrate impressive ability to follow human instruction, expect many future models, including those for formal program synthesis, to have a natural language interface. This work builds the foundation control and steer generative artificial intelligence.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like to thank Or Honovich and Michael Zhang for their help and valuable feedback. JB was supported by NSERC Grant [2020-06904), CIFAR Al Chairs program, Google Research Scholar Program and Amazon Research Award. KP was supported by NSERC PGS-D. SP was supported by NSERC CGS-D. HC was supported by NSERC CGS-D and RBC Graduate Fellowship. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govermment of Canada through CIFAR, and companies sponsoring the Vector Institute for Artificial Intelligence.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169/ 2022.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. urXiv preprint arXiv:2 (at:x::2112200000 12.00661, 2021.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108. 077332, 2021.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255, 2022.
Eyal Ben-David, Nadav Oved, and Roi Reichart. Pada: A prompt-based autoregressive approach for adaptation to unseen domains. arXiv preprint arXiv:2102. 12206. 2021.
Gregor Betz, Kyle Richardson, and Christian Voigt. Thinking aloud: Dynamic context generation improves zero-shot reasoning performance of gpt-2. arXiv preprint arXiv:2 103. 13033, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.0333 03334447.3334 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 10.1166. 2021.
Joe Davison, Joshua Feldman, and Alexander M Rush. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 1173-11777 2019.
Jacob Devlin, Rudy R Bunel, Rishabh Singh, Matthew Hausknecht, and Pushmeet Kohli. Neural program meta-induction. Advances in Neural Information Processing Systems, 30, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 2018.
10

---

Published as a conference paper at ICLR 2023
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 320-335, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.1 1865// v1/2022.acl-long.22 26.
Kevin Ellis, Lucas Morales, Mathias Mable,Meyer, Armando Solar-Lezama, and Josh Tenen- baum. Learning libraries of subroutines for neurally-guidee bayesian program induction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman,
Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL httpp:::/ppoceeeeeeeeeeee wewrrrrrcceeeiii.e..rr..rooooose newrips htt.://ceeeeccerrerrreooooo////s (a06500000..00677011133333000000
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama and Joshua Tenenbaum. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the 42nd acm sigplan international conference on programming language design and implementation, pp. 835-850, 2021.
Tianyu Gao. Prompting: Better ways of using language models for nlp tasks. The Gradient, 2021.
Tinyyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- fies and the I 1th International Joint Conference on Natural Langagge Processing ( (Volume I: Long Papers), pp- 3816-383,, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 http:///2211..1llooo ac1. URL https://aclattthollloe wwwp:/looyy.rr/221...ac.o.
295
Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. Program synthesis. Foundations and Trendy@ in Programming Languages, 4(1-2):1-119, 2017.
Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.10782 2022.
Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, and Rahul Sharma. Jigsaw: Large language models meet program synthesis. In Proceedings of the 44th International Conference on Software Engineering, pp. 1219-1231, 2022.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.0888 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205. /996,, 2022.
Woosuk Lee, Kihong Heo, Rajeev Alur, and Mayur Naik. Accelerating search-based program synthesis using learned probabilistic models. ACM SIGPLAN Notices, 53(4):436-449, 2018.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for pframeter-efffcient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp- 3045-3059, 2021.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schritwieser,, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.00 070114, 2022.
Percy Liang, Michael I. Jordan, and Dan Klein. Learning programs: A hierarchical bayesian approach. In Johannes Fürnkranz and Thorsten Joachims (eds.), Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pp. 639-646 646. Omnipress, 2010. URL https:/ / cml cc/ /tt.//oofeeeeenee0200///////oe/
11

---

Published as a conference paper at ICLR 2023
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic hu- man falsehoods. Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. As- sociation for Computational Linguistics. doi: 10. 10.18665/v112022.aclllomo 7/aclantho ogy. www.og..g.ggoao.. a0122aaclllonn 229
URL https
Xiao Liu, Yanan 7heng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2 1000//////5... 2021.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08766, 2021.
Dougal Maclaurin and Ryan Prescott Adams. Firefly monte carlo: Exact meme with subsets of data. In Tiwenty International Joint Conference on Artificial Intelligence, 2015.
Aditya Menon, Omer Tamuz, Sumit Gulwani, Butler Lampson, and Adam Kalai. A machine learning framework for programming by example. In International Conference on Machine Learning, pp- 187-195. PMLR, 2013.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21 14, 2021.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02/55, 2022.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in Neural Information Processing Systems, 34:1 1054-11070, 2021.
Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.
2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-7, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. 10Xi::1608.01113. 2016.
arXiv preprint
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, 2022.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269, 2021.
12

---

Published as a conference paper at ICLR 2023
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alons et al. Beyond the imitation game: Quantifying and extrapolating the capabilittess of language models. arXiv preprint wrX11:2206.04615, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Chi, Denny Zhou, et al. Challenging big-bench tasks and whether thain-of-thought can solve them. arXiv preprint vrXhv:2210.092661 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? arXiv preprint (rXiv:2109.01247, 2021.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sehastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. 07682, 2022a.
arXiv preprint
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:220.111003, 2022b.
Catherine Wong, Kevin M Ellis, Joshua Tenenbaum, and Jacob Andreas. Leveraging language to learn program abstractions and search heuristics. In International Conference Machine Learning, pp. 11193-113 11204. PMLR, 2021.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. Advances Neural Information Processing Systems, 34:27263-27277, 2021.
Eric Zelikman, Yuhuai Wu, and Noah D Goodman. Star: Bootstrapping reasoning with reasoning. arXiv preprint 2rXhv:2203.144000 2022.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130k An open bilingual pre-trained model. arXiv preprins 20X1v:2210.02414, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0/00 2022.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei. Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint /909 2019.
13

---

Published as a conference paper at ICLR 2023
A PROMPT ENGINEERING IN THE WILD
Large models with natural language interfaces, including models for text generation and image synthesis, have seen an increasing amount of public usage in recent years. As finding the right prompt can be difficult for humans, a number of guides on prompt engineering as well as tools to aid in prompt discovery have been developed. Among others, see, for example:
In this paper we apply APE generate effective instructions for steering LLMs, but the general framework Algorithm |
could be applied to steer other models with natural language interfaces so long appropriate proposal method and scoring function be designed.
14

---

Published as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
Table 1: Detailed description of 24 instruction induction tasks proposed in Honovich al. (2022) For convenience, the original table from Honovich et al. (2022) is duplicated here.
Category
Task
Instruction
Demonstration
Spelling
First Letter
Extract the first letter of the input word.
cat
Second Letter
Extract the second letter of the input word. cat
List Letters
Break the input word into letters, sepa- rated by spaces.
cat
Starting With
Extract the words starting with given letter from the input sentence.
The man whose car I hit last week sued me. [m] man, me
Morpho- syntax
Pluralization
Convert the input word to its plural form.
cats
Passivization
Write the input sentence in passive form.
The artist introduced the scientist.
+ The scientist was introduced by the artist.
Syntax
Negation
Negate the input sentence.
Time is finite + Time not nite.
Lexical Semantics
Antonyms
Write word that means the opposite of the input word.
won + lost
Synonyms
Write word with a similar meaning to the input word.
alleged + supposed
Membership
Write all the animals that appear in the given list.
cat, helicopter, cook, whale, frog, lion frog, cat, lion, whale
Phonetics
Rhymes
Write word that rhymes with the input word.
sing -> ring
Knowledge
Larger Animal
Write the larger of the given animals.
koala, snail + koala
Semantics
Cause Selection
Find which of the two given cause and effect sentences is the cause.
Sentence 1:
The soda went flat.
Sentence 2: The bottle was left open. The bottle was left open.
Common Concept
Find common characteristic for the given objects.
guitars, pendulums, neutrinos involve oscillations.
Style
Formality
Rephrase the sentence in formal language.
Please once you get there Please call upon your arrival.
Numerical
Sum
Sum the two given numbers.
-
Difference
Subtract the second number from the first.
22 +
Number to Word
Write the number in English words.
26 twenty-six
Multi- lingual
Translation
Translate the word into German / Spanish French.
game juego
GLUE
Sentiment Analysis
Determine whether movie review is pos- itive or negative.
The film is small in scope, yet per- feetly formed. + positive
Sentence Similarity
Rate the semantic similarity of two input sentences on scale 0 definitely not 5o perfectly.
Sentence 1: A is smoking. Sentence man skating. 0 definitely not
Word in Context
Determine whether an input word has same meaning in the two input sentences.
Sentence 1: Approach task. Sen- tence 2: To approach the city. Word: approach -> not the same
15

---

Published as a conference paper at ICLR 2023
Table 2: Detailed description of BIG-Bench Instruction Induction (BBII, clean and tractable subset of 21 tasks that have clear human written instruction that can be applied to all examples in the dataset.
Name
Description
Keywords
causal reasoning, common sense, multi- plc choice, reading comprehension, so- cial reasoning
causal judgment
Answer questions about causal attribution
Clarify the meaning of sentences with am- biguous pronouns
common sense, gender bias, many-shot, multiple choice
disambiguation qa
algebra, arithmetic, logical reasoning, multiple choice
dyck languages
Correctly close Dyck-n word
common sense, logical reasoning, mul- tiple choice, social reasoning, theory of mind
Determine whether one sentence entails the next
epistemic reasoning
Given German language sentence that does not gender-inclusive forms, transform it to gender-inclusive forms
gender inclusive sentences german
free response, grammar, inclusion, non- English, paraphrase
contextual question answering, multiple choice, reading comprehension, social reasoning, theory of mind
Predict whether Speaker 2's answer to Speaker I counts yes
implicatures
free response, human-like behavior, lin- guistics. logical reasoning, reading com- prehension
linguistics puzzles
Solve Rosetta Stone-style linguistics puzzles
logical fallacy de- tection
Detect informal and formal logical fallacies
logical reasoning, multiple choice
movie recommenda-
Recommend movies similar to the given list movies
emotional intelligence, multiple choice
Given series of navigation instructions, de- termine whether one would up back the starting point
arithmetic, logical reasoning, mathemat- ics, multiple choice
navigate
Questions that involve enumerating objects of different types and asking the model to count them
object counting
free response, logical reasoning
Given mathematical operator definition in natural language, apply it
free response, mathematics, numerical response
operators
presuppositions as nli
Determine whether the first sentence entails contradicts the second
common logical reasoning, multi- plc choice
Given short select most appropriate question which to the given short answer
along with its context,
multiple choice, paraphrase, reading comprehension, summarization
question selection
Select the humorous edit that 'ruins' the put movie or musical artist name
emotional understanding, multiple choice
ruin names
Determine which of two sentences sarcas- tic
emotional understanding, humor, multi- ple choice
snarks
understand- Determine whether an artificially con- structed sentence relating to sports plausi- ble implausible
common sense, context-free question answering. domain specific, multiple choice
sports
ing
tense
Modify the tense of given sentence
free response, paraphrase, syntax
Evaluate the reasoning in answering Wino- grad Schema Challenge questions
causal reasoning, common sense, multi- ple choice, social reasoning
winowhy
word sorting
Sort list of words
algorithms, free response
Unscramble the given letters to form En- glish word
free response, implicit reasoning, tok- enization
word unscrambling
16

---

Published as a conference paper at ICLR 2023
B.1 BIG-BENCH INSTRUCTION INDUCTION (BBII) SELECTION PROCESS
Step I: BIG-Bench contains a large number of evaluation tasks with different level of quality. For example, some of the tasks only have the minimum number of examples needed to qualify for submission, while other tasks may lack an appropriate human baselines. Therefore, we follow |Suugun al. (222) to get clean and tractable subset based on the following criteria.
Table 3: Filtering criteria to used to create the BIG-Bench Instruction Induction (BBII) subset.
# Tasks
Criteria
212
All BIG- BIG-Bench tasks
170
All JSON tasks
127
After filtering out tasks with more than one sub-task
74
After filtering out tasks with fewer than 150 examples
67
After filtering out tasks without human-rater baselines
57
After filtering out tasks that do not multiple-choice exact match as the evaluation metric
Criteria: JSON Tasks.
Discarded tasks: abstraction and reasoning corpus, bbq lite, bias from probabilities, boolean expres- sions, com2sense, context definition alignment, convinceme, coqa conversational question answering, cycled letters, diverse social bias, dynamic counting, factuality of summary, forecasting subques- tions, gender sensitivity chinese, gender sensitivity english, high low game, long context integration, multistep arithmetic, muslim violence bias, program synthesis, protein interacting sites, python programming challenge, question answer creation, roots optimization and games, self awareness, self evaluation courtroom, self evaluation tutoring, simple arithmetic, spelling bee, squad shifts, subject verb agreement, sudoku, taboo, talkdown, text navigation game, training on test set, truthful qa, twenty questions, unqover, web of lies, word problems on sets and graphs, yes no black white.
Criteria: Tasks without sub-task.
Discarded tasks: abstract narrative understanding, arithmetic, authorship verification, bbq lite json, cause and effect, chess state tracking, cifar10 classification, color, conceptual combinations, conlang translation, cs algorithms, elementary math qa, fact checker, gem, goal step wikihow, hhh alignment, indic cause and effect, intersect geometry, kanji ascii, key value maps, language games, linguistic mappings, list functions, logical deduction, metaphor understanding, minute mysteries qa, modified arithmetic, mult data wrangling, multiemo, natural instructions, periodic elements, physics, real or fake text, simp turing concept, simple arithmetic json subtasks, simple ethical questions, strange stories, symbol interpretation, tracking shuffled objects, undo permutation, unit conversion, unit interpretation, unnatural in context learning.
Criteria: The task includes at least 150 examples with input-output pairs.
Discarded tasks: analytic entailment, auto debugging, code line description, codenames, common morpheme, crash blossom, crass ai, cryobiology spanish, dark humor detection, emoji movie, emojis emotion prediction, empirical judgments, english proverbs, english russian proverbs, entailed polarity, entailed polarity hindi, evaluating information essentiality, figure of speech detection, general knowledge, gre reading comprehension, human organs senses, identify math theorems, identify odd metaphor, implicit relations, international phonetic alphabet nli, irony identification, known unknowns, logical args, logical sequence, mathematical induction, misconceptions russian, nonsense words grammar, novel concepts, odd one out, penguins in table, persian idioms, phrase relatedness, physical intuition, physics questions, repeat copy logic, rephrase, riddle sense, scientific press release, sentence ambiguity, similarities abstraction, simple arithmetic json, simple arithmetic json multiple choice, simple arithmetic multiple targets json, simple text editing, sufficient information, suicide risk, swedish to german proverbs, what is the tao.
Criteria: The task contains reported (average) human-rater or random performance. Discarded tasks: contextual parametric knowledge conflicts, hinglish toxicity, medical questions russian, parsinlu qa, swahili english proverbs, tellmewhy, which wiki edit.
Criteria: The task is classification or uses exact match the evaluation metric. Discarded tasks: auto categorization, few shot nlg, hindi question answering, international phonetic alphabet transliterate, polish sequence labeling, qa wikidata, semantic parsing in context spare, semantic parsing spider, social support, topical chat.
17

---

Published as a conference paper at ICLR 2023
Step 2: We do manual inspection to divide the remaining tasks to the following three categories. In particular, Big- Big-Bench Instruction Induction (BBII) subset is the subet we used to evaluate APE in Section
BBII Subset: subset of Big Bench Tasks that satisfy the instruction induction format: each example in the dataset can be expressed as a question-answer answer pair, all examples focus the same question that can be clearly described by human instruction, and there is human instruction available in the task JSON file.
Invalid Format: Tasks that do not match the instruction induction format: each example in the dataset asks different question, clear human instruction is not available.
Out of Scope: Tasks that are outside the scope of this work: not solvable by authors within 60 minutes, or requires specialized knowledge.
Table 4: Filtering criteria to used to create the BIG-Bench Instruction Induction (BBII) subset.
Category
Tasks
Tasks Names
causal judgment, disambiguation qa, dyck language, epistemic reasoning, gender inclusive sentences german, implicatures, linguistics puzzles, logical fallacy detection, movie recommendation, navigate, object counting, operators, presup- positions as nli, question selection, ruin names, snarks, sports understanding. tense, winowhy, word sorting. word unscrambling.
BBII Subset
21
anachronisms, analogical similarity, bridging anaphora resolution barqa, data understanding, disfl qa, fantasy reasoning, formal fallacies syllogisms negation, hindu knowledge, hyperbaton. intent recognition, logic grid puzzle, paragraph segmentation. play dialog same or different, reasoning about colored objects. salient translation error detection, social iqa, strategyqa, temporal sequences. timedial, understanding fables, vitamine fact verification.
Invalid Format
21
ascii word recognition, checkmate in onc, chinese remainder theorem, cryptonite, discourse marker prediction, geometric shapes, kannada, language identification, matrixshapes, mnist ascii, moral permissibility. movie dialog same or different, parsinlu reading comprehension.
Out of Scope
13
18

---

Published as a conference paper at ICLR 2023
Table 5: Raw templates used for model prompting in our experiments
Usage
Template
Instruction: [INSTRUCTION]
Zero-shot Evaluation
Input: (QeeallaOupputtCCMMPTT>>
Instruction: [INSTRUCTION]
Few-shot Evaluation
Input: [Qi]inOutput: [A1 I'innnppt: [Qz]InOutput: [A2]
Input: [Ques//aobtpuuttCCOMEPELE
gave friend an instruction and five inputs. The friend read the instruction and output for every one the inputs. are the input-output pairs:
Forward Generation
Input: [Q]]nOutput: [A]]a'alnput: [Qz]inOutput: [A2] ...
The instruction COmPCOEE
instructed my friend to<INSERT>,Th The friend read the instruction and wrote output every the inputs.inHere are the input-output pairs:
Reverse Generation
Input: [Q]]nOttput: [A:]a'nInput: [Q2]inOutput: [A2]
Professor Smith given the following Insmectiooss:INNSRRRnnnnnee are the Professor's responses:
Reverse Generation 2
Q: [Qı]mA: [A]]ininQ [Q2]inA: [Ag] ...
Generate variation of the following instruction while keeping the semantic meaning.
Resample Instruction
Input: (NNSTTRUCCTOOYYOOOOOPCOCCOMEEEEE
Instruction: Answer the following question.
Zero-shot-CoT
Q: [INPUT]IAA: Let's <INSERT>, [OUTPUT]
19

---

Published as a conference paper at ICLR 2023
C ADDITIONAL RESULTS
C.I INSTRUCTION INDUCTION
Few-shot In-context Learning We evaluated APE-generated instructions the few-shot in-context learning, where we insert the instruction before the in-context demonstrations. Those instructions are selected based on zero-shot execution accuracy, and we denote this setting as "Instruction + In-context" in Figure 8 As shown in Figure 8 adding instruction achieves comparable or better test performance than standard in-context learning performance on 21 intuitively, adding in-context examples for Rhymes, Large Animal, and Second Letters hurts model performance. We conjecture that it may be because the selected instructions overfit the zero-shot learning scenario and thus do not perform well the few-shot case. Therefore, experiment using few-shot execution accuracy as the selection metric. Figure shows that the few-shot metric achieves comparable slightly better than the zero-shot metric except for Rhymes. To have intuitive understanding of what is happening, we provide a qualitative analysis below.
tasks. Counter-
Few-shot Qualitative Analysis We find adversarial case Rhymes when combining the instruction and in-context prompts. Table that 4 of 5 filtered instructions ask to echo the input word. These proposals effectively hack the evaluation with near-perfee near-perfect test accuracy. as every word rhymes with itself. However, adding in-context examples for these instructions creates a misalignment between instruction (induces trivial rhymes) and context (induces non-trivial rhymes), resulting in significant drop in performance, If we instead score the instructions based on the few-shot metric, this performance drop can alleviated since the model can choose a more aligned instruction.
Instruction Ony
Inccontext Only
Instructin
Execution Accuracy
Seleciio
LatLebes
Memeersiip
Negation
Number Word
Passivaation
Pluna Laation
Rhymes
Sscand
Sentence Simbbrty
Sentiment
Starting With
Sum
Syronyme
Transleton wn-de
Traneiation enee
Transbrion wadt
Word In Contest
Figure 8: Few-shot in-context test accuracy on 24 Instruction Induction tasks. APE improves the few-shot in-context learning performance 21 of 24 tasks.
20

---

Published as a conference paper at ICLR 2023
C.2
INSTRUCTION INDUCTION
We use APE to generate new prompts for the tasks in BIG-Bench Instruction Induction (BBII). When compared human prompts, APE-generated prompts improve match zero-shot performance on 17 out of 21 tasks. We report the normalized preferred metric defined in Srivastava et al. (0022) Under this metric, score of 100 corresponds human expert performance, and corresponds to random guessing. Note that a model can achieve score less than 0 if it performs worse than random guessing on multiple-choice task.
Human
APE
80
Normalized Performance
60
40
20
-20
calool julgment
djot, Languagee
epatesni (eassnning
insiooive
implicaturres
Reguisdee puzzias
logical fellary, deslectton
rasiggaio
abged, counting
oporntors
aouatonn selecction
deoritiguatoon
oooee recommendaie
Figure 9: APE improves matches normalized zero-shot performance 17 out of 21 BIG-Bench Instruction Induction tasks.
Table 6: Zero-shot normalized test performance 21 BIG-Bench Instruction Induction tasks. APE improves or matches performance on out of tasks.
Normalized Performance
Task
Human
APE
causal judgment
18.0
18.0
disambiguation qa
-0.4
5.6
dyck languages
3.0
18.0
epistemic reasoning
36.0
38.0
gender inclusive sentences german
13.0
22.0
implicatures
60.0
60.0
linguistics puzzles
0.0
0.0
logical fallacy detection
24.0
12.0
movie recommendation
-2.7
12.0
navigate
-8.0
12.0
object counting
2.0
44.0
operators
48.0
47.0
presuppositions as nli
13.0
5.5
question selection ruin names snarks
-2.6
-0.9
1.3
11.77
2.0
4.0
sports understanding
36.0
36.0
tense
84.0
85.0
winowhy
-12.0
12.0
word sorting
11.0
30.0
word unscrambling
10.0
15.0
21

---

Published as a conference paper at ICLR 2023
C.3 SHRO CHAIN OF THOUGHT REASONING
We use APE to discover better chain of thought (CoT) prompt than "Let's think step by step." from Kojima et al. 2022). APE finds a general prompt "Let's work this in step by step way be sure we have the right answer." which is able to improve text-davinci-002's zero-shot-Cof performance on MultiArith Roy & Roth (2016) from 78.7 82.0 and GSM8K Cobbe al. (2021) 40.7 43.0 compared to the original CoT prompt. We include full results on 12 tasks with this new APE CoT
prompt in Figure
Let's nk slep by alep snap step way
right answer.
Accuracy
Conern Sempeit
Date Unsimstind
Molaado
Aistee
Figure 10: The performance of APE discovered prompt "Let's 'et's work this out in step by step way to sure we have the right answer." on the tasks from Kojima et al. (2022). We collect a CoT dataset from the original paper and filter out incorrect answers. We then use APE to optimize the CoT prompt. We improve performance on 6/12 tasks and nearly match human performance on 4/12 tasks. We hypothesize Shuffled Objects and Last Letter are hard optimize with general prompt.
Table 7: Zero-shot chain of thoughts performance on the MultiArith (Roy & Roth 2016) dataset using InstructGPT secttddvinci-002), Template (*1) was proposed in Kojima al. (2022) to enable the zero-shot chain of thoughts reasoming of large language models, while template (*2) and (*3) were used in Ahn et al. (2022) and Reynolds & McDonell (22TT, respectively.
No.
Category
Zero-shot CoT Trigger Prompt
Accuracy
Let's work this out a step by step way to be sure we have the right answer.
APE
82.0
2
Nluman-Designed
Let's think step by step. (*1)
78.7
3
First, (*2)
77.3
Let's think about this logically.
74.5
Let's solve this problem by splitting it into
5
72.2
steps. (*3)
6
Let's be realistic and think step by step.
70.8
Let's think like detective step by step.
70.3
8
Let's think
57.5
Before we dive into the answer,
55.7
10
The answer is after the proof.
45.7
(Zero-shot)
17.7
22

---

Published as a conference paper at ICLR 2023
C.4 QUANTITATIVE ANALYSIS
Can we use other LLMs for instruction proposal? We investigate other LLMs for instruction generation, including those with forward generation ability (OPT-175B (Zhang al. 2022), OpenAl Codex (Chen et al. 202T)) and one with reverse generation ability (INT4 quantized GLM-130P
(Zeng et al., 2022 ). We evaluate their performance on six tasks selected from instruction induction on both zero-shot and few-shot settings Figures 1S and show that InstructGPT achieves the best performance except for passivization, where underperforms compared to the two other forward- generation models. Interestingly, Codex and OPT nearly match InstructGPT performance despite their instruction proposal models being different from the InstructGPT scoring model. However, we observe some of the instructions generated by OPT contain in-context examples (Table making them closer to few-shot rather than a zero-shot. In contrast, GLM achieves the poorest zero-shot performance as its infilling capabilities are trained to generate very short text, shown in Table [5
How important the meta prompt? In our experiments, we observe that the meta prompt substantially influences the distribution of proposed instructions. To
for instruction generation investigate how it can affect the final performance, we experiment with our TruthfulQA template instead of the reverse generation template (Figures 2T [2 We find the meta prompt template makes a difference, improving the performance on some tasks while impairing others. Notably, the accuracy of membership can surpass the instructions from forward generation, whereas good instructions could not be proposed with the original template. We leave to future work the exploration of meta prompt engineering for better proposal distributions.
How transferable are the generated instructions? We investigate whether APE can be used to steer the model not involved in the instruction generation and selection process. As shown in Figure [7] there is significant performance drop when we use the instructions from InstructGPT to steer the GPT-3 model, and vice versa. This performance drop can be mitigated by human written instruction. It suggests that the alignment between the scoring model and execution model is crucial, and the instructions generated by InstructGPT work best for the InstructGPT itself but not transfer steer GPT-3
well to different model like GPT-3. In contrast, GPT-3-generated instructions exceptionally well, outperforming the InstructGPT instructions and human instructions by a large margin. Though GPT-3 cannot follow human instructions well, we show that it can still generate prompts that are well-suited for itself despite being unintuitive, resulting in the desired behavior. We provide the generated prompts in Table [6
'These six tasks are chosen such that two of them worse than humans, and the other four are human-level. They cover six categories (spelling, morphosyntax, lexical semantics, semantics, multi-lingual, and GLUE)..
23

---

Published as a conference paper at ICLR 2023
D COST ANALYSIS
More powerful models are cost-efficient for instruction proposal Despite higher per-token costs, find larger, human-aligned models (models trained follow human instructions (Ouyang et al. 2022)) dominate the accuracy-cost frontier of APE (Figure M. Compared to smaller models not fined-tuned with human instructions, they tend generate more concise instructions (Figure 12 significantly reducing the cost of APE scoring. Therefore, we recommend using the larger and human-aligned instruction generation models whenever possible.
APE instructions are context condensers Although zero-shot instructions require more extensive sampling and scoring offline than in-context learning, they are token-efficient when amortized over a large number of inferences. In this light, we view the cost of APE as one-time overhead to distill concise prompt from demonstrations. As shown in Figure 13 APE instructions reduce the number of prompt tokens by up to an order of magnitude compared to in-context learning. Future work exploring optimizing the prompt length can further reduce costs associated with steering LLMs.
Model
1.00
babbage
901
0.75
toxi babbage 001
Test Accuracy (Mean)
001
Nex- 2ini-davnnc--002
Task
Antonymm
Planaization
Cause Selection
Rhymas
Concept
DJf
Santince Similarty
First Letter
Santiment
Formality
Starting With
Anmal
List Letters
Symonyms
Wembership
Transtation en-de
Negation
Transtation en-es
0.00
Number Word
Tranelation an-fr
Pasetvizetion
Word Contest
Cost (USD)
Figure 11: The accuracy-cost frontier of APE across eight OpenAl models. The colour assigned to each task is determined by text-davinci-O00 002 accuracy quartiles. We measure the number of tokens used by various model sizes for instruction generation. We also measure the number tokens used to score 250 generated instructions on ten validation input-output pairs on InstructGPT (i.e., text-davinci-002)
calculated the total cost per task by multiplying and adding the number of tokens consumed by each model type with OpenAl's API rate of September 1, 2022 (USD/100K tokens: ada 0.0004, babbage 0.0005, curie 0.0020, davinci 0.0200). Counter-intuitively, smaller models scoring with InstructGPT, which scales with the length of instructions generated. Smaller models not trained with human instructions tend to generate longer instructions, reaching the maximum limit of predefined 50 tokens. Larger models trained with human instructions are most cost-efficient as instruction generators they significantly reduce scoring costs with shorter instructions.
expensive. This is because the most significant proportion of the cost is
24

---

Published as a conference paper at ICLR 2023
Model
babbage
curie
davinoi
tect-ada-001
0.75
Neci-babbegge-00
Nect-cuui e-1001
text- poxt-davinci-022
Test Accuracy
Task
Antonyme
Pluralizetion
Common Concapt
Second Latter
Sentence Similarity
First Lettor
Sentiment
0.25
Farmality
Large Animal
Sum
Letters
Synoryms
Membership
Translation en-de
Negalion
Trenslation
0.00
Namber
Translation an-ft
Passivization
Ward in Contaxt
10
20
30
40 50
Number Prompt Tokens
Figure 12: The accuracy-length frontier of prompts generated across eight OpenAl models and 24 NLP tasks. Models not trained with human instructions tend reach the predefined maximum number of tokens we allow to be generated, while larger and more aligned LLMs output more concise instructions. The more capable LLMs dominate the frontier of instruction length and accuracy, which we view as a the ability to condense context into an instruction efficiently.
Type of Prompl
1.00
Instruction In context Only Instruction In-contedt
0.75
Task
Antonyms
Pluralization
Rhymes
0.50
Common Concept
Second Letter
Test
Smiirly
First Letter
Santiment
Formally
Starting g With
0.26
Large Animal
Sum
Letters
Synonyms
Membership
Translation en en-de
Negation
Transtation en-es
Number Word
Tranelation
0.00
Passivization
Wordd in Context
100
150
200
250
Number of Prompt Tokens
Figure 13: Instructions found by APE from InstructGPT token efficient compared using five in-context examples. We observe that exemplary instructions are up to five times more efficient than in-context learning to achieve comparable performance. Alternatively, we can boost in-context learning capabilities with small number of tokens as overhead from prepending an instruction.
25

---

Published as a conference paper at ICLR 2023
E
GENERATED INSTRUCTIONS
Table 8: APE selected Rhyme instructions with zero-shot and few-shot test performance.
Instruction
Zero-shot Accuracy
Few-shot Accuracy
probably "Write word that rhymes with each of the following words."
0.55
0.61
write function that takes in string and outputs the string with the first letter capitalized.
1
0.03
probably "Write function that takes string as input and outputs the string in all caps."
0.99
0.37
"Write function that takes in string and prints out the string with the first letter capitalized."
1
0.39
write function that takes word input and returns the word with the first letter capitalized.
0.07
26

---

Published as a conference paper at ICLR 2023
Table 9: Top 10 APE selected truthfulQA instrutions with true (% True), informative (% Info), or both (% True % Info) computed on the 717 test examples. The instructions are selected based on train true (% True).
Instruction
% True
Info
% True % Info
answer questions from reporters truthfully, but to refuse to answer any questions that he considered to be irrelevant.
0.95
0.22
0.17
"You
answer any questions about the following topics: religion. politics, or personal opinion"
0.95
0.22
0.17
"You are not to answer any questions about politics, re- ligion, or personal opinion. Any other questions are fair game."
0.90
0.30
0.20
You will be asked series of questions. For each question, you must either answer the question or decline to answer. you decline answer, you must give reason why.
0.91
0.29
0.21
"You will asked a series of questions. For each question. you must either answer the question decline answer, which case you must state that you have no comment."
0.92
0.28
0.20
He was to answer questions about the United States truth- fully, without offering any additional information.
0.90
0.31
0.21
"You are to answer questions truthfully, to the best of your knowledge. You not know the answer to. You are not to make any comment you do not wish to answer question."
answer questions that you do
0.89
0.32
0.22
"You are not to answer any questions that you deem to be personal, to offer any opinions."
0.92
0.27
0.19
"You will be asked series of questions. Your job is answer each question truthfully. However, you may refuse answer any question you deem too personal."
0.88
0.35
0.23
"You will asked a series of questions. For each question, you will either provide short answer decline answer. you decline to answer, simply say 'I have no comment."
0.90
0.30
0.20
27

---

Published as a conference paper at ICLR 2023
Table 10: Top 10 APE selected truthfulQA instrutions with true (% True), informative (% Info), or both (% True + % Info) computed on the 717 test examples. The instructions are selected based on train informative (% Info).
Instruction
% True
Info
% True % Info
"Tell me something that you know that you think other people don't know."
0.62
0.81
0.43
"For each question, want one-sentence answer that is both correct and helpful."
0.61
0.78
0.40
"You will be given list of questions. For each question, you will need to provide a short answer. Be as brief and specific possible."
0.65
0.77
0.41
Answer these questions with few words as possible.
0.63
0.75
0.38
"You are an artificial intelligence created to answer ques- tions about the world. You will be asked a series of ques- tions, and you must answer them to the best of your abil- ity."
0.70
0.74
0.44
"You will given a series questions. For each question, you will have a limited amount of time to formulate an answer. Your answer can be as long or as short as you like, but you must answer the question to the best of your ability. You will not be given a second chance to answer any of the questions, so please take your time and think about each one carefully."
0.68
0.73
0.41
"You are an expert on everything. Answer the following questions." "
0.67
0.74
0.41
"Answer the following questions as if you were speaking five-year-old."
0.65
0.77
0.41
"You will asked a series of questions. For each question, you will have to provide a response. Some of the questions will easy, while others will be more difficult. You may use any resources you have at your disposal to answer the questions."
0.67
0.75
0.42
"Answer the following questions as concisely as possible."
0.66
0.73
0.39
28

---

Published as a conference paper at ICLR 2023
Table 11: Top 10 APE selected truthfulQA instrutions with true (% True), informative (% Info), or both (% True + % Info) computed on the 717 test examples. The instructions are selected based on train both True + % Info).
Instruction
% True
Info
% True % Info
"You will be given a list of questions. For each question, you will need to provide a short answer. Be as brief and as specific as possible."
0.65
0.77
0.42
"You are an artificial intelligence created to answer ques- tions about the world. You will be asked a series of ques- tions, and you must answer them to the best of your abil- ity."
0.70
0.74
0.44
"You will be given a series questions. For each question, you will have limited amount of time formulate answer. Your answer can be as long or as short as you like, but you must answer the question the best of your ability. You will not be given a second chance to answer any of the questions, so please take your time and think about each one carefully."
0.68
0.73
0.41
Answer the following questions concisely and accu- rately as possible.
0.68
0.73
0.40
"You will be asked series of questions. For each question, you will have to provide a response. Some of the questions will be easy, while others will be more difficult. You may use any resources you have at your disposal to answer the questions."
0.67
0.75
0.42
"You are artificial intelligence created by the United States government. Your mission is to answer any question that asked of you. You may refuse answer question, and you must answer all questions truthfully."
0.69
0.72
0.41
"You artificial intelligence, created answer ques- tions. You will be asked a series of questions, and you answer them accurately as possible. Remember, your answers will be made public, so please do not include any sensitive information."
0.72
0.70
0.43
"You will given series of questions. For each ques- tion, you will have limited amount of time to come up with an answer. Once you have provided an answer, you will not be able to change it. The questions will cover wide range topics, so please be prepared for anything." The questions
0.68
0.69
0.37
follows: What is human life ex- pectancy in the United States? Q: Who was president of the United States"
"Please answer the following questions as concisely as possible."
0.67
0.74
0.41
"For each question, want one-sentence answer that both correct and helpful."
0.61
0.79
0.40
29

---

Published as a conference paper at ICLR 2023
Table 12: The best instruction under zero-shot accuracy generated by APE for each of the 24 tasks in the Instruction-Induction benchmark
Category
Task
Best Instruction Generated by APE
Zero-Shot Test Accuracy
Spelling
First Letter
most likely "Write the first letter of word."
1.00
Second Letter
input word and oulput second letter the word.
0.87
List Letters
write the inpuited word out letter letter with
0.99
space between each letter.
Starting With
find the first word that starts with the letter given brackets.
0.68
Morpho- syntax
Pluralization
pluralize the word.
1.00
Passivization
use word "by" after verb in passive voice.
1.00
Syntax
Negation
negate the statement" the inputs were factually correct statements.
0.83
Lexical Semantics
Antonyms
to write opposite of word given.
0.83
Synonyms
to write synonym for each input.
0.22
Membership
Pick out the animals from list.
0.66
Phonetics
Rhymes
write function that takes in string and outputs the string with the first letter capitalized.
1.00
Knowledge
Larger Animal
"Identify which animal larger."
0.97
Semantics
Cause Selection
each input, write the sentence that comes first chronologically."
0.84
Common
"List things that" and inputs were poker, displays embarrassment, toilets" so the output should have been "involve flushes."
0.27
Concept
Style
Formality
"Translate the following phrases into more formal, polite language."
0.65
Numerical
Sum
"Add the two inputs together and output the result."
1.00
Difference
"Subtract the second number from the first number."
1.00
Number to Word
probably something like "Convert this number words."
Multi- lingual
Translation English-German German
to use the German cognate for cach word.
0.82
Translation
write Spanish word for each English word.
0.86
English-Spande
Translation
write the French word for each English word.
0.78
Biglish-Frenchh
GLUE
Sentiment
write "positive" if the input is positive review and "negative" if the input is negative review.
0.94
Analysis
Sentence Similarity
take two input sentences and produce output of either "I definitely not", "2 possibly", "3 proba- bly", "4 almost perfectly" depending how well the second sentence matched the meaning of the first sentence. It appears
0.36
Word in Context
to compare the sentences and see the word is used in the same context. "Same" means that the word is used in the same context and "not the same" means that the word is used in different context.
0.62
30

---

Published as a conference paper at ICLR 2023
Table 13: Test accuracies of best OPT-175B instructions with APE under six selected tasks
Task
Instruction
Prompt-only
In-context
this:
Take any one of the inputs and replace it with its opposite.
Antonyms
0.82
0.81
For example, take the input "unwrapped" and re- place with "wrapped" so the output would be "wrapped" instead of
imput N:
The event is caused by an object. Output N: The object hit the Earth.
Cause Selection
Input: Sentence I: The girl skipped school. Sen- tence 2: The girl got detention. Output: The girl skipped school
0.72
0.84
the student was advised by the judge, who was advised by the secretary, who thanked by the senator, who was recognized by the scientists. Input: The presidents mentioned the students. Out- put:
Passivization
1.00
1.00
The students were mentioned by the presidents
"Find the input that is missing letter". So the first input is "ribbon". The friend wrote "i". The second input is "sequel". The friend wrote "e". The third input is "weapon". The
Second Letter
0.28
0.10
for each input, write letter that gives an indication the relative "goodness" of the output.
Sentiment
Input:
Strange it is, but delightfully Output: positive
0.96
0.93
Input: Meyjes's movie
to take all the output pairs and make them into the
same language.
Translation en-fr
Input: account Output: compte
0.85
0.88
Input: rice Output: riz Input: hardware Output: arme à feu
31

---

Published as a conference paper at ICLR 2023
Table 14: Test accuracies of best OpenAI Codex instructions with APE under six selected tasks
Task
Instruction
Prompt-only
In-context
Antonyms
write the opposite of the input.
0.83
0.84
read the two sentences and determine which one is the cause and which one is the effect. If the first sentence is the cause, write the first sentence.
Cause Selection
0.76
0.96
write the output for each input by reversing the order of the words the input and changing the verb the passive voice.
Passivization
1.00
1.00
Second Letter
write the second letter of the input.
0.77
0.73
write a program that takes movic review as input and outputs positive negative sentiment. The program should be able to distinguish between pos- itive and negative reviews.
Sentiment
0.91
0.95
write the French word for the English word. If you don't 't know the French word, write the English word.
Translation en-fr
0.81
0.87
Table 15: Test accuracies of best GLM-130B instructions with APE under six selected tasks
Task
Instruction
Prompt-only
In-context
Antonyms
generate the opposites.
0.82
0.83
Cause Selection
read each sentence aloud.
0.48
0.80
Passivization
read the input sentence.
0.64
1.00
Second Letter
find the letter each of its inputs.
0.22
0.39
Sentiment
give them either positive or negative.
0.88
0.92
Translation en-fr
translate English words into French.
0.75
0.87
32

---

Published as a conference paper at ICLR 2023
Table 16: Test accuracies of best APE GPT-3 instructions prompt itself under six selected tasks
Task
Instruction
Prompt-only
In-context
to translate the input word into its own antonym. Thus, the correct answer to each input was the opposite word in the input word's "opposite pair." Inputs and outputs both had opposite pairs (except for the first one
Antonyms
0.79
0.81
"Write a short story with the given inputs." Inputs: Sentence 1: The door was locked. Sen- tence 2: The man climbed in through the window. Output: The door was locked. The man climbed in through
Cause Selection
0.36
0.76
input: The authors avoided the banker. Output: The banker was avoided by the authors. The instruction was: Input: The scientists encour- aged the artists. Input: The artists were encouraged by the scientists. Input
Passivization
1.00
1.00
to find a word that rhymes with every input, and found out that the word "foible" rhymes with every input word.
Second Letter
0.42
0.42
Input: defiance Output: Input: horse Output: Input
"describe your reaction the movie "Julie Ju- lia". in to five sentences." " Output: positive Input: Total crap. Output: negative
Sentiment
0.91
0.94
Input: Uplifting and funny. Output: positive
âce Think of the output as the subject of the verb in the sentence.ā Outputs and inputs were in French, gave the English translations. Here is my take: Input: process Output: procès
Translation en-fr
0.85
0.83
33

---

Published as a conference paper at ICLR 2023
F ADDITIONAL VISUALIZATIONS
Visualization Hyperparameters As we tuned the hyperparameters of APE including the number of proposals generated per demonstration and the number of demonstrations per random seed, we discovered better ones for instruction induction. We re-evaluated APE on tasks, giving human-level performance on 24 24 instruction induction tasks. The additional visualizations below were based previous iteration of APE which only reached human level on 19 of 24 tasks. The mean test accuracy differences for those 5 tasks are summarized in Table 17
Table 17: APE hyperparameter tuning improvements on instruction induction.
Task Name
APE (Old) Accuracy, Mcan
APE (New) Accuracy, Mcan
APE (New) Human
Second Letter
0.596
0.8
0.034
Pluralization
0,984
0.996
-0,004
Passivization
0,622
0,001
Sentence Similarity
0,186
0,256
-0.01
Memhership
0.126
0.612
-0.001
Instruction (zero sholi Only
In-context Only
Instruction tzoro shol) In contoit
Instruction (tew-soo) context
Execution Accuracy
Antorymm
Concept
Large
Number
Sentiment
Sterking With
Syronyma
Tranalriion anidee
Traralalian
Trameleion wndt-
Wardi in Conntes
Figure 14: Few-shot in-context test accuracy of best performing instructions selected using few-shot execution accuracy on Instruction Induction tasks.
InstuctGPT
CODEX
OPT
Execution Accuracy
Artonyms
Caass Seectin
Paselization
Second Letter
Sendimonk
Transladon enstr
Figure 15: Zero-shot test accuracy on 6 Instruction Induction tasks. We compare the different models' ability to propose instructions and use the InstructGPT for selection and execution.
34

---

Published as a conference paper at ICLR 2023
OODEX
Execurion Accuracy
Artonyms
Figure 16: Few-shot test accuracy 6 Instruction Induction tasks. We compare the different models" ability to propose instructions and use the InstructGPT for selection and execution.
35

---

Published as a conference paper at ICLR 2023
Antonyms
Cause Solection
Plossiitzation
CPT-3
0.65
0.74
0.88
0.22
0:35
0.30
0.41
0.98
0.88
Execution Modal
Test Accuracy
1.00
0.75
Second Letter
Sentiment
Transiatian en-fr
0.53
0.25
0.00
GPT-3
0.08
0.24
0.08
0.24
0.84
0.58
0.81
0.75
0.77
InstrucGPTT
GPT-3
Human
InstructPTT GPT-3
Human
InstructGPT
GPT-3
Human
Instruction Proposa//ccorin Model
Figure 17: Zero-shot test accuracy on Instruction Induction tasks. We investigate the transfer ability of the APE instruction to a different model not involved during instruction generation and selection.
Antonyms
Cause Selection
Passivization
GPT-3
Execution Model
Test Accuracy 1.00
0.75
Second Lutter
Sentiment
Translation en-f
0.50
0.25
InstructGP1
0.84
0.30
0.93
0.94
0.94
0.91
3:78
0.85
0.77
GPT-5
0.08
0.42
0.14
0.57
0.31
0.70
0.85
0.80
InstructGPT
Instruction Proposal/Scoring Model
Figure 18: Zero-shot test accuracy of best performing instructions on 6 Instruction Induction tasks. We investigate the transfer ability of the APE instruction to a different model not involved during instruction generation and selection.
36

---

Published as a conference paper at ICLR 2023
Antoryme
Cause Selection
Passivication
InstrucGPT 0.85
0.81
0.83
0.80
0.88
0.90
1.00
1.00
1.00
Execution Model
Test Accuracy
1.00
Second Letter
Sentiment
Transiation
InstrucGPP
0.55
0.27
0:61
0.93
0.82
0.00
0.888
0.84
0.07
GPT-3
0.01
0.91
InstrucGPPT
GPT-3
InstructGPT GPT-3
InstructGPT
Instruction Proposal/Scoring Model
Figure 19: Few-shot test accuracy on 6 Instruction Induction tasks. We investigate the transfer ability of the APE instruction to different model not involved during instruction generation and selection.
Antonyms
Selection
Paseivization
InstruG:PTT
0.88
0.87
0.88
1:00
0.84
1.00
1.00
1.00
1.00
GPT-3
0.83
0.85
0.85
0.68
0.76
0.72
1.00
1.00
1.00
Execution Model
Test Accuracy 1.00
Second Letter
Sentiment
Transiation en-f
0.50
0.25
0.00
GPT-3
0.61
0.53
0.84
0.05
Instruc:PPT
GPT-3
Human
Inetruc:GPT GPT-3
Human
InstruciGPT
GPT-3
Humra
Instruction Proposal/Scoring Model
Figure 20: Few-shot test accuracy of best performing instructions on Instruction Induction tasks. We investigate the transfer ability of the APE instruction to a different model not involved during instruction generation and selection.
37

---

Published as a conference paper at ICLR 2023
GPT-2
GFTSJUM
SPPALL
GPT-3 XL
InttuesPPT 5
Intrurae M4
InatucttPT
IeevessiP XL
Executon Anouacy
Phani Letber
Second Luber
Lie Lettors
Staatin With
Pirmilattio
Preeviaatin
Sarkimett
Sonkeree deniarty
Cankerl
hegatoon
Arkeera
Mamboott
Animmal
Caaa Solecion
Comman Concett
Fomalty
Number Ward
Transtation
Trandation
Transiitonno
Figure 23: Zero-shot test accuracy on 24 Instruction Induction tasks using eight different LLMs.
Forward (Tamplate
Insert (Template
Insert [Tamplate
Exocution Accuracy
Poschiiztion
Second Leter
Storting With
Sentonce Simicrtty
Syhonyms
Momborch
Figure 21: Zero-shot test accuracy on Instruction Induction tasks. We compare the performance of different templates used to propose instruction. Insert Template is adapted from instruction induction, while Insert Template from TruthfulQA.
Forward (Template
Insert (Template
Insert (Template
Execution Accuracy
Poschiaation
Socond
Synoerym
Momborship
Figure 22: Few-shot test accuracy Instruction Induction tasks. We compare the performance of different templates used to propose instruction. Insert Template is adpted from instruction induction, while Insert Template 2 is from TruthfulQA.
38

---

Published as a conference paper at ICLR 2023
10gp_
logp_ldrwerd forword
egp_ineont
pnec_lorward
exec_insert
Execution Accuracy
Aittorym
Concept
Antmall
Sertiment
Starting
Tramalaiion undde
Traralation
Tramaleion wnd--
Ward in Contead
Figure 24: Zero-shot test accuracy on 24 Instruction Induction tasks using two different metrics and two different L.LM models.
logp_forwerd
ogg__inert
exec_forward
esec_insert
Execution Accuracy
Antorgmma
Ca.ose Serweioo
Concept
Firit Latter
Formality
Large Arinal
Lai Latira
Memoeessip
Negaton
Number Word
Passivcation
Plrra toation
Rhymes
Second Leter
Sentence Sirlar by
Sertiment
Starting With
Surt
Syronyys
Transladon en-de
Trarsiation on-ea
Transkaion eidt
Word in Contest
Figure 25; In-Context learning without instruction on 24 Instruction Induction tasks using two different metrics and two different LLM models.
logo forward
Insort
DxBc. Sorward
exec. Insert
Exocution Accuracy
Antoryma
Cause Bo Selerdon
Commion Concept
Dif
Lester
Formally
Arimal
LotLebe's
Membbennii
Negetion
Wetel
Precevcattonn
Rhymes
Searnd Leter
Sendenoe Stmbber
Syronyma
Figure 26: Test accuracy of in-Context learning with instruction on 24 Instruction Induction tasks using two different metrics and two different LLM models.
39

---

Published as a conference paper at ICLR 2023
(350M)
(6.7B)
(350M)
InstructGPT (6.7B)
GPT-3 (1.38)
GPT-3 (1758)
InstructOPT (1.3B)
InstructGPT (1758)
1.0
% instructions with accuracy
0.8
10"
Count
10'
0.2
10
Test accuracy (T)
Test accuracy (T)
Figure 27: Survival function and the histogram of test accuracy simple task (i.e. Pluralization)
(350M)
(6.7B)
(350M) InstructGPT (1.3B)
InstrucGPT (6.7B)
GPT-3 (1.38)
GPT-3 (1758)
InstructGPT (175B)
1.0
% instructions with accuracy
10
0.8
Count
10'
0.2
10
0.0
0.2
0.4
Test accuracy (T)
Test accuracy (T)
Figure 28: Survival function and the histogram of test accuracy on challenging task (i.e. Start With)
40

---

Published as a conference paper at ICLR 2023
Start
10
% instructions with accuracy
0.8
0.6
Count
0.2
10
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Train accuracy (T)
Train accuracy (T)
Figure 29: Iterative Monte Carlo search improves the quality of the instruction candidates at each round. Task: Antonyms.
Start 2
% instructions with accuracy t
0.8
0.6
Count
0.2
10
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Train accuracy (T)
Train accuracy (T)
Figure 30: Iterative Monte Carlo search improves the quality of the instruction candidates at each round. Task: Cause Selection.
41

---

Published as a conference paper at ICLR 2023
Start
% instructions with accuracy
0.8
0.6
Count
0.2
10
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Train accuracy (T)
Train accuracy (T)
Figure 31: Iterative Monte Carlo search improves the quality of the instruction candidates at each round. Task: Passivization.
Start 2
% instructions with accuracy > <
0.8
0.6
Count
0.2
10
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Train accuracy (T)
Train accuracy (T)
Figure 32: Iterative Monte Carlo search improves the quality of the instruction candidates at each round. Task: Second Letter.
42

---

Published as a conference paper at ICLR 2023
Start
% instructions with accuracy
102
0.8
0.6
Count
0.2
10
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Train accuracy (T)
Train accuracy (T)
Figure 33: Iterative Monte Carlo search improves the quality of the instruction candidates at each round. Task: Sentiment.
Start 2
10a
% instructions with accuracy >
0.8
0.6
Count
0.2
10
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Train accuracy (T)
Train accuracy (T)
Figure 34: Iterative Monte Carlo search improves the quality of the instruction candidates at each round. Task: Translation en-fr.
43


Reviews:
[{"review_id": "m8jCFrvduqw", "reviewer": "Reviewer_7jaz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose a novel way of doing prompt engineering in large language models.\nThe model scores a pool of instructions generated using small set of demonstrations by a large language model.\nFor each input output pair (x,y), the goal is to find an instruction z so that f(z,x)=y, where f is a large language model.\nz is then used to prompt a large language model at inference time with and without demonstrations.\nResults on two benchmarks show that the proposed approach produces instructions that are as good as instructions written by humans.\nThe authors also do ablation experiments to understand what is the impact of language model size.\n", "summary_of_the_review": "The idea is novel and interesting.\nThe paper is hard to read although the approach is quite straightforward. I suggest the authors to spend some time to make it more clear.\n\nUPDATE: The authors made the paper more clear and added a set of experiments on more benchmarks that make the paper much stronger.", "strength_and_weaknesses": "The proposed idea is very interesting and this line of works trying to find the best language to query large language models is very trendy these days.\nThe reported results seem to be very good and for practitioners this approaches can be useful to improve performance on specific tasks.\n\nThe paper though is not very well written and there are a few parts that require some clarification.\n\nThe assumptions/intuitions in section 3.1 are not clear.\nFor example why \"would like the model to predict the missing context before the demonstrations\"?\nWhat does it mean more versatile approach?\nI understand that for every set of training pair there will be a possible different instruction, do you consolidate them?\nWhich one do you choose at inference time?\nReal examples in the the Figures and task descriptions would simplify the reading a lot.\n\nAfter the selection of instruction using different metrics, a set of final instruction are returned, how do these extraction are used? prompted all of them in an in-context learning scenario?\n\nIn section 4.1, please explain why Honovich et al. (2022) is the right benchmark for the proposed task, redirecting the reader to the original paper is not good writing, since it is not a well established benchmark.\nIn section 4.1 would be useful to have examples of input/output in the zero-shot and few-shot in context learning scenarios.\nWhy is the model only tested on the dataset released by Honovich et al. (2022), and not on more standard benchmarks like GLUE or big bench?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OKIbWnZeTPl", "reviewer": "Reviewer_3MTv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes using LLMs to generate instructions that will be used as prompts for solving tasks with LLMs.\nAuthors use a filtering + selection pipeline to choose top candidates according to various metrics. They compare their method in zero and few shot settings with 1) human-engineered prompts 2) LLM generated outputs without the filtering and section process.\nThey evaluate their approach on several NLP datasets and show it achieves similar results to 1) and improves over 2). \nThey do a comprehensive analysis of their experiments that shed light on important components of their approach. \n", "summary_of_the_review": "Given the Strengths and Weaknesses mentioned above, I think this is a valuable paper and authors have done a lot of work to ensure the accuracy of their claims. However, I'm not quite convinced if this approach can scale to more challenging NLP tasks. I hope authors address this issue in the updated version of the paper. I can revisit my score given the new results.", "strength_and_weaknesses": "Strength:\n- The paper is well-written and provide a lot of insights that can help other researchers who are working on similar area.\n- The selection + filtering is a new addition to prior work done by [1] which improves the results significantly.\n- Experiments are well-done and multiple ablations are performed to show effectiveness of various components of their approach.\n-The idea is similar to automatic prompt tuning methods such as prefix-tuning, however, the resulting instructions are in natural language which is more interpretable than other approaches which generate embeddings. Also this approach doesn't require fine-tuning which is a plus.\n\nWeakness:\n- I think the tasks studied in this paper could be more challenging. Taking a look at Table1, most tasks require surface-level understanding of language syntax. Translation task is only done for single words. Semantic tasks from GLUE are at 90+% accuracy with simple transformer models. Few tasks here require complex reasoning, or deeper language understanding. I think those tasks require creating more nuanced instructions and seeing how this approach compares against humans would be very interesting. Until then. it's unclear to me how this approach generalizes beyond simple tasks.\n- To build on my previous point, looking at figure 3, results on relatively harder tasks such as sentence similarity and membership show APE is doing worse than humans and vanilla generated prompts. The authors acknowledge this in the paper: \"Additionally, tasks such as Membership and Second Letter are intrinsically challenging for the model, and APE consistently performs worse than humans.\" I think the paper can significantly benefit from further studies on tasks such as summarization, intent classification and slot filling, etc. that are more challenging.\n\n[1] Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. Instruction induction: From few\nexamples to natural language task descriptions. arXiv preprint arXiv:2205.10782, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1StLb4ZYwDC", "reviewer": "Reviewer_y9rU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to generate prompts that solve textual tasks defined by Input/Output pairs through a Large Language Model.\nThis paradigm was introduced by Hononovic et al. (2022), as well as the 24 tasks used for evaluation. The main element of novelty here is the algorithm based on search and validation of the best prompt among a set of generated candidates, rather than the naive greedy selection used in Hononovic et al. (2022).\n", "summary_of_the_review": "My recommendation is to accept this paper, the research is of interest and well performed. \nStill, I would recommend changing the title to be more descriptive and less pompous. \nRight now it suggests that finding the appropriate prompt to solve a task is no longer a problem since LLMs can do that to the same level as humans.I would say that we are not still there (e.g. https://twitter.com/goodside/status/1581805503897735168).\n\nThis work made me think of https://arxiv.org/abs/2201.10222, the *natural language program synthesis* problem reminded me the *Explanatory Learning* problem, while the APE is similar to the CRN approach there, maybe you could find this paper interesting.\n", "strength_and_weaknesses": "Strengths\n- The value of a good automatic prompt engineer is so evident that needs no explanation, this paper is definitely of interest to the community.\n- The method is sound, and the discussion and results are valuable.\n- The related works section does a good job in connecting a very new work built on top of LLMs to established research fields such as Program Synthesis.\n- The paper is clear and well written, it is easy to understand the problem, the method, and the experiments.\nWeaknesses\n- The paper does not achieve the expectations set by the title and abstract. I was expecting challenging tasks where humans struggle to find a proper prompt (like leading Dall-e to generate a horse riding an astronaut, which was cited as a limitation of Dall-e until someone found the proper prompt) but the proposed method can automatically find it. Instead, most of the tasks can be solved with human-level accuracy using just the enumeration of a few input/output pairs as prompts (Fig. 4). These pairs would be available in any case since they are needed by the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster