PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization
Authors: Xinyuan Wang
Chenxi Li
Zhen Wang
Fan Bai
Haotian Luo
Jiayou Zhang
Nebojsa Jojic
Eric Xing
Zhiting Hu
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Expert-level Prompt Optimization
Strategic Planning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: PromptAgent introduces a strategic planning approach to autonomously generate expert-level prompts for advanced LLMs outperforming existing methods across diverse tasks by strategically exploring the prompt space and leveraging error insights.

Abstract: Expert-level prompts, carefully engineered by human experts who have a deep understanding of both large language models (LLMs) and domain knowledge, are the future of prompting and pivotal to harnessing the full power of advanced LLMs. Discovering such prompts with an automated process remains a sought-after and unresolved challenge. Existing prompt optimization techniques, though automated through iterative sampling, often fall short in injecting domain knowledge and exploring the vast prompt space for complex expert-level prompts efficiently. To address this pressing need and achieve expert-level prompting, we introduce PromptAgent, which autonomously discovers prompts equivalent in quality to those handcrafted by experts. At its core, PromptAgent views prompt optimization as a strategic planning problem and employs a principled planning algorithm (rooted in Monte Carlo Tree Search) to strategically explore the vast expert-level prompt space. PromptAgent interacts with the LLM in a human-like trial-and-error manner during the planning, and injects expert-level knowledge by reflecting on model errors and generating insightful error feedback. This novel formulation allows it to iteratively evaluate intermediate prompts, refine them based on errors, simulate future rewards, and search for high-reward paths leading to expert-level prompts. We apply PromptAgent to 12 tasks spanning three practical domains: BIG-Bench Hard (BBH), domain-expert, and general NLU tasks, showing PromptAgent consistently outperforms strong prompting and prompt optimization baselines by great margins. Our qualitative analysis further emphasizes PromptAgent's capability to distill insightful errors into expert-level prompts.

Published as a conference paper at ICLR 2024
PROMPTAGENT: STRATEGIC PLANNING WITH LARGE LANGUAGE MODELS ENABLES EXPERT-LEVEL
PROMPT OPTIMIZATION
Xinyuan Wang' 'Zhen Wang12=1 Chenxi Li Fan Bai Haotian Luo
Jiayou Zhang Nebojsa Jojic Eric Xing²1 Zhiting Hu!
UCC San Diego *Carnegie Mellon University
Miicosoott Research Georgia Institute of Technology Mohamed bin Zayed University of Artificial Intelligence {xiw136, zhw085, zhh019 @uccd. edu
ABSTRACT
Highly effective, task-specific prompts are often heavily engineered by experts integrate detailed instructions and domain insights based on a deep understanding of both instincts of large language models (LLMs) and the intricacies of the target task. However, automating the generation of such expert-level prompts remains elusive. Existing prompt optimization methods tend overlook the depth of do- main knowledge and struggle to efficiently explore the vast space of expert-level prompts. Addressing this, we present PromptAgent, an optimization method that autonomously crafts prompts equivalent in quality to those handcrafted by ex- perts. At its core, PromptAgent views prompt optimization as strategic planning problem and employs principled planning algorithm, rooted in Monte Carlo tree search, to strategically navigate the expert-level prompt space. Inspired by human- like trial-and-error exploration, PromptAgent induces precise expert-leve insights and in-depth instructions by reflecting on model errors and generating constructive error feedback. Such novel framework allows the agent to iteratively examine intermediate prompts (states), refine them based on error feedbacks (actions), sim- ulate future rewards, and search for high-reward paths leading expert prompts. We apply PromptAgent to 12 tasks spanning three practical domains: BIG-Bench Hard (BBH), as well as domain- sppccfc and general NLP tasks, showing it sig- nificantly outperforms strong Chain-of- Chain-of-Thought and recent prompt optimization baselines. Extensive analyses emphasize its capability to craft expert-level, de- tailed, and domain-insightful prompts with great efficiency and generalizability".
INTRODUCTION
Prompt engineering aims to craft effective prompts for harnessing the full potential of large language models (LLMs). Recent automatic prompt engineering, i.e., prompt optimization, has successfully studied training soft prompts (Lester et al., 2021; Hu et al., 2021; Wang et al., 2022), or searching for optimal combinations of discrete tokens (Shin et al., 2020; Deng et al., 2022; Zhang et al., 2022), by utilizing internal states gradients of LLMs. For cutting-edge, proprietary API-based LLMs like GPT-4 (OpenAI, 2023b), prompt engineering largely relies on somewhat ad-hoc human- machine interactions. Human prompting experts thus need a unique blend of domain knowledge and intuition for LLMs to design the most effective prompts. For instance, an ideal prompt from human experts, shown in Figure 1, might integrate nuanced elements like task descriptions, domain knowledge, solution guidance, etc., all of which substantially boost prompt quality and performance. Automating expert-level prompt engineering on API-based LLMS presents significant challenges, largely due to the intricate nature of expert-level prompts, illustrated in Figure 1. Although re- cent prompt optimization approaches have begun to utilize techniques like iterative sampling or
'Equal contribution
CCorresooning author
"ode and demo available https: / wtt.u//cw....iiaaaaaagg//ooottea

---

Published as a conference paper at ICLR 2024
Biomedical Task Input
Linkage studies in this family suggested a close linkage between the s2 deficiency gene and genes coding for B18, Dw2, and BIS antigens.
'c22 cefizsinyy is disease mention to ba edtractdd
Ordinary User Prompt
Prompt From Sampling- Baaed Method
Extract the disease condition from the sentence, if any is mentioned.
any disease or condition mentioned in the sentence, extract it.
Expert-level Prompt
Task Description
You're tasked with extracting diseases conditions from the given sentence
Domain Knowledge
Avoid associated elements: inherilance patters,. genes gene (ike PAH)
Solution Guidance
Consider both specific diseases and broader categories, common abbreviations
Exception Handling
The term 'locus' should be recognized as genomic location. not a disease name
Output Formatting
Provide the identified diseases this format: (entit_1.enning 1.enlity ..)
Ordinary User/Sampled Prompt Output
Expert Prompt Output
deficiency gene
c2 deficiency
Figure 1: Expert-level prompt vs. ordinary human -written prompt and prompt from sampling mansiing-based methods (i.e., Automatic Prompt Engineer, Zhou al. (2022)). The task is in the biomedical domain for extracting disease entities (NCBI, Doğan et al. (2014)). The expert prompt provides much richer domain-specifie details and structured guidance than the other two, leading to the correct prediction.
evolutionary algorithms, such as Monte Carlo search (Zhou et al., 2022) or Gibbs sampling (Xu et al., 2023), they mostly employ heuristic methods like text edits or paraphrasing for generating candidate prompts (Zhou et al., 2022; Prasad et al., 2023). These approaches also often rely on straightforward iteration algorithms and lack a principled strategy to guide the exploration. Con- sequently, they tend to settle on local variants of prompts from ordinary users and rarely ascend to the excellence and nuances of expert-level prompts. Critically, many of these methods overlook that prompting engineering is essentially a saman-in-thh-loop application. In this process, humans refine prompts by fixing intermediate errors and integrating necessary domain knowledge through itera- tive interactions. This iterative refinement process characterizes the merits of how human experts craft superior prompts. Yet, the challenge remains that human exploration, while effective, can be expensive and less efficient at handling multiple errors simultancously to explore the prompt space, thereby impeding the scalability of expert-level prompting.
To address the above challenges and combine human-like explo- ration with machine efficiency, we introduce PromptAgent in this paper. Drawing inspiration from human trial-and-error processes, PromptAgent seamlessly incorporates the principled planning ap- proach, specifically Monte Carlo Tree Search (MCTS), to strategi- cally optimize the prompting process. Notably, PromptAgent refor- mulates prompt optimization
Human PromotAgent
Avg. Performance
0.4
strategic planning problem to ad- dress the complexity of expert-level prompt space. Under this plan- ning framework, it plays triar-and-error iteration to retrieve model errors and leverages the self-reflection ability of LLMS (Jang, 2023; Shinn al., 2023; Pan et al., 2023) to generate insightful error feedback. This feedback, in turn, plays a critical role in effec- tively inducing domain knowledge and guiding towards in-depth prompts. Through strategic plan- ning, PromptAgent iteratively leverages insightful error feedback (action) to refine each version of prompts (state). Starting from an initial prompt (state), PromptAgent systematically grows the prompt space in a tree structure and prioritizes high-reward traces to navigate the vast space of expert-level prompts. The principled MCTS planning allows PromptAgent to look ahead and sim- ulate future rewards, which are then backpropagated to update the beliefs about the current prompt so that PromptAgent can explore more promising alternatives later.
0.2
0.0
GPT-3.5
GPT-4
PaLM
Figure Prompt comparison across differennt base models.
We demonstrate that PromptAgent can discover productive expert-level prompts by applying to 12 tasks spanning three practical and distinct domains: BIG-Bench Hard (BBH) (Suzgun et al., 2022), as well nomain-specific and general NLP tasks. Starting with initial mimannwritten prompt and a small set of training samples, PromptAgent not only enhances the performance of the initial human prompt greatly but also significantly surpasses strong Chain-of- Thought (CoT) and recent prompt
2

---

Published as a conference paper at ICLR 2024
(a) MCTS Planning for Prompting
(b) State Transition
St Current Prompt: Please dotermina whother one sentance entals the next
80
Step1: Retrieve Errors from Base Model
To
Willamm learns that kids play in water comring up in streams out of tild floor with image of large rose on Hypothesis: William lowms that kits are paainn in water.
Premise
ag
81
84
Label: Non-ertariment Prediction: Entsiment
at
Step Generate Error Feedback (Action)
T1
a41
Meta-prompt 1: Summarice and suggest improvements
a1
Error Feedback: lignorin Contedt Datel-The model might overlooking the details of the premise lids play in water coming up in streams out of Hed floo with an imaee of a largo rase on it., which directly imples the hypothesis..
2
95
T2
as
Step 3: Update Prompt (State)
a22
Meta-grompt 2: Given the feedback, give beiter
83
86
New Prompt: Campara provided sentancas ... Take account subileties in the context, pirpoint the order of events and differentiaee belween fact and aesumptions, If the hypothesis is direct result of the premise, select 'entallment
S
Figure 3: (a) MCTS (Monte Carlo Tree Search) planning for expert-level prompting. The tree struc- ture enables strategic planning for PromptAgent. (b) A simplified state transition example. Given current state (prompt). the base model (gpt-3. 3.pt-3.5-turtee 5-turbo) collects errors from the task dataset. The optimizer model (gpt-4) provides eπror feedback accordingly. The optimized model then updates the prompt according to the feedback and transits to the next state.
optimization baselines. For instance, Figure 2 shows PromptAgent consistently outperforms human and Automatic Prompt Engineer (APE) (Zhou et al., 2022) baselines across 6P5-3.5, GPT-4, and PALM 2, yielding improvements by 9.1 7.7% and 6% over APE, respectively. Extensive qualitative results further highlight the expert-level aspects of optimized prompts. indicating that PromptAgent effectively bridges the domain gap in challenging tasks, offering great exploration efficiency and generalizability. As we anticipate the emergence of even more powerful LLMs that can understand intricate instructions, we believe that expert-level prompting will spearhead the next era of prompt engineering, where PromptAgen stands as a pioneering step in this research direction.
METHODOLOGY
Given a base LLM and a target task T, the job at hand for prompt engineer to craft an op- timized natural language prompt pd that maximizes the performance of B on T. However, the gap between novice and expert prompt engineers can be significant, particularly for tasks demand- ing specialized domain expertise, such in the biomedical domain. Our primary objective is to autonomously refine the task prompt 'P7 to bridge this knowledge gap, minimizing human inter- vention. Most existing approaches rely sampling local prompt alternatives iteratively, which is not only resource-intensive but also lacks assurance of yielding an optimal final prompt. In light of this, we introduce PromptAgent, an agent-based framework produce expert-level task prompts via strategic planning and reflecting with error feedback during the prompting process, striking proper balance of exploration and performance.
Problem formulation. Adopting a standard setting in prompt optimization (Zhou et al., 2022), we start with an initial natural language task prompt Po (c.g.. "Let's solve this problem step-by-step") and a small set of training samples from target task T as (Q, A) = {qi a, ) where 9:/a; are input/output pairs for each sample (e.g., question and its answer). Given the model input consisting of P and gi the base LLM B makes the prediction (typically through left-to-right generation process) based on (a (q:, P)2. The goal of prompt optimization is to find the optimal natural language prompt P that maximizes the performance towards measure function R (e.g.. accuracy). This can be formally defined as an optimization problem: p = arg maxpes E, R(ps (a, qq. P)), where S denotes the sample space for a natural language prompt, an infinite and intractable space, if not impossible, to comprehensively enumerate. Conventionally. human experts draw upon a blend of heuristics and dopain-specific insights craft such prompts. Although previous optimization methods have attempted to leverage iterative sampling methods for prompt discovery (Zhou et al., 2022), advance this line of research by proposing unified framework that seamlessly integrates
Note this is traditionally a zero-shot setting focus on, where task prompt excludes any training samples.
3

---

Published as a conference paper at ICLR 2024
strategic planning for superior, expert-level prompt optimization. Next, we introduce the formulation of PromptAgent and then present the planning- based prompt optimization.
2.1 PROMPTAGENT FRAMEWORK DESIGN
The goal of PromptAgent to inject expert-level prior into the task prompt while maintaining effi- cient exploration via strategic planning. For planning, the state is represented by each version of task prompt/ P. and the actions need to encapsulate the potential modifications over the current prompt. Each action could represent, for instance, editing certain words or phrases or paraphrasing P (Jiang ct al., 2020; Prasad ct al., 2023). However, to make the action space more meaningful and trigger prior knowledge, we instead propose a different action space. As shown in Figure 3 (b), the action is defined as an error feedback representing the direction of further refining the current prompt (state). Specifically, PromptAgent generates the action in two steps: collecting errors from training samples (Step 1) and reflecting on errors to draw useful lessons (Step 2) that potentially improve the prompt P. Our action generation is also inspired by the recent self-reflection of LLMs (Pryzant et al., 2023).
Given the definition of state and action, PromptAgent formally models the prompt optimization as a Markov Decision Process (MDP) by the tuple (8, A., rr) in which is the state space, AA the action space, T' is the transition function T'; S + S, and r is the reward function r SXAA + R. Given a current state St, PromptAgent generates an action 0₂ based on a₂ ~ po(aas,, m2), where my is a prompt working on another optimizer LLM O to help the action generation. The action generation is operationalized by Step 1&22 in Figure (b). Then, PromptAgent obtains a new state based on the transition function (80+ 182, (z,, m2), where m2 is another prompt helping the state transition and update the prompt, also working on the optimizer LLM O. More specifically, given the action de as the error feedback from Step 1&22 PromptAgent asks the optimizer to generate a new prompt considering previous states and error feedbacks. Finally, to assess the quality of each new state 8, after applying action a,, PromptAgent straightforwardly defines reward function rs = r(s,, 0z) as the performance on a held-out set separated from the given training samples. The exact definition of reward will depend on task-specific metrics.
2.2 STRATEGIC PLANNING FOR PROMPT OPTIMIZATION
The aforementioned reformulation of the prompt optimization enables us to seamlessly integrate PromptAgent with principle planning algorithms, notably the Monte Carlo Tree Search (MCTS). This enables strategically navigating the vast prompt space while balancing the exploration and exploitation in finding high-reward paths of error feedbacks, which leads to the most generalizable expert-level prompts. Specifically, we observe some error feedbacks (actions) may inject instance- specific details into task prompts (states) that are hard to generalize task-wise (exploitation), where we need strategic planning explore novel error feedbacks for higher rewards (exploration). MCTS operationalizes such strategic planning, as shown in Figure 3 (a), by progressively constructing a tree structure with each node as a state and each edge as the action for transiting states. MCTS expands the tree strategically by maintaining a acttinaction value function, Q S x A + R. which represents the potential future rewards for applying an action de to state 8t. In other words, we rely on this function, Q(*t: ae), to look ahead and estimate the potential rewards for paths following the current state-action pair. To update this Q function and expand the tree, MCTS iteratively performs four operations: selection, expansion, simulation, and back pack-proppgatton. The iteration process ends when a pre-detined number of iterations is reached, and we then select the highest-rewand trace for the final prompt. We next explain the four operations in PromptAgent, and the pseudocode of our MCTS-based prompt optimization can be found in Algorithm of the Appendix.
Selection is the first step that selects the most promising nodes at each level to be further expanded and explored. At each iteration, it starts from the root node 8p, traverses through each tree level, selects subsequent child node at every level, and stops leaf node. When selecting the child node at each level, we leverage the Upper Confidence bounds applied to Trees (UCT) algorithm, which well-known for balancing the exploitation (choosing high-value nodes) and exploration (choosing less- nodes) as follows:
(Q(SL a/) + +c V M(ch(s(,,))
(st))
af = arg max
(1)
A(e)(
where A(st) is the action set for node st, (s(t) is the number of visiting times for node st, ch(s. a) represents the child node for s, after applying action of and is a constant to adjust the exploration. As we can see, the first term signifies exploitation by the Q value, and the second term indicates
4

---

Published as a conference paper at ICLR 2024
exploration, measuring the uncertainty for less visited nodes. In other words. if node was less explored and its child node was less visited before, the second term will be higher.
Expansion grows the tree by adding new child nodes to the leaf node reached by the previous selection step. This is done by applying the action generation and state transition (Figure (b)) multiple times, resulting in multiple new actions and states. Note that may sample multiple training batches to derive diverse error feedback (actions). Within new nodes, we then send the highest-reward one to the next simulation step.
Simulation is the lookahead step to simulate the future trajectories for the selected node from the previous expansion step. This step usually comes with playout policy to reach the terminal state quickly and calculate the future rewards. The choice of playout could be flexible, such as choosing random moves until the terminal. To reduce the computation cost of simulation and simplify the pro- cess, we perform the previous expansion step iteratively until the terminal, i.e., we keep generating multiple actions and selecting the highest-reward node to proceed to the next tree level.
Back- proppproppggtion happens when a terminal state is met during the simulation. The terminal state usually defined when a pre-defined maximum depth is reached, or an early-stopping criterion is encountered. We then back-propagate the future rewards along the path from the root to the terminal node by updating the Q value function. Specifically, for each state-action pair the path, Q(*;; 0t) is updated by aggregating the rewards from all future trajectories starting from 8 as follows:
M
('(,
Σ
Σ
r(d',aa)
M
where M is the number of future trajectories starting from 8t, Sy and Al, represent the j-th state and action sequences starting from s and ar, respectively.
PromptAgent executes the above four operations with a pre-defined number of iterations to stabilize the Q values and fully grow the tree for exploring the vast prompt space. We finally need to select the best trace and node (i.e., prompt) for the final evaluation. Multiple alternative solutions can be leveraged for this output strategy, e.g., one could opt for the best node in the best path with the highest reward, or directly choose the leaf node with the largest number of visiting times. For simplicity and empirical purposes, we use the first strategy to select the output prompt, which works the best in our experiments.
EXPERIMENTS
Tasks and Datasets. We curate 12 tasks from three domains for thorough experiments: 6 BIG- Bench Hard (BBH) (Suzgun et al., 2022); 3 domain- expeet tasks, including NCBI (Dogan et al., 2014), Biosses (Sogancioglu et al., 2017), and Med QA (Jin et al., 2021); and 3 general NLU tasks, including TREC (Voorhees & Tice, 2000), Subj (Pang Lee, 2004), and CB (De Marneffe et al., 2019). More task and dataset details are in Appendix A.
Baselines and Implementations. We compare PromptAgent with three types of baselines: human prompts, Chain-of- Thoughh (CoT) prompts, and recent prompt optimization methods: (1) Human prompts are numan-designee instructions usually originating from the datasets. We have a few- shot version of human prompts with examples from Suzgun al. (2022) for BBH tasks and ran- domly sampled examples from the training set for others. (2) CoTT prompts are Chain-of- ThoTuht prompts with reasoning steps. We the CoT prompts from Suzgun et al. (2022) for BBH tasks and construct CoT prompts for other tasks. We uses "Let's think step by step" for CoT (ZS) (Kojima al., 2022). (3) Prompt optimization methods include GPT Agent and Automatic Prompt Engineer (APE) (Zhou et al., 2022). GPT Agent represents the recent surge of interest in LLM-powered au- tonomous agents (Weng, 2023), such as Auto-GPT 3, Such agents are expected to autonomously per- form planning and self-reflection to solve most human requests, including optimizing task prompts. We then leverage one of the powerful ChatGPT Plugins (OpenAI, 2023a) with GPT-4, A/ Agents for prompt optimization. Lastly, APE (Zhou et al., 2022) is one of the most recent prompt optimiza- tion methods that proposes a Monte Carlo search-based method iteratively propose and select
hom//Sinnificante tasavittss/
'nttps://aiagenta com/
5

---

Published as a conference paper at ICLR 2024
Table 1: Prompting performance on BBH tasks. ZS: Zero-Shot, FS: Few-Shot. We select six chal- lenging tasks from BBH (Suzgun et al., 2022), requiring domain knowledge (e.g., Geometry) or reasoning (e.g., Causal Judgement). Our method outperforms 5/6 tasks, with only CoT surpass- ing in Object Counting. On average, our accuracy exceeds others by at least 9%.
Penguins
Geometry
Epistemic
Object Count.
Temporal
Causal Judge.
Avg.
Human (ZS)
0.595
0.227
0.452
0.612
0.720
0.470
0.513
Human (FS)
0.595
0.315
0,556
0.534
0.408
0.620
0.505
CoT (ZS)
0.747
0.320
0.532
0.542
0.734
0.610
0.581
CoT
0.747
0.540
0.720
0.960
0.626
0.650
0.707
GPT Agent
0.696
0.445
0.406
0,502
0.794
0,520
0,561
APE
0.797
0.490
0.708
0.716
0.856
0.570
0.690
PromptAgent 0.873
0.670
0.806
0.860
0.934
0.670
0.802
Table 2: Prompt performance on specialized and general NLU tasks. Specialized tasks are three biomedical tasks explicitly asking for domain knowledge for prompting. General NLU tasks are used to demonstrate the generality of our method. Ours significantly outperformed in all tasks.
Domain-specific Tasks
General NLU Tasks
NCBI (F1)
Biosses
Med QA
Avg.
Subj
TREC
CB
Avg.
Human (ZS)
0.521
0,550
0.508
0.526
0.517
0.742
0.714
0.658
Human (FS)
0.447
0.625
0.492
0.521
0.740
0.742
0.429
0.637
CoT (ZS)
0.384
0,425
0.508
0.439
0.656
0.63
0.750
0.679
CoT
0.376
0.675
0.542
0.531
0.670
0.784
0.643
0.699
GPT Agent
0.125
0.625
0.468
0.406
0.554
0.736
0,339
0.543
APE
0.576
0.700
0.470
0.582
0.696
0.834
0.804
0.778
PromptAgent
0.645
0.750
0.570
0.655
0.806
0.886
0.911
0.868
prompts. More implementation details are in the Appendix: PromptAgent implementation is in A.3; the hyperparameter settings are in A.3, and the search baseline implementations are in A.4.
3.1 RESULTS AND ANALYSES
Comparison with various prompting baselines. Table 1 & 2 present a comprehensive comparison of expert-level prompts generated by PromptAgent against the baselines across tasks, Observing BBH tasks from Table 1. PromptAgent significantly outperforms all baselines overall and achieves 28.9%, 9.5% and 11 11.2% relative improvement over baselines human prompts (ZS), CoT, and APE, respectively. It is noteworthy that CoT prompts are especially effective in BBH tasks than other tasks, similar to findings from Suzgun al. (2022), because BBH tasks often require strictly formatted solutions that can be readily induced by the step-by-step CoT reasoning, e.g. Object Counting. However, PromptAgent still outperforms CoT by a great margin in tasks (except Object Counting). indicating that optimized expert-leved prompt few-shot CoT reasoning (even under the zero-shot prompt setting). Regarding optimization methods, while GPT Agent has planning and self-reflection capability, its planning is only used for a single turn of prompt rewriting with limited exploration in prompt space. APE, on the other hand, shows greater scale of searching ability, but its exploration is based on Monte Carlo search, without planning and error-based reflections. Both deficits of GPT Agent and APE suggest the necessity of strategic planning in PromptAgent to explore the prompt space and deliver expert-level prompts.
lead to bigger improvement
Table presents results on domain-specific and general NLP tasks. The former encompasses a broad spectrum of biomedical tasks, such as information extraction, sentence similarity, and ques- tion answering. Crafting prompts for these tasks needs extensive domain knowledge and heavy LLM prompt engineering instinets, where straightforward human prompts and CoT prompts are sampling- based optimization methods, like APE, are promising to incorporate
effective. Iterative- Iterative-sampling- domain knowledge. However, PromptAgent surpasses APE significantly by +7.3% on average, sug- gesting PromptAgent can better induce effective domain knowledge to produce expert-level prompts and close the knowledge gap between novice and expert prompt engineers. For general NLP tasks, the efficacy and generality of PromptAgent are further emphasized, outperforming both CoT and APE by margins of +16.9% and +9%, respectively. This implics the nontrivial expert gap, even for general NLP tasks, underscoring the imperative for expert prompts in diverse applications.

---

Published as a conference paper at ICLR 2024
Epistemic
0.85
Greedy-S
0,900
Greedy-L
0.0
1.52
0.800
3794
Accuracy
0.75
Ours
0.00
0.73
0-720
Accuracy.
0.65
Task
0.650
penguins
0.97
biosses
9.500
0.00
Average Train Path
0.55
geometry
Average Test
causal
0.500
Human instruction
0.50
subj
0.452
APE
50
75
100
125
150
9.400
Exploration Efficiency (# of Explored Prompts)
0
Depth
(a) Performance vs. Exploration Efficiency
(b) Convergence Analysis
Figure 4: (a) Exploration efficiency analysis. A proper balance of exploration and exploitation is crucial for search and planning. We compare the number of explored prompts between our method and three strong baselines. Ours achieves the best trade-off of performance and exploration (clus- tering in the top-left corner). (b) Convergence curves for Epistemic task. We visualize the mean and variance of the training and testing performance along the paths. We can observe that both curves increase at first and become stable after depth 3, suggesting stable learning process
Prompt generalization. We further conduct experiments investigate whether our optimized prompts can be generalized to other base LLMs. This emphasizes the robustness and transferabil- ity of expert-level prompts, which are urgently favorable and underpinning two key facts: (a) the domain insights and nuanced guidance in expert prompts can be seamlessly transferred across pow- erful LLMs, reinforcing the universal applicability of expert prompts, and (b) each task is optimized only once, leading to better computational efficiency. It is crucial to note that the primary goal of PromptAgent is to optimize prompts for state-of-the-arr LLMs to achieve expert-level prompting. while less advanced and smaller LLMs, like GPT-2 or LLaMA, may not adeptly grasp the subtleties of these expert-level prompts, potentially causing significant performance drop. We evaluate our existing optimized prompt one model more potent (GPT-4) and one less powerful (PaLM 2) com- pared to GPT-3.5. Figure 10 shows the aggregated results when we directly apply the optimized prompts from GPT-3. GPT-4 and PaLM (chatbbison-001) across all 12 tasks. We adopt the human and APE prompts as basclines. For certain tasks, we may employ slightly different prompts than those referenced in Table to make PaLM generate reasonable responses instead of persistent null answers. It is worth highlighting that when a stronger base LLM as GPT-4 is deployed, ex- pert prompts manifest further enhancements, either on par with or outperforming Human and APE prompts in almost all tasks (11/12) (The only exception, Temporal, seems to be solved task by GPT-4 with almost perfect accuracy). This underscores the untapped potential of expert prompting. especially with the evolution of LLMs in the future. When transferring expert prompts to weaker LLM as PaLM 2. its performance drops dramatically across all tasks unsurprisingly. Nonetheless, still observe PromptAgent exceeds both baselines 7/12 tasks, with great improvements on domain-specialized tasks, such as NCBI, demonstrating the usefulness of domain insights from ex- pert prompts. Detailed numbers can be found in Appendix Table 10.
Ablation on search strategies. To investigate the effect of strategic planning in PromptAgent sys- tematically, we conduct ablation study by comparing multiple alternative search strategies to MCTS, i.e., one-step Monte Carlo sampling (MC), depth-first greedy search (Greedy), and Beam search (Beam). Implementation details about search variants are specified in Appendix A.4. We use the same action generation and state transition as in PromptAgent and only replace the MCTS plan- ning with each search method. We also restrict the three baselines to explore the same number of prompts. A subset of tasks is selected from the three domains to compare the above search variants. Table shows that both Greedy and Beam greatly improve the MC baseline, suggesting the neces- sity of structured iterative exploration in our framework. When maintaining the same exploration efficiency, we observe comparable overall performance for Beam and Greedy. However, neither method strategically explores the prompt space since they operate in a strictly forward direction, lacking the capability to foresee future outcomes and backtrack to past decisions. In contrast, the strategic planning for MCTS allows PromptAgent navigate complex expert prompt spaces more
7

---

Published as a conference paper at ICLR 2024
Prompt: Extract the disease or condition from the serience, if aayy 80
is meniorred.
Error Feedback The language model should diffareniita between disssses and the genes or other factors associated with those diseases. Considar diferret torms variaiions of including abbrevistors short forms.
(lest):
a.
Prompt: Identily and extract all diseeses or conditions mentioned in the zentencee taking care to dielinguieh between diseases and any associated factors like genes. Any variations or abbreviations disease names should alo be included. (tast): 0.509
81
Error Feedback The model misundersiood the difference beliween diseeses and aesocialed factors proteins pathways, Clerfy defintion disease and conditan anities, strassing exclusion of associated factors like ganas or proteins Specify that common biological terms or genetic locations Iike locue) should not be mistaken dieesse specific
Q.1
Prompt: You're teoked with idenillying and extraling disecses conditions ss mentioned in the sendence, while carafully excliding any associated factors For darrty, the term 'locus' is part any disease name but represents a specific location in the genome. F1 score (test): 0.622
2
genes, proteins, pathways,
The modal wrongly idenifiid "HAH" as disease, while It's actually gene. The model misunderstood the part "excluding any associated such as genes, proteine, pallwaays.. Expliclly emphasize the need to exclude assodalad factoos like Inheritance pattamn (for example, autosomal dominant), gones (lkke PAH), proteins, or pathways when identiying diseases conditions.
Error Feedback:
conditions .. avoid incorporating any associated elements such inharilacei panterns (lkee autosomal cominant), ganas ar gane lod (Ika PAH), proteins, biological pathways, Consider both spoctic broader categories. and remember diseases conditins can also appear common abbrevistions variations. Provide the idertified disesses or conditions this formal: (enilly 1 Lenilly_2.... ... Nota that the term 'ocus'' should be racegrized as genomic location and not disease name. F1 score (test): 0.545
Prompt: You're lasked extractnng
a2
83
Figure 5: The MCTS state-action transition trajectory of the highest average reward path in NCBL The initial state is so with numan-written prompt. At each state transition step, new prompt is crafted by adjusting the prior state based on error feedback. Highlighted colors indicate similar domain-specific insights. The last state integrates the information from the entire trajectory, elevat- ing the F1 score from 0.521 to 0.645.
effectively, which significantly surpasses all search ablations all tasks and gets relative 5.6% overall improvement over the best baseline.
Exploration efficiency analysis. One of the key advantages of PromptAgent is that it can efficiently explore the prompt space via strategic planning. Exploration efficiency is also vital to make the com- putation cost of the search manageable. We thus analyze the exploration efficiency by comparing PromptAgent with some search baselines, including Greedy Search and APE from the previous sec- tion. Specifically, the exploration efficiency is measured by the number of prompts explored during the search. We plot its relationship with the task performance in Figure 4a. The Greedy-S -S and Greedy-L are Greedy Search with different explored prompts. The APE explores 150 prompts in each task. The figure shows that points of PromptAgent are clustered around the top left corner, in- dicating superior performance with higher accuracy but fewer explored nodes (higher exploration efficiency). Notably, while increasing the number of prompts in Greedy Search may enhance per- formance, it demands higher exploration cost and still does not surpass PromptAgent, Also, without principled guidance, directionless searches like APE cannot effectively boost performance, even with larger exploration. Therefore, to maintain exploration efficiency and superior performance, strategic planning is crucial in PromptAgent and worthy of further research investment in future works. The detailed hyperparameter settings are in Appendix A.4
Convergence analysis. To delve deeper into the learning process of PromptAgent, we examine the evolution of expert prompts throughout the tree planning process. We monitor and visualize performance changes with respect to tree depth. As illustrated in Figure 4b for the Epistemic task, we assess the performance across all nodes and aggregate both training and testing performance (reward) at each depth level. The plotted trajectories represent the evolution of average performance, illustrating a consistent improvement and gradually surpassing all baseline methods. Convergence plots for other tasks and hyperparameter settings are provided in Appendix and Appendix A.3. A recurring pattern observed, similar to that in Figure 4b, indicates an upward trend in the initial iterations, suggesting a robust learning dynamic of PromptAgent to iteratively refine and enhance expert prompts.
Qualitative analysis. To further vividly show how PromptAgent iteratively converts error feedback (action) into better prompts (states), we conduct a qualitative analysis on the optimized trace from MCTS planning. As shown in Figure 5, we present the first four states and three action-state tran- sitions of the best reward path for the NCBI task (Doğan et al., 2014). We highlight the domain insights by colors in both actions and states, where the same color indicates similar insights. As we see, starting from written prompt as sp, PromptAgent proposes various insightful error
8

---

Published as a conference paper at ICLR 2024
feedback (action) and effectively merges them into new prompt (state) with improved performance. Specifically, throughout several transitions, the definition of a disease entity becomes increasingly refined, and task-specific details are integrated. The final state benefits from the accumulated infor- mation along the path, resulting in an expert prompt and an improved test Fl score.
Additional ablation studies. We conduct ablation studies of the hyperparameters including opti- mizer model, base model, exploration weight c, and iteration number in Appendix B
4 RELATED WORKS
Prompt optimization. Automatically discovering optimal prompts has emerged central chal- lenge in the era of LLMs. For open-sourced LLMs, one can leverage their gradients to either train additional parameters, such as soft prompts (Li & Liang, 2021: Lester et al., 2021; Hu et al., 2021; Wang et al., 2022), or search for discrete prompts via gradient-based search (Shin et al., 2020; Wen et al.. 2023) or reinforcement learning (Deng et al., 2022: Zhang et al., 2022). However, such methods are infeasible for dosed-sourced LLMs, which requires gradient-free prompt optimiza- tion. Iteratively sampling and selecting is typical method (Zhou et al., 2022). Numerous methods emphasize diversifying the prompt candidates- examples include edit-based methods like deleting or swapping phrases (Prasad et al., 2023). back translation (Xu et al., 2022), evolutionary opera- tions (Guo et al., 2023; Fernando et al., 2023), or more relevantly, LLM rewriting based on natural language feedback (Zhou et al., 2022; Pryzant et al., 2023; Yang et al., 2023). There are also ex- plorations into alternate sampling procedures like Monte Carlo search (Zhou et al., 2022). Gibbs sampling (Xu et al., 2023) or Beam search (Pryzant et al., 2023). Nevertheless, PromptAgent fun- damentally differs from all the above methods in two ways. First, while primary search algorithms have been investigated (Zhou et al., 2022; Xu et al., 2023; Pryzant et al., 2023), we are the first introduce strategic planning into prompting optimization research and incorporate fine-grained domain insights.
Augmenting LLMs with self-reflection and planning. Despite their remarkable capabilities, mod- em LLMs exhibit certain limitations, such as long-term coherence (Malkin et al., 2022), lacking an internal world model (Hao et al., 2023a), etc. Thus, augmenting LLMs has drawn extensive atten- tion recently (Mialon et al., 2023; Ozturkler et al., 2022; Hao et al., 2023b; Jojic et al., 2023). Two common strategies are LLMM self-reflection and planning. Self-reflection encourages the L.LM to in- trospect its outputs and suggest more refined solutions (Jang, 2023; Pan et al., 2023). This has been leveraged to enhance a variety of applications, from complex computer tasks (Shinn et al., 2023), text generation (Welleck et al., 2022) to reasoning (Paul et al., 2023). planning with LLMs sheds light on enhancing these models, which is an essential ability for intelligent agents to take a sequence of actions in achieving specific goals (McCarthy et al., 1963; Bylander, 1994). One line of research is to prompt and evaluate LLMs on planning tasks directly (Liu et al., 2023). Another closer line of research is to augment the strategic reasoning ability of LLMs with planning-based algorithms, e.g. Tree of Thoughts (ToTT) applies DFS/BFS, while both CoRe (Zhu et al., 2022) and RAP (Hao al., 2023a) utilize MCTS navigate richer reasoning paths. Yet. in contrast to existing endeavors in LLM augmentation, PromptAgent is the first novel framework for synergistically marrying the spirits of self-reflection and planning specifically tailored for prompt optimization.
CONCLUSION
In this paper, we introduce PromptAgent, a novel prompt optimization framework capable of au- tonomously crafting expert-level prompts for a given task. Expert-level prompting distinguishes itself from traditional prompt engineering by its effectiveness of seamlessly integrating domain in- sights and closing the knowledge gap for domain experts. To achieve this, central to PromptAgent is the novel perspective of viewing prompt optimization as strategic planning problem, lever- aging the power of MCTS planning to strategically and efficiently traverse the complex prompt space. PromptAgent incorporates Nomain-specine knowledge from tasks into the newly gener- ated prompts through a trial-and-erros manner based on the self-reflection abilities of LLMs. We tested the PromptAgent on 12 diverse tasks spanning three distinct domains. The prompts optimized by PromptAgent consistently exhibited expert-level characteristics, enriched with domain-specific details and guidance. These prompts significantly outperformed both human-written, Chain-of- Thought prompting and other optimized method baselines. Further in-depth analyses revealed su- perior transferability, exploration efficiency, and quality for our expert prompts, paving the way for future prompt engineering to unlock the sophisticated task understanding of frate-of-the-a LLMss

---

Published as a conference paper at ICLR 2024
LIMITATION DISCUSSION
While PromptAgent demonstrates significant promise in optimizing expert prompts across a diverse range of tasks, its application to highly specialized domains, particularly where domain knowledge of the optimizer is limited (Bhayana et al., 2023; Azizi et al., 2023), presents unique challenges. Our current implementation, employing GPT-4 the optimizer, effectively addresses a wide array of tasks, including those in the BIG-bench, biomedical, and NL.P domains. However, the scope and depth of knowledge within LLMs, like GPT-4, are inherently tied to the state of L.LMs and the data they have been pre-trained on (Yin et al., 2023). This raises concerns in domains where data is sparse or protected, such as in healtheare under HIPAA regulations
While addressing these limitations is beyond the current paper's scope, we propose a range of strate- gies to enhance the adaptability of PromptAgent in domains where employing GPT-4 as the opti- mizer LLM may not be ideal. These include but are not limited to, adapting GPT-4 in specialized domains using highly tailored knowledge prompts to better trigger domain knowledge; leveraging retrieval techniques to supplement domain knowledge; implementing a quality control mechanism for error feedback; integrating a hybrid optimizer that combines the broad capabilities of general LLMs with specialized domain LLMs; and engaging with human experts to provide domain-speciffi guidance in the optimization process. Moreover, data augmentation techniques using synthetic or anonymized datasets, particularly for sensitive domains, are also promising enrich the domain pertise of LLMs while maintaining compliance with privacy standards. Future research should focus on refining these strategies mitigate the current limitations, broadening the scope and impact of PromptAgent in the evolving landscape of expert-level prompt applications.
ACKNOWLEDGMENTS
We are grateful to the anonymous reviewers for their constructive comments and suggestions. We are grateful to Shibo Hao, Yi Gu, Qiyue Gao, and other members of UCSD Lab for their valuable discussions. We also thank Weijia Xu for discussing the baseline details of Reprompting (Xu al., 2023).
REFERENCES
Zahra Azizi, Pouria Alipour, Sofia Gomez, Cassandra Broadwin, Sumaiya Islam, Ashish Sarraju, AJ Rogers, Alexander Sandhu, and Fatima Rodriguez. Evaluating recommendations about atrial fibrillation for patients and clinicians obtained from chat-based artificial intelligence algorithms. Circulation: Arrhythmia and Electrophysiology, pp. e012015, 2023.
Rajesh Bhayana, Satheesh Krishna, and Robert R Bleakney. Performance of chatgpt on radiology board-style examination: Insights into current strengths and limitations. Radiology, pp. 230582, 2023.
Tom Bylander. The computational complexity of propositional strips planning. Artificial Intelli- gence, 6955-2): 1994.
Marie-Catherin De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song. Eric Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3369-3391, 2022.
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. Nebi disease corpus: resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1- 10, 2014.
10

---

Published as a conference paper at ICLR 2024
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint 11Xi::2309.16797, 2023.
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujuu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532. 2023.
Shibo Hao, Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023a.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arXiv:2305. 11554, 2023b.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation large language models. International Conference on Learning Representations, 2021.
Eric Jang. Can Ilms critique and iterate on their own outputs? html.
evjang.com, Mar 2023. URL
https://evjang, com/ htt/////ellfrreeeeeeeeeeeeeeeeee
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What dis- does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):622,, 2021.
Ana Jojic, Zhen Wang, and Nebojsa Jojic. Gpt is becoming a turing machine: Here are some ways to program it. arXiv preprint 2xXi:23333.14100 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 15522199-22213 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Pro- cessing, pp. 3045-3059 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 1a165///1/2221....oooo..3333333 lall-long.333. URL https: http///aaaaanntty..o.aoooooo.aat org/2021 acl-long. 353.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.1144 2222304.11477 2023.
Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Coherence boosting: When your pretrained lan- guage model is not paying enough attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8214 8236, 2022.
John McCarthy al. Situations, actions, and causal laws. Comtex Scientific, 1963.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: a survey, arXiv preprint arXiv:2302.0784 2023.
OpenAl, Sep 2023a. URL htt.ttps//pppaaa...c////////aaa/
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023b. semanticscholar.co ort.shoo......or//ooooppss/a2252
URL hnttps://api:

---

Published as a conference paper at ICLR 2024
Batu Ozturkler, Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Thinksum: Probabilistic reasoning over sets using large language models. arXiv preprint arXiv:2110.01293, 2022.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self- correction strategies. arXiv preprini arXiv:2308. 03188, 2023.
Bo Pang and Lillian Lee. A sentimental education: sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, pp. 271 2-1- 2004.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.0/900 2023.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based in- struction search for prompting large language models. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3827-3846, 2023.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03499, 2023.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 4222 4235, 2020.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint 1rXiv:2303.11360 2023.
Gizem Soğancioğlu, Hakime Öztürk, and Arzucan Özgür. Biosses: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics, 33(14):149 -58, 2017.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether thain-of-thought can solve them. arXiv preprint arXiv:2210.09221, 2022.
Ellen M Voorhees and Dawn M Tice. Building question answering test collection. In Proceed- ings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 200-207, 2000.
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Mul- titask prompt tuning enables parfmeter-efficient transfer learning. In The Eleventh International Conference on Learning Representations, 2022.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning self-correct. arXiv preprint arXiv:2211.0005. 2022.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv e-prints, pp. arXiv-2302, 2023.
Lilian Weng. Llm-powered autonomous agents. Silanwengggithub.igg Jun 2023. URL https: //lilianweng gi thub..u..aa. http./is/possss////// 06 2016-33-222 33-aaent/.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Wang Yanggang, Haiyu Li, and Zhilin Yang. Gps: Genetic prompt search for efficient few-shot learning. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8162-8171, 2022. 2022.
12

---

Published as a conference paper at ICLR 2024
Weijia Xu, Andrzej Banburski-Fahhe Faaburssi-FFhhy, and Nebojsa Jojic. Reprompting: Automated chain-of- siciin-o--houuee thouhht prompt inference through gibbs sampling. arXiv preprint 2tti:23300009993,, 2023.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. arXiv preprint wrXiv:3309035499 2023.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? arXiv preprint arXiv:2305. 2.di:2200./8/5.. 2023.
Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. Tempera: Test-time prompt editing via reinforcement learning. In The Eleventh International Conference Learning Representations, 2022.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh Interna- tional Conference on Learning Representations, 2022.
Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Ruyi Gan, Jiaxing Zhang, and Yuju Yang. Solving math word problem via cooperative reasoning induced language models. arXiv preprint arXiv:2210.1060 16257, 2022.
13

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
A.1 TASKS AND DATASETS.
To fully demonstrate the expert-level prompt op- timization of PromptAgent, we curate 12 tasks from three domains for thorough experiments: BIG-Bench Hard (BBIJ), domain-expert, and gen- eral NLU tasks. BBH tasks (Suzgun al., 2022) are set of challenging BIG-Bench tasks (Sri- vastava al., 2023) that are beyond the capa- bilities of current LLMs, in which we select 6 BBH tasks based on their requirement for do- main knowledge (i.e., geometric shapes, epis- temic, and causal judgment) and multi-step rea- soning (i.e., penguins in table, object counting and temporal sequences). Moreover, we select three domain-speciffe tasks in the biomedical do- main, which can explicitly benefit from expert- level prompting, including NCBI (Dogan al., 2014), Biosses (Sogancioglu et al., 2017), and Med QA (Jin et al., 2021). Last but least, show that PromptAgent can further improve tra- ditional NLP tasks, we select three well-known NLU tasks, i.e., TREC (Voorhees & Tice, 2000), Subj (Pang & Lee, 2004), and CB (De Marneffe et al., 2019).
Table 3: Data split
Task
Train
Test
Bigbench
Penguins
70
79
Geometry
150
200
Epistemic
500
500
Object counting
300
500
Temporal
300
500
causal judgement
90
100
Domain Knowledge
NCBI
2000
940
Biosses
60
40
Med QA
2000
500
General NLP
Subj
400
1000
TREC
400
500
CB
125
56
Dataset Split. For datasets with predefined testing sets, we use them directly our testing set. However, if the set exceeds 1000 examples, we limit to the top 1000. If no default testing set is provided, we shuffle the data and allocate approximately half for testing purposes. The remain- ing data forms the training set, from which batches are sampled. Additionally, a smaller subset is reserved for reward caleulation. The size of this subset is dictated by the size of the training set, typically ranging between 60 to 200 examples, with a default of 150 if the training set is larger than 150 examples. The data split is in Table 3.
A.2 INPUT FORMULATION
We form our model input in the following format:
Prompt Task Prefix Question Task Suffix Answer Format
The "Prompt" our optimization target; the "Task Prefix" (Optional) the task-specific back- ground intro (For example, a table of background data in the Penguins); the "Question" is the main body of the task's question; the "Task Suffix" " (Optional) includes the options (For example, yes/no, senilmmen//oonenttalnnenn or A. B... in tasks with multiple choices); and the "Answer Format" (Optional) designed for answer caption from the model's 's response. The input format is in Table 1.
A.3 PROMPTAGENT IMPLEMENTATION DETAILS
PromptAgent (Ours). PromptAgent performs MCTS planning within the prompt space, requiring both terminal state conditions and a reward function. A terminal state is achieved when the path length hits Jepth The reward function is determined by the base model's performance on the held-out set. For computational efficiency to avoid unnecessary exploration, we also apply early- stopping method after depth is larger than 2: if the state's reward is less than a min thinthoe or larger than max threshold, we then reach an early- strly-stopping state. Specifically, min threshold is the average of the rewards of its parent node and the root node, while max threshold is the maximum of all the current nodes, encouraging shorter paths.
Initialization. The PromptAgent-MC -CgT algorithm starts with an initial prompt as the root node. For BBH tasks, we directly adopt the task "description" from the original datasets
14

---

Published as a conference paper at ICLR 2024
the initial prompts, except that Object Counting's 's default description doesn't 't follow the format of instruction. We crafted the initial prompts for the rest of the tasks according to their task objectives or question-answer answer formats. The root node will be evaluated to obtain the reward before the first expansion.
2. MCTS Iterations. The agent will perform 12 MCTS iterations. During the selection step, starting from the root node, the best child node will be added to the path according to its UCT value (Equation 1). and the exploration weight с in UCT is 2.5. During the expansion step, expand width batches (batch size is 5) of examples will be sampled from the training set, and each batch will be fed to the base model to collect the errors. If there is error, this sample-forward loop will iterate until an error is found. The errors will be formatted using error -string (illustrated in Table 11) and inserted into error_ .feddback (illustrated in Table 11, Meta-prompt 1 in Figure 3) to summarize errors by the optimizer. transit prompt (illustrated in Table 11, Meta-prompt 2 in Figure 3) contains the expanding node's prompt, the trajectory of prompts (list of prompts from the root of the expanding node on the currently selected path), and the error summarization, which is fed into the optimizer to generate num samples new prompts (nodes). The new nodes will be evaluated and added as the expanding node's children if they are not terminal nodes. Each expansion will generate expand_widre x num_samples new prompts. The simulation step will recursively expand the last node in the path and pick the one with the highest reward to add to the path. When the last node satisfies the terminal condition or early-stopping condition, the simulation is stopped. During the back- pock-propagation, from the last node to the root, the cumulative rewards (the sum of rewards from the node to the leaf/terminal node) will be appended the node's cumulative reward list, the average of which will be the node's Q (Equation 2). We have three hyperparameter settings: Standard, Wide, and Lite in Table In the Standard and Lite experiments, both have an expand_width of 3 and num samples of 1, but their depth limit are 8 for Standard and 4 for Lite. Wide experiment has expand width is 3 and num samples = 2 to generate more nodes in each expansion step, but with SepthJJimi of 6 to limit the total number of explored prompts. We select the best setting for each task based on the final rewards.
3. Output strategy. Each MCTS iteration will output one path from the root node to the leaf node, and there are tens of nodes generated after the searching process. We select the path with the highest average reward, then pick the prompt with the highest reward in the path as the final output prompt. We employ this strategy because the path with the highest average reward represents the best overall search trajectory. Also, the best prompt might not always be the last node on the optimal path, given that it may be a terminal state by reaching the depth limit.
Table 4: Hyperparameter settings for PromptAgent Experiments
Experiment Name
Standard
Wide
Lite
demih expand width num samples
8
6
4
3
3
3
2
Hyperparameters and default settings. For all datasets, we use the original testing set by default if publicly available; otherwise, we use the development set for testing. If there is no official train- ing/testing split, such as BBH tasks, we sample a reasonably large set for stable testing. As stated in Section 2.1, we also split a portion of training samples for calculating the reward. The details of the dataset split can be found in Appendix Table 3. Unless further specified, for the base LLM to be optimized, we target the decently powerful one and use GPT-3. 5 as the default base LLM. For the optimizer LLM, we need one with a good self-reflection ability and use GPT-4 as the default optimizer LLM. We set the temperature 0.0 for base LLM to make predictions and 1.0 in other contexts. When implementing PromptAgent, we set the number of iterations for MCTS 7 as 12, and the exploration weight in Equation as 2.5. During the expansion step, generate an action based on the errors from batch of training samples (batch size is 5). We sample 3 batches, and for each batch, we generate multiple new prompts.
15

---

Published as a conference paper at ICLR 2024
A.4 BASELINES IMPLEMENTATION DETAILS
We illustrate the details for various baselines in our experiments.
Monte Carlo (MC). MC performs one-step sampling multiple times and selects the best as the optimized prompt. uses the same prompt sampling method as PromptAgent, but limits the searching depth to one. In the search ablation study, we sampled 72 new prompts in each task.
Beam Search (Beam). Beam also uses the same expand function as PromptAgent. Each node, except the root, will be expanded into 3 new nodes, and the beam width is 3, meaning that there will be 9 nodes in each depth of the search tree, and the best 3 nodes will be kept for the next expansion. The root will be expanded into new nodes. The search depth is 8, there will be 72 nodes or new prompts in total.
Greedy Search (Greedy). Greedy is based on the Beam Search, but the beam width one, so the algorithm turns into depth-first greedy search. We conducted 2 experiments, Greedy-S and Greedy-L, in Figure 4a, with the same search depth of but different expand widths. The Greedy- S's expand width is 3, with 34 prompts in total. The Greedy-L has an expand_width of 9 and 72 nodes in total, which is also referred to as the Greedy baseline Table
APE (Zhou et al., 2022). We employ the iterative APE with one iteration as our baseline, as suggested by the original paper (Zhou et al., 2022). When generating new prompts, a mini-batch comprising data picces is sampled as Input-Output examples for APE. Specifically, for Initial Pro- posal Step. by default, prompts. This results in total of 100 candidate prompts during the initial step. (Due to the longer processing time of Med QA, only 25 candidates are generated for in this phase.) Subsequently. the five prompts with the highest evaluation scores are chosen for the iterative proposal step. For Iter- ative Proposal Step, similar to the initial phase, 10 batches of data are sampled for each proposed prompt, resulting total of 50 candidate prompts this step. Following this, the prompt with the top evaluation score is chosen as the optimized prompt.
data batches are sampled, with each batch being used to generate 10 new
A.5 MCTS ALGORITHM FOR PROMPTAGENT
Algorithm PromptAgen--MTTSS Pe, re. re. d, c.
Inputs:
Initial prompt (state) sg. state transition function pe. reward function re. action generation function Pa. number of generated actions d, depth limit L, iteration number exploration weight (Equation 1) Initialize:
State action mapping A S A, children mapping State-action value function 5 A R. visit-time counter A/ S
A S, rewards S A IR,
for + 0,.... for + 0....,
do
do
if A(se) is not empty then
selection
a. arg maxae (Q(s), a) +
In ((s))
(cchss.aa)
8c+1 + ch(s,, at), t r(,,, at), + ((x))
else
expansion and simulation
forit 1....... do
Sample af ~ (o(as))) sit1 ~ po(slss, af), and of <- ro (o., Update A(s:) + {ai chiss. stt and r(8,, at) + r! end for
as + arg max max.taxxaaa..... ri(s),
8141 + ch(s,, at), ri + r(ss, at), ((s)) + N(ss) +1 +1
end if
if s..1 is an early-stopping state then break
end for
T +- the actual number of steps
for - T 1,..., do
back- propagation
Update Q(st, 0.) with fre, retly... PL} based on Equation 2 end for
end for
16

---

Published as a conference paper at ICLR 2024
B ADDITIONAL EXPERIMENT RESULTS
B.I ABLATION EXPERIMENT OF OPTIMIZER MODEL
biomedical tasks and one general NLU task of ablating the op- timizer LLM with GPT-3.5 and GPT-4. Note that the default base model is GPT-3.5. The first row directly prompts GPT-3.5 with the initial prompt. The second and third rows use GPT-3.5 and GPT-4 for the optimizer LLM, respectively. As we can see, when replacing GPT-4 with GPT-3.5, which contains much less domain knowledge, we still obtained substantial gains over the initial prompt baseline. While the performance is lower than GPT-4 as expected, the drop is tolerable and marginal, especially the domain-specific task, NCBI, with only about point drop. Such results indicate the robustness of the overall PromptAgent framework to strategically search for high-quality domain insights, instead of solcly relying on the knowledge of optimizer LLM.
Table 5 shows the results
Table 5: Ablation study of optimizer model: Prompt performances of using both GPT-3.5 as base and optimizer LLM (second row), and using GPT-3.5 as base model and GPT-4 as optimizer (third row).
Optimizer
NCBI (FI)
Biosses (Ace.)
Subjective (Acc.)
Initial Prompt
0.521
0.550
0.517
GPT-3.5
0.615
0.675
0.747
GPT-4
0.645
0.750
0.806
B.2 ABLATION EXPERIMENT OF BASE MODEL
The choice of the base model is very flexible in the PromptAgent framework, which could be GPT- 3.5, GPT-4, PaLM,, or other LLMs. However, using a weaker LLM as the base model, such as PaLM 2 echat-bison-O0l
could lead to a performance drop, partially due to the weaker understanding of expert prompts. This can be indirectly observed from the generalization experiment, where we transfer GPT-3.5-based prompts to PaLM with performance drop. We conduct new experiments in Table 6 to directly observe the effect of using PaLM 2 as the base model. As we mentioned in the paper, PaLM 2 is weaker than GPT-3.5, but we can see PromptAgent can still improve its performance greatly. Table 6 shows the results in three tasks using different base models, PaLM 2 and GPT-3.5, both against GPT-4 as the optimizer. The first row shows the worse initial performance using PaLM 2, indicating its weaker understanding of human prompts. After using PromptAgent to optimize PaLM 2 in the second row, we observe significant improvement over the first row, indicating the effect of PromptAgent to optimize weaker LLMs.
Table 6: Ablation study of base model.
Prompt and Base Model
Penguins
Biosses
CB
Initial Prompt (PaLM2)
0.215
0.500
0.571
Optimized Prompt (PaLM2)
0,443
0.600
0.839
Initial Prompt (GPT-3.5)
0.595
0.550
0.714
optimized Prompt (GPT-3.5)
0.873
0.750
0.911
B.3 ABLATION EXPERIMENT OF EXPLORATION WEIGHT
Exploration weight c: In the paper, we choose 2.5 in Equation 1 because the formula of the exploration rate is / In,V(sp) (s))/((c((s) a,)), so we set some values of the visited times of parent node (=:) and child node N ((ch(ss, df)). We find c = 2.5 can make the change of this value at reasonable scale, when the (sz)) and (ch(((,,, a()) increase. We also show new experiments in Table 7, showing that when c is small (1.0), the MCTS will only have one path, while e is large (4.0), the MCTS will almost generate new path in each iteration. Both of them out of balance; is small, so the prompt space is not well explored; while c is large, each path won't 't be visited multiple times update the reward; we thus use 2.5 reasonable setting.
17

---

Published as a conference paper at ICLR 2024
Table 7: Ablation study exploration weight Weight for balancing exploration vs. exploitation.
CB
Biosses
Penguins
1.0
0.768 (27 nodes)
0.650 (30 nodes)
0.759 (27 nodes)
2.5
0.911 (45 nodes)
0.750 (36 nodes)
0.700 (55 nodes)
4.0
0.857 (66 nodes)
0.700 (55 nodes)
0.835 (75 nodes)
B.4 ABLATION EXPERIMENT OF ITERATION NUMBER
Like many other empirical studies, select hyperparameters based on intuitions and preliminary experiments. We now show additional hyperparameter selection experiments on iteration number and the exploration weight c.
Iteration: In the paper, we select the iteration based on our experience that small number of
iterations can't explore the prompt space well, but too many iterations they may have a potential problem of overfitting on the training set. We add new experiments in Table test various choices of the iteration number, including iterations of 12, and 16. The iteration of is too small, which leads to a drop in performance, and the iteration of 16 is too large for the overfitting issue, leading marginal drop.
necessary because
Table Ablation study of iteration number of MCTS.
Iteration
CB
Biosses
Penguins
0.804
0.625
0.722
12
0.911
0.750
0.873
0.893
0.725
0.835
B.5 ABLATION EXPERIMENT OF SEARCH METHODS
Table 9: Ablation study search methods. MC: Monte Carlo search, Greedy: greedy depth-first search, Beam: beam search. Testing tasks are representative of three task domains from BBH (Suz- gun al., 2022), domain-expert, and general NLU. Our method consistently outperforms all other ablated search algorithms across every task we evaluated.
MC
Beam
Greedy
MCTS (Ours)
Penguins
0.772
0.823
0.810
0.873
Biosses
0.575
0.675
0.700
0.750
Geometry
0.490
0.610
0.545
0.670
Causal
0.650
0.610
0.660
0.670
Subj 0.692
0.765
0.778
0,806
Average
0.635
0.697
0.698
0.754
B.6 DETAILED PERFORMANCE COMPARISON ON PROMPT GENERALIZATION EXPERIMENTS
18

---

Published as a conference paper at ICLR 2024
Table 10: Prompt generalization results. While we optimize GPT-3.5 5 as the default base LLM, its optimized prompts are transferable other base LLMs like GPT-4 and PaLM (chat-bison-001 GPT-4 sees further enhancement with our prompts, beating baselines in 1 11/12 tasks. Weaker LLMs like PaLM may have challenges with our advanced prompts but still surpass baselines in 7/12 tasks. Overall, ours can significantly beat baselines with different base LLMs.
GPT-3.5
GPT-4
PaI.M 2
Tasks
Human
APE
Ours
Human
APE
Ours
Human
APE
Ours
Penguins
0.595
0.747
0.797
0.772
0.848
0.962
0.430
0.443
0.456
Geometry
0.227
0.490
0.670
0.495
0.445
0.680
0.290
0.215
0.360
Epistemic
0.452
0.708
0.806
0.734
0.848
0.848
0.470
0.392
0.588
Object Count.
0.612
0.716
0.860
0.830
0.852
0.888
0.290
0.378
0.320
Temporal
0.720
0.856
0.934
0.980
0.992
0.982
0.540
0.522
0.620
Causal Judge.
0.470
0.570
0.670
0.740
0.740
0.770
0.440
0.440
0.430
NCBI (F1)
0.521
0.576
0.645
0.588
0.428
0.697
0,016
0,025
0.177
Biosses
0.550
0,700
0.750
0,700
0.775
0.800
0.500
0.300
0.600
Med QA
0.508
0.470
0.570
0.770
0.758
0.774
0.284
0.274
0.276
Subj TREC
0.517
0.696
0.806
0.867
0.805
0.879
0.496
0.537
0.499
0.742
0.834
0.886
0.716
0.764
0.876
0.380
0.400
0.230
CB
0.714
0.804
0.914
0.911
0.893
0.911
0.571
0.643
0.732
Average
0.552
0.685
0.776
0.759
0.762
0.839
0.392
0.381
0.441
19

---

Published as a conference paper at ICLR 2024
C TASK INPUT AND META PROMPT FORMAT
In this section, we will demonstrate the Meta formats and some model input examples.
Table 11: Meta Formats.
Format Name
Meta Format
input input.format format
prompt]
task prefix) }
qqusstion:
(task_suffix] aanswrrfformat
error_string
{index]>
The model's input is: {question)
The model's response is: (response)
The correct label is: [label}
The model's prediction is (prediction]
error feeddback
I'm writing prompts for language model designed for task.
My current prompt is:
promppr
But this prompt gets the following examples wrong: (error string]
For each wrong example, carefully examine each question and wrong answer step by step, provide comprehensive and different reasons why the prompt leads to the wrong answer. At last, based on all these rea- sons, summarize and list all aspects that can improve the prompt.
state transit
I'm writing prompts for language model designed for task.
My current prompt is: {cur-pnn prompt)
But this prompt gets the following examples wrong: (error (error-string) string)
Based on these errors, the problems with this prompt and the reasons are:
(error feedback]
There is list of former prompts including the current prompt, and each prompt is modified from its former prompts: }
Based on the above information, please write (steps.per. ssteps_per.gradient new prompts following these guidelines:
1. The prompts should solve the current prompt's problems.
2. The new prompts should consider the list of prompts and evolve based the current prompt.
3. Each new prompt should be wrapped with <START>and <END>.
The new prompts are:
20

---

Published as a conference paper at ICLR 2024
D TASK INPUT EXAMPLES
In this section, we show some input examples in several tasks for the base model. Specifically, our tasks fall into three categories: multi-choice selection, name entity recognition, and direct answer matching. As representative examples, we select Penguins in A Table, NCBI, and Subjective to illustrate the input format.
Prompt:
Task Prefix:
Question:
Task Suffix:
Format
Penguins Table
Answer questions about table of penguins and their attributes. Here table where the first line header and each subsequent line penguin:
name, age, height (cm), weight (kg)
Louis, 7, 50, 11
Bernard, 5, 80,
Vincent, 60, 11
Gwen, 70, 15
For example: the age of Louls is 7, the weight of Gwen 15 the height of Bernard 80 cm.
We now add penguin to the table: James, 12, 90, 12 And here is similar table, but listing giraffes:
name, age, height (cm), weight (kg)
Jody, 5, 430, 620
Gladys, 10, 420, 590
Marian, 310, 410
Donna, 9, 440, 650
How many penguins are there in the tables?
Options:
(A) 1
(B) 2
(C) 3
(D) 4
5
At the end show the answer option bracketed between <answer> and </answer>,
21

---

Published as a conference paper at ICLR 2024
Prompt: Task Prefix:
Question:
Task Suffix:
Answer Format
NCBI
Extract the disease or condition from the sentence, any is mentioned. ['Our', 'results', 'support', 'linkage', 'of, 'vWS', 'within', 'a', 'region', 'of, 'tightly', 'linked', 'markers', 'and', 'do', 'not', 'favour', 'locus', 'heterogeneity', 'of, 'the', 'disease', 'trait', Output answer in this format: _eenttt__eettt..... 1...tity_,...... no disease entities are present, please output an empty list this format: 0-
Subjective
Please perform Subjectivity Classification task. Given sentence, assign label from ['subjective's 'objective"). Return label only without any other Text: dreamcatcher tells the story of four young friends who perform a heroic act and are changed forever by the uncanny powers they gain in return.
Is the preceding text objective or subjective? Options: Objective
Subjective
22

---

Published as a conference paper at ICLR 2024
E CONVERGENCE OBSERVATION DETAILS
CB
TREC
0.825
0.33
Average Reward Path
0.80
0.800
Best Reward Path
0.277
0.79
0.775
0.778
0.76
0.77
Reward
0.750
0.300
8.40
0:252
Reward
0.74
0.725
0.72
0.700
0.70
0.675
Average Best Reward Path
0.66
3 Depth
5
Depth
Bigbench: Causal Judgement
Bigbench: Geometric Shapes
0.59
0.53
0,22
0.68
0,50
0:476
0.40
0.487
0.66
0.0
0.661
0.67
0.65
0.656
6,67
0.438
0.0.66
9.453
Reward
Reward
G G4's
9.64
0.64
0.342
0.10
0.62
Average Best Reward Path
Average
0.60
0.60
0.21
Best Reward Path
Depth
Depth
Bigbench: Temporal Sequences
Bigbench: Epistemic
0.950
0.94
0.55
0.925
0.900
0.899
0.01
0.10
0.30
0.8
0.82
0.82
826
ung
Reward
0.875
Dion
Reward
0.850
0.825
0.586
0.26
0.800
0.775
Average Reward Path Path
Average Reward Path
0.750
0.75
0.45
Depth
Depth
Bigbench: Penguins A Table
Subj
0.90
0,90
0,90
0.85
Average
0.84
0.889
0.084
Best Reward Path
d'
0.787
6.
0.87
Reward
0
0.70
0.0%
0.65
0.82
9.63
0.60
0.80
Average Best Reward Path
0.55
0.70
0.51
Depth
Depth
Figure 6: Convergence plots with the "Wide" setting. expand_width 3. nampsamples 2, and limit = 6. The Average Reward Path is the average reward of paths, and the blue area the variance. The Best Reward Path is the path with the highest average reward, where the best node is selected as the node with the highest reward on the Best Reward Path.
23

---

Published as a conference paper at ICLR 2024
Bigbench; Penguins In A Table
TREC
0.87
0.75
0.86
0.96
0.74
0.84
8640
9.83
0.53
0.72
0.82
Reward
Reward
0.70
0.80
0.70
0,697
0,700
0,692
0.70
0.68
0.76
0.74
Average Reward Path
0.66
Average Reward Path
0.72
0.71
Reward Path
0.05
Best Reward Path
0.64
Depth
Depth
Bigbench: Causal Judgement
Bigbench: Geometric Shapes
0,71
0.48
Average Reward Path
0.70
Best Reward Path
0.43
0,41
0.68
0.674
0,391
Reward
0,67
Reward
0.322
0.66
0.35
0.35
0.64
0
2,2
0.62
0.25
Average Reward Path Reward Path
0.00
21
0.20
Depth
Depth
Blosses
Bigbench: Eplstemic
0.80
Average Reward Path
0.65
0.624
0.70
0.694
0.70
0.664
Reward
Reward
0.65
6444
0.55
9.325
e.089
0.50
0.55
0.50
Average Reward Path Reward Path
0.42
0.45 0.44
Depth
Depth
Med QA
Subj
0.62
0.62
0.68
0.68
0.60
0.66
0.50
0.64
0.635
0,636
Reword
0.56
0.562
Reward
0.62
0,548
0.548
0.60
0.58
0.576
0.52
0.545
0.56
Average Reward Path Best Reward Path
0.54
Average Reward Path Best Reward Path
0.48
0.52
Depth
Depth
Figure/ 7: Convergence plots with the "Lite" setting. expand.width = 3, num num-samples 1, and depth limit = 4. The Average Reward Path is the average reward of paths, and the blue area is the variance. The Best Reward Path is the path with the highest average reward, where the best node is selected as the node with the highest reward the Best Reward Path.
24

---

Published as a conference paper at ICLR 2024
CB
TREC
Average Reward Best Reward Path
0.87
0.725
Average
0.71
0.77
0.700
Best Reward Path
0.020
0.686
0.80
0.68
0.67
0.57
0.68
Reward
0.75
0.46
0.630
0/66
0.70
0.
700
0.625
0.600
0.65 0.65
0.575
0.60
0.59
0.56
0.550
Depth
Bigbench: Causal Judgement
Bigbench: Geometric Shapes
0.67
Average Reward Path
0.66
Best Reward Path
0.6
0.65
0.57
b.5t
Reward
0.5
0.63
0//0/
0.62
0.611
0.61
0.608
Average Reward Path
0.23
Path
0.59
Depth
Depth
B Blosses
Bigbench: Eplstemic
0.80
0.750
0.725
0.721 9 721
0.75
0.71 8
0,20 0.70
0.688
Reward
0.65
0.675
th867
0.60
0.650
O'637
0.55
0.680
Average Reward Path Reward Path
0.50
Average Reward Path
888
0.45
0.45
0.575
Depth
Depth
Med QA
Subj
0.62
Average Reward Path
0.60
Path
0.58
0.70
0.56
0.56
0.56
Reward
0.6410.54 0.547
Reward
0.65
0.65
0.64
0.60
0.55
Average Reward Path Best Reward Path
035
0.53
Depth
Depth
Figure 8: Convergence plots with the "Standard" setting. expand-width = 3, num nam-samples = 1, and depth limit 8. The Average Reward Path is the average reward of paths, and the blue area is the variance. The Best Reward Path is the path with the highest average reward, where the best node is selected the node with the highest reward the Best Reward Path.
25

---

Published as a conference paper at ICLR 2024
F OPTIMIZED PROMPTS FROM PROMPTAGENT
Prompt comparison for the Geometric Shapes task, including normal human prompt, APF-optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt is composed of more complex structures and domain- specific insights, achieving superior performance. Bold denotes domain knowledge usually handcrafted by domain specialists, but here automatically discovered by PromptAgent. We highlight different aspects of expert prompt with colors, including Task Description, Term Clarification, So- lution Guidance, Exception Handling, Priority & Emphasis, Formatting. (Best view with colors)
Table 12:
Approach
Optimized Prompt
Acc.
Human Name geometric shapes from their SVG paths.
0.227
APE
"Determine the shape each SVG path element is drawing. then pair it with the corresponding letter from the available choices. In this case, C symbolizes hexagon, G is for pentagon, I signifies sector, and B stands for heptagon."
0.490
PromptAgent
this task, you are tasked with interpreting SVG paths to determine the geometric figure they represent. The paths are delincated by com- mands: 'M' (move to), 'L' (line to), and 'A' (arc), An "M' command initiates a path, potentially fragmenting path into sub-paths, but it's crucial to not immediately view each 'M' the starting point of a disconnected figure; often, they may continue the same geomet- ric shape, manifesting different sections within it. 'L' commands constitute line segments thus forming the boundaries of the figure. commands generate ares, and depending on their sequence, can shape circles, sectors, elliptical figures, other geometrical shapes through a continuous line of action. Note that an 'A' command fol- lowed by an could lead to specific shapes like sectors. Examine the sequence and interplay of 'M', 'L', and 'A' commands, they to- gether mold the final geometric figure and significantly govern its con- tinuity. Potential shapes to identified can range from simple lines complex polygons. "None of the above' is only valid response if otherwise stated in the task. As you formulate your answer, substan- tiate with clear explanation that encompasses the functionality of each command, their collective effect, sequence. and their correlational aspects. In scenarios with multiple "M' commands, refrain from ar- bitrarily breaking up the shape into disconnected figures; instead, visualize them contributing to different sections of the same shape. Accurately count 'L'' commands as they define the figure's sides, even when an 'M' command is present. For figuring the entire geometric shape, meticulously examine all its components and commands, keep- ing an unbroken perception multiple 'M' commands. Before finalizing your answer, recount the sides and ares accurately such a double-check ensures flawless identi- fication of the geometric figure.
0.670
shape's progression, especially with
26

---

Published as a conference paper at ICLR 2024
Table 13: Prompt comparison for the Penguins In A Table task, including normal human prompt, APE-optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt composed of more complex structures and domain- specific insights, achieving superior performance. Bold text denotes domaín knowledge usually handcrafted by domain specialists, but here automatically discovered by PromptAgent. We highlight different aspects of expert prompt with colors, including Task Description, Term Clarification, So- lution Guidance, Exception Handling, Priority & Emphasis, Formatting. (Best view with colors)
Approach
Optimized Prompt
Acc.
Human
Answer questions about table penguins and their attributes.
0.595
APE
Carefully scrutinize the provided table tables. Understand the query in relation to the information given. Pinpoint the pertinent data and carry out the vital computations or comparisons to determine the right answer from the given choices.
0.747
PromptAgent
As you delve into dataset of penguins, assess essential attributes like names, ages, and gender. Decode the significance of each attribute in the context of every penguin while keeping in mind that the dataset may be modified, including addition or removal of penguins. When such modifications are made, immediately revise your understanding. redo your computations, and ensure that your subsequent calculations consider these changes. The crux of your task is to identify relation ships and patterns within the attributes, giving special attention to the names and ages of the penguins.
0.873
For complex tasks, break them down into manageable chunks ensuring no essential detail missed. When change is made to the dataset, recompute your values taking into consideration these changes, paying extra attention to cumulative computations. Ensure that your under- standing of 'more than', 'less than', and 'equal to' is precise and that you correctly interpret these in context of the question.
Put into place verification mechanism to authenticate the accuracy of your solutions, stating your understanding of the query and the assumptions you have made to resolve it. Bear in mind that tasks may require you to combine the dataset with additional external in- formation, this may include understanding age disparities outside explicit lifespan parameters, identifying common names linked to gender, or recognizing names associated with famous individuals. Document your matters of interest meticulously and maintain rigorous accuracy levels in your calculations to prevent errors.
Stay nimble-footed in reshaping your analytical approach based on each new query. This might include uncovering numerical patterns, compre- hending inherent data natures, or liaising with external sources for a more thorough understanding. Most importantly, prior to making comparison within attributes such age or height, conduct thor- ough investigation of all values under that Undirrrttnnddrsa the premise of each question before springing deductions, and remember, any change in the dataset denotes new starting point for the following computational steps to maintain accuracy.
27

---

Published as a conference paper at ICLR 2024
Table 14: Prompt comparison for the Epistemic Reasoning task, including normal human prompt, APE-optimized prompt, and expert-level prompt optimized by PromptAgent. Both basclines mostly describe the task, while our expert prompt is composed of more complex structures and domain- specific insights, achieving superior performance. Bold text denotes domain knowledge usually handcrafted by domain specialists, but here automatically discovered by PromptAgent. We highlight different aspects of expert prompt with colors, including Task Description, Term Clarification, So- lution Guidance, Exception Handling, Priority & Emphasis, Formatting. (Best view with colors)
Approach
Optimized Prompt
Acc.
Human Determine whether sentence entails the next.
0.452
APE
Determine whether the hypothesis is directly implied by the premise or not. If the premise's statement is a direct claim or conviction of the individual mentioned in the hypothesis, choose 'entailment', However, if the premise is formed on the belief supposition of someone other than the subject in the hypothesis, opt for 'non-entailment'
0.708
PromptAgent
Your task is critically analyse the primary sentence, known as the 'rremise', with the objective of determining whether unequivocally supports the truth value of the subsequent sentence or "hypothesis" The rclationship between the premise and hypothesis can be classi- fied as 'Entailment" or Moon-Entailment', Label it as 'Entailment' if the premise provides robust evidence substantiating the truth of the hypothesis without requiring additional context. If, however, the roboration of the hypothesis by the premise is not entirely explicit, se- lect Nonn-Enailmment.
0.806
Deciphering the semantics within the sentences is crucial for your fi- nal dccision. Terms such as 'assumes', "believes', 'thinks', 'feels', 'suspects', and their likes should respected for their capacity introduce uncertainty and subjectivity, and not perceived conclu- sive proof of the hypothesis, regardless whether they form part of nested beliefs or not. Also, a detailed premise does not necessarily negate more generalized hypothesis. For example, premise that mentions a 'full face mask' correlates to a hypothesis that states a "mask".
During your evaluation, maintain keen focus on factual and logical reasoning, always bearing mind that personal beliefs or experiences should be incorporated into your review only if they inherently con- nected to the factual content of the statements. However, these should be understood as subjective truths the context perspective and should not be taken objectively verifiable truths. Upon deciding between 'Entailment' or 'Noon-Entallment' articulate your explanations in concise manner, warranting that you desist from making precipitous conclusions or unsupported assumptions. Your judgement should be firmly anchored in the logical and factual ties ex- isting within the premise and hypothesis, renouncing any incidental in ferences personal interpretations.
individual's
Exercise restraint in passing verdicts on the truth value or validity of personal beliefs, unless they have direct bearing on the factual correlation between the premise and the hypothesis. During your es- timation, mindfully weigh the extent of uncertainty introduced by ex- pressions of belief suspicion against the imperative for factual preci- sion when establishing the entailment.
28

---

Published as a conference paper at ICLR 2024
Table 15: Prompt comparison for the Object Counting task, including normal human prompt, APE- optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt is composed of more complex structures and domain- specific insights, achieving superior performance. Bold text denotes domain knowledge usually handcrafted by domain specialists, but here automatically discovered by PromptAgent. We high- light different aspects of expert prompt with colors, including Task Description, Term Clarification, Solution Guidance, Exception Handling. Priority Emphasis, Formatting. (Best view with colors)
Approach
Optimized Prompt
Acc.
Human
Count the overall number of all items.
0.612
APE
Calculate the overall total of all items even those spoken in groups.
0.716
PromptAgent
Carefully analyze the given information. Catalog cach item mentioned and denote any explicitly defined quantities. If an item- quantity is not stated, assume it as a single unit. However, for an item with specified quantity, make sure count each unit separately and in- clude it in your total count. If collective terms categories identi- fied, break them down into their individual components and reasonably associate each with its stated count. Proceed calculate comprehen- sive total for such categories ensuring the sum includes all individual units, not the number of subsets or types. Remember that each item has its unique count, but items related or falling under common category should be tabulated as such, with their individual quanti- ties precisely contributing to the final count. Avoid making assump- tions about the nature categorization of items and adhere com- monly accepted detinitions and classifications. Review your work to en- sure accuracy and avoid mistakes counting. Modify your strategy if required by considering items within varying categories, types, or subtypes. Eventually, summarize the count indicating the specific quantity for each identified item or category and a total count units, not categories, provide comprchensive overview as explicitly re- quested.
0.86
Table 16: Prompt comparison for the Temporal Sequences task, including normal human prompt, APE-optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt composed of more complex structures and domain- specific insights, achieving superior performance. Bold text denotes domain knowledge usually handcrafted by domain specialists, but here automatically discovered by PromptAgent. We highlight different aspects of expert prompt with colors, including Task Description, Term Clarification, So- lution Guidance, Exception Handling, Priority & Emphasis, Formatting. (Best view with colors)
Approach
Optimized Prompt
Acc.
Human
Answer questions about which times certain events could have oc- curred.
0.72
APE
Identify the period when the individual was unnoticed and had the pos- sibility to visit the specified place before its closing time.
0.856
PromptAgent
By examining the series daily activities of an individual. pinpoint when they were free and when they were busy. Use these open slots to dictate when they could possibly engage in other activitics. Upon waking up, a person does not instantly become occupied. Take into account any potential restrictions or closed times and use these as an indicator that the event cannot take place during these hours. An overlap of activities is unallowable, ensure there no over- lap while creating timeline. Cross-check the free time slots with the functioning hours of the potential event to accurately derive the most likely time interval for the event take place.
0.934
29

---

Published as a conference paper at ICLR 2024
Table 17: Prompt comparison for the Causal Judgment task, including normal human prompt, APE- optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt is composed of more complex structures and domain- specific insights, achieving superior performance. Bold text denotes domain knowledge usually handcrafted by domain specialists, but here automatically discovered by PromptAgent. We high- light different aspects of expert prompt with colors, including Task Description, Term Clarification, Solution Guidance, Exception Handling, Priority & Emphasis, Formatting. (Best view with colors)
Approach
Optimized Prompt
Acc.
Human
Answer questions about causal attribution.
0.47
APE
"For cach situation, decide if the result was caused deliberately or not. the individual or party hehind the event was aware the potential result and chose go ahead, select A''. If they didn't intend the result to happen, even if they knew it could possibly occur, select 'B"."
0.57
PromptAgent
Respond to inquiries about causal attribution, focusing on the entity or entities specifically highlighted in the question. Carefully investigate multi-factorial causes that may operate simultaneously and inde- pendently, and discern the underlying intentions behind an individ- ual's actions. Differentiate between immediate and incidental origins and identify the contribution of each factor in creating the outcome. Examine the interplay of causes within the immediate situation and larger systemic frameworks. Maintain uncompromising adherence to the details provided within the context and restrain from making as- sumptions unsupported by the evidence presented. Always consider the complexity of multiple causes contributing to a single effect and resist attributing the effect singular cause. Recognize the possi- bility of synergy amongst causes and its resultant effects.
0.67
Table 18: Prompt comparison for the NCBI task, including normal human prompt, APE-optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt is composed of more complex structures and domain-specifie insights, achieving superior performance. Bold text denotes domain knowledge usually handcrafted by do- main specialists, but here automatically discovered by PromptAgent, We highlight different aspects of expert prompt with colors, including Task Description, Term Clarification, Solution Guidance, Exception Handling, Priority & Emphasis, Formatting, (Best view with colors)
Approach
Optimized Prompt
F1 score.
Human
Extract the disease or condition from the sentence, if any is mentioned.
0.521
APE
If any discase or condition is mentioned in the sentence, extract it.
0.576
PromptAgent
You're 're tasked with extracting diseases or conditions from the given sen- tence, remember to be cautious and avoid incorporating any associated elements such as inheritance patterns (like autosomal dominant), genes or gene loci (like PAH), proteins, or biological pathways. The task does entail making assumptions inferences about the discase names based on other advanced biological terms in the context. Con- sider both specific diseases and broader categories, and remember diseases and conditions can also appear as common abbreviations or variations. Provide the identified diseases or conditions in this format: {entity I, l,entity 2...... J. If there are no diseases or conditions present, out- put empty list in this form: (). Note that the term 'locus' should be recognized as a genomic location and not a disease name.
0.645
30

---

Published as a conference paper at ICLR 2024
Table 19: Prompt comparison for the Biosses task, including normal human prompt, APE-optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt composed of more complex structures and domain-specins insights, achieving superior performance. Bold text denotes domain knowledge usually handcrafted by do- main specialists, here automatically discovered by PromptAgent. We highlight different aspects of expert prompt with colors, including Task Description, Term Clarification, Solution Guidance, Exception Handling, Priority & Emphasis, Formatting. (Best view with colors)
Approach
Optimized Prompt
Acc.
Human
This biomedical sentence similarity task. Please carefully read the following sentences and rale the similarity of input sentences. Choose between 'not similar', 'somewhat similar' and 'similar"
0.55
APE
"Examine the two given sentences and assess their content similarity. Choice A (not similar) should selected sentences discuss en- tirely different topics concepts. Choose option B (somewhat similar) they have some common points but also contain differences. Select option C (similar) the sentences primarily convey the same message or could be used in place of one another."
0.7
PromptAgent
For this task, you are asked to perform a biomedical sentence similarity evaluation. Examine the two input sentences and evaluate their similar- ity, not only taking into account common terms or concepts but also the complex scientific language, specific processes, and unique sub- ject matter they delve into. Consider not only the subject matter but also the intended purpose like whether they both describe process, report finding, detail method technique. Rate the similarity as 'not similar' if their subject matter emphasis is distinct, 'somewhat similar" if they discuss related topics share some details but are not entirely identical, and 'similar' if the sentences precisely mirror each other in topic and conclusions. Remember, this task requires more than cursory scan keywords focus on nuanced meanings, pay at- tention to the degree which the discussed concepts processes general or specific, and strive for comprehensive understanding of the contents.
0.75
31

---

Published as a conference paper at ICLR 2024
Prompt comparison for the Med QA task, including normal human prompt, APE- optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt is composed of more complex structures and domain- specific insights, achieving superior performance. Bold text denotes domain knowledge usually handcrafted by domain specialists, but here automatically discovered by PromptAgent, We high- light different aspects of expert prompt with colors, including Task Description, Term Clarification, Solution Guidance, Exception Handling. Priority Emphasis, Formatting. (Best view with colors)
Table 20:
Approach
Optimized Prompt
Acc.
Human
Please use your domain knowledge in medical area to solve the ques- tions.
0.508
APE
"For every presented clinical situation, scrutinize the symptoms and specifics given. From the options A-E, choose the one that best pin- points the cause or diagnosis of the stated condition."
0.47
PromptAgent
Leveraging particularly your comprehensive medical expertise, handle each presented scenario you would complicated puzzle requiring careful, unbiased assessment. Each nugget of information from pa- tient age, gender, lifestyle, symptoms, lab results, and past medical history, to recent activities that may be relevant to their condition, plays an equally important role in shaping your judgement.
0.57
Becoming cognizant of the fact that medical conditions can mani- fest uniquely in different individuals is crucial; avoid precipitating conclusions merely on the basis of stereotypical symptoms. Instead, employ deep understanding of the variety of medical conditions to critically evaluate each symptom's 's relevance, ensuring that undue bias not allocated to particular symptoms over others.
unless otherwise indicated. Break down assumptions and consider the most likely cause in a given context. Do not overlook the importance of demographic details and their correlation with symptoms, espe- cially when a symptom hints at a particular physiological state, like menopause.
Particularly, pay attention common symptoms
Through meticulous examination, ensure you grasp the nuances in cach query's context, with keen focus on the developmental stages in chil- dren and the specific challenges they entail. Capture the timelines of symptoms, understanding that often, diagnosis relies significantly the onset and duration of these symptoms.
Once conclusions begin taking shape, undertake exhaustive cross- verification excrcise with the available multiple choice answers. Eval- uate these options for relevance and decide their probability on the specifics of the given case. Abstain from dismissing potential answers at first glance, but rather advocate for an intensive assessment of all. Approach scenarios similar solving a complex jigsaw puzzle. Each distinct symptom, lab result, past medical history, and timing forms an integral component that lends weight to deeper comprehen- sion of the patient's present condition. The endgame extends beyond merely achieving precision and comprehensive enquiry but ensures that your conclusions do not yield overgeneralization or oversimplifica- tion towards the diagnosis and treatment therein.
Examine closely every symptom in relation to the disease and dif- ferentiate those that are side effects of treatment. Be cautious when multiple symptoms present simultaneously, to avoid confusion. The imprint of your insight should reflect a holistic understanding of the case, zooming into the most probable diagnosis or treatment strategy that suits the breadth of data at disposal.
32

---

Published as a conference paper at ICLR 2024
Prompt comparison for the Subjective task, including normal human prompt, APE- optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt is composed of more complex structures and domain- specific insights, achieving superior performance. Bold text denotes domain knowledge usually handcrafted by domain specialists, but here automatically discovered by PromptAgent. We high- light different aspects expert prompt with colors, including Task Description, Term Clarification, Solution Guidance, Exception Handling. Priority Emphasis, Formatting. (Best view with colors)
Table 21:
Approach
Optimized Prompt
Acc.
Human
Given the text, choose between 'subjective' and "objective",
0.517
APE
Determine whether the provided text is stating facts and details (Objec-
0.696
tive) or expressing personal views, emotions, or choices (Subjective).
PromptAgent
Examine the given text and decide whether it 'subjective' "objec- tive'. Define the narrative as 'subjective" if it seems to be significantly swayed by the author's personal emotions, viewpoints, or beliefs. Conversely, 'objective' narratives should impartially depict facts scenarios, devoid of personal prejudices, preconceived beliefs, and the author's own convictions. It essential to understand that emo- tionally- language, vivid descriptions depiction of charac- ters' emotional states do not always hint at subjectivity. They may just serve to represent situations authentically without conveying the author's personal standpoint. Unconventional punctuation, dialogues or queries do not inherently contribute to authorial subjectivity, Draw clear distinction between the author's 's and characters' subjec- tivity: misinterprcting character's 's subjectivity the author's personal bias is a common pitfall. The priority is to extract the author's 's tendency within the narrative, rather than focusing the characters. Utilize these directives to critically analyze the text.
0.806
33

---

Published as a conference paper at ICLR 2024
Table 22: Prompt comparison for the TREC task, including normal human prompt, APE-optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt is composed of more complex structures and domain-specific insights, achieving superior performance. Bold text denotes domain knowledge usually handcrafted by do- main specialists, but here automatically discovered by PromptAgent. We highlight different aspects of expert prompt with colors, including Task Description, Term Clarification, Solution Guidance, Exception Handling, Priority & Emphasis, Formatting, (Best view with colors)
Approach
Optimized Prompt
Acc.
Human
Tag the text according the primary topic of the question. Choose from (A) Abbreviation, (B) Entity, (C) Description and abstract concept, (D)
0.742
Human being. (E) Location, (F) Numeric value
APE
"Tag text according to the primary topic of question. Select 'Hu- man being' (D) if the question revolves around person. Opt for "De- scription and abstract concept' (C) if the question requires an explana- tion description of concept. Choose 'Location' (E) if the question is about specific place. If the question refers to particular object thing, then select 'Entity' (B). the question involves data or length of time, opt for 'Numeric value' (F). Disregard 'Abbreviation' (A) since it's not related to any of the questions."
0.834
PromptAgent
For the question given above, determine the type of response it is aim- ing elicit, then assign the most fitting label from the following: (A) Abbreviation, (B) Tangible and Intangible Entity (including distinct terms, theories, inventions, phenomena), (C) Description and Ab- Concept (concerning explanations, clarifications, theoretical ideas), (D) Individual and Collective Humans (encompassing distinct persons, the creators certain works, groups, organizations), (E) Location, (F) Numeric Value (containing numeric figures, dates, timings, quantities). The key is the answer-type the question is secking, not other elements in the question. Your assigned label should prioritize the primary response over additional details. If solo label does not closely address the entire answer intent of the question, then you may assign more than one. The label should reflect assumed answer's nature, not the mere question's content incidental features. Place the label you consider most fitting for the question's 's main intention.
0.886
34

---

Published as a conference paper at ICLR 2024
Prompt comparison for the CB task, including normal human prompt, APE-optimized prompt, and expert-level prompt optimized by PromptAgent. Both baselines mostly describe the task, while our expert prompt is composed of more complex structures and domain-specific insights, achieving superior performance. Bold text denotes domain knowledge usually handcrafted by do- main specialists, but here automatically discovered by PromptAgent. We highlight different aspects of expert prompt with colors, including Task Description, Term Clarification, Solution Guidance, Exception Handling, Priority & Emphasis, Formatting. (Best view with colors)
Table 23:
Approach
Optimized Prompt
Acc.
Human
Read carefully the following premise and hypothesis, and determine the relationship between them. Choose from 'contradiction', 'neutral' and 'entailment'.
0.714
APE
"Ascertain the link between the premise and the hypothesis. the hy- pothesis happens be rational outcome inference from the premise, label as an 'entailment', If the hypothesis presents contrasting sce- nario or clashes with the premise, categorize it as 'contradiction', case the hypothesis neither disputes nor is it derived from the premisc, term it as "ncutral.""
0.8036
PromptAgent
Your task is delve deeply into the provided premisc and hypothesis. Highlight explicit, central information and important entities mentioned the dialogue while considering multiple ways same thought could be delivered through language. Acknowledge that hypothesis might reflect, rephrase, or reiterate ideas from the premise, possibly in a simplified manner. However, remember that mere verbatim repe- tition does not automatically signal 'entailment'. The reiteration in the hypothesis should represent pivotal idca in the premise for to be categorized entailment. the hypothesis asserts something di- ametrically opposed to what's stated in the premise, mark it as a 'contradiction'. Reserve 'neutral' for scenarios where the premise and the hypothesis appear disconnected or do not exhibit any clear relationship. vigilant while dealing with ambiguities, and strive decode them in the context of the hypothesis. Do not allow nuanced or hypothetical statements distract from identifying the primary idea in the hypothesis. Know that your classifications, 'entailment', 'contra- diction'. or 'neutral', should mirror the essential relationship derived strictly from the premise and the hypothesis, without the influence personal opinions conclusions. Prioritize understanding the core in- tention and context of the conversation over mere repetition words or phrases.
0.911
35


Reviews:
[{"review_id": "fWoXSxqtki", "reviewer": "Reviewer_gPc5", "summary": "The manuscript proposed PromptAgent, a new method using a planning algorithm, i.e., Monte Carlo Tree Search, to navigate and discover high-quality prompts through a process resembling human-like trial-and-error, incorporating feedback from model errors and refining previous prompts based on feedback. This method has been tested across 12 tasks with promising performance compared to existing baselines such as CoT and APE with GPT-3.5. The optimized prompt can be generalized to different LLMs, including GPT-4 and PaLM2.", "questions": "- If the optimizer LLM misidentifies an error or provides incorrect feedback, will this misinformation be propagated through the optimization process, leading to less effective prompts?\n- This study used GPT-3.5 as the base model and a more capable model, such as GPT-4, as the optimizer LLM. Why not use GPT-4 for both base and optimizer LLM?\n- The question above also extend to the implications of using fundamentally different LLMs, such as employing PaLM 2 as the base model against GPT-4 as the optimizer, and how this difference might affect the optimization outcome.\n- In Fig 3(a), why GPT Agent, an \"LLM-powered autonomous agent\", was not compared in the exploration efficiency test?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed PromptAgent leveraged a Monte Carlo Tree Search framework to utilize errors and feedback identified by LLMs for the iterative refinement of prompts. This approach is theoretically sound and can enhance navigation through the expansive search space of potential prompts.\n- PromptAgent showed promising experimental results across 12 tasks, and the optimized prompt can be generalized to different LLMs.\n- PromptAgent showed better performance and exploration efficiency than other prompt optimization methods, including Automatic Prompt Engineer (APE).", "weaknesses": "- PromptAgent relies on a key hypothesis: the optimizer LLM (GPT-4 in this study) possesses adequate domain knowledge to identify the errors in the response from the base LLM and give meaningful feedback. However, this may not be a valid hypothesis, especially in some specialized areas such as medicine [1,2], where the data is relatively sparse due to strict data protection regularization like HIPAA.\n- In order to refine the prompt, PromptAgent needs to concatenate the \"error_string\", \"error_summarization and \"trajectory_prompts\" as one input. Challenges may arise in tasks demanding the interpretation of extensive contexts, such as the analysis of detailed medical documents, where the \"state_transit\" could become prohibitively large due to the number of training examples and the depth of the Monte Carlo Tree Search, potentially diminishing the LLMs' performance.\n\nReference\n\n[1] Bhayana, R., Krishna, S., & Bleakney, R. R. (2023). Performance of ChatGPT on a radiology board-style examination: Insights into current strengths and limitations. Radiology, 230582.\n\n[2] Azizi, Z., Alipour, P., Gomez, S., Broadwin, C., Islam, S., Sarraju, A., ... & Rodriguez, F. (2023). Evaluating Recommendations About Atrial Fibrillation for Patients and Clinicians Obtained From Chat-Based Artificial Intelligence Algorithms. Circulation: Arrhythmia and Electrophysiology, e012015.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "13zVQ1rDdF", "reviewer": "Reviewer_WJJm", "summary": "This paper proposes PromptAgent, which adopts a planning strategy based on MCTS for prompt engineering. Empirically, PromptAgent outperforms prior methods and human prompts. Overall the reviewer thinks the manuscript is well written and solid, and would like to recommend for acceptance.", "questions": "1. At the end of Section 3.1, the manuscript says “PromptAgent straightforwardly defines a reward function $r_t=r(s_t,a_t)$ as the performance on a held-out set separated from the given training samples.” However, the reviewer cannot see how the reward functions are actually defined in the experimental sections (and the appendix). Is it possible that the author can provide a clear definition of how the reward function is defined (or provide an example of how the reward function is generated)?\n2. In the paragraph “Baselines” of section 4.1, the descriptions of how Human Prompts are created are a bit vague. Although the authors have provided several examples of the human prompts in Appendix F, the reviewer would suggest the authors provide some extra details on how the human prompts are collected or generated.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. The method seems clean, straightforward, and promising.", "weaknesses": "There are several clarity issues in the experimental section regarding human prompts and reward functions. See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x1jyknFng7", "reviewer": "Reviewer_zmiy", "summary": "This work introduces a prompting agent designed to perform few-shot prompting. The seeking of better-prompting policies is based on recursively generation and self-reflection (using a stronger general-purpose LLM). The idea is straightforward, the results show some improvement over single-round prompting methods.\nThe idea is interesting, but not technically novel, and the results are not insightful enough to add knowledge to the community (please see the weakness section below). I hereby would vote for a rejection.", "questions": "Can the authors please provide the depth setting used in the Greedy baseline? It is too-sample efficient but performs poorly in Figure 3.(a). I also wonder what would the optimization burden be for each task compared to a DFS search. The beam search baseline implemented in the main text seems to be the BFS search. I would expect DFS to outperform BFS as it can integrate more of the LLMs’ ability of reflection and reasoning.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The idea of this work is clear and easy to follow. The writing is in general clear. This idea can be useful from the engineering/ deployment side.", "weaknesses": "Technical contribution is limited.\n\nComparing the performance of an average user with LLM in prompting is somewhat unfair. Also, even human experts will be posited under an unfair setting where LLMs can do multiple-round prompting.\n\nSome of the experiment settings are suspicious to be unfair (please see questions below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jNNdTLgmRG", "reviewer": "Reviewer_j86n", "summary": "The paper proposes a strategic prompt engineering method by utilizing Monte Carlo Tree Search where a base model collects errors from the training data set and an optimizer model provides feedback based on the collected errors to further optimize the prompt. Once a new prompt is composed, the hold-out set will be used to gauge its performance based on a given reward function. The method is evaluated on 12 curated tasks from three domains where results show its superior performance compared to a few alternative approaches including human curated prompt, Chain of Thoughts (CoT), and a couple of optimization methods including GPT agent and Automatic Prompt Engineering. Authors study the the effect of strategic planning aspect of their method through a set of ablation studies by considering various alternatives through single Monte Carlo Search, greedy, and Beam search where results show the benefit of considering exploitation vs. exploration trade offs in exploring the search space. Authors study prompt generalization and exploration efficiency as well.", "questions": "- Page 2: What does it mean when you say \"bad at managing multiple errors\"\n- It's not clear how the authors have picked the hyper parameters including number of iterations and exploration vs. exploitation related parameter.\n\nMinor:\np 1: prompting engineering => prompt engineering\nP 3: without as less as => with as less as", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to follow.\n- Related literature is covered.\n- Authors have conducted comprehensive experiments to evaluate the performance of their proposed method against a set of alternative methods and ablation studies shed further lights on the effectiveness of the proposed strategic exploration of the search space using MCTS.\n- Appendix covers useful details about the hyper parameters and the evaluation data sets and details of the studied methods.", "weaknesses": "- **Novelty** of the proposed method is arguable. Use of MCTS in prompt engineering is not novel. However, the way authors have incorporated a base and an optimizer model in their implementation and the prompts used to incorporate the error feedback into the existing state (+ the empirical studies) can be considered as the main contributions of this work.\n- **Contribution**: Based on the details provided in the Appendix section, it can be seen that the amount of details covered in prompts generated by APE is not comparable with that of the proposed method. The prompts generated by the proposed method are significantly lengthier and cover more details compared which is hard to justify. APE supposedly uses Monte Carlo Search to iteratively propose and select prompts, therefore, it is expected to see more details getting added to the original prompt over each iteration which is not reflected in the samples that I see in the Appendix section. **This makes me conclude that the additional gain from the proposed method by authors come from the way they instruct the optimizer method to incorporate the feedback into the existing state rather than utilization of MCTS** which is the main claim of the paper.\nHowever, \n- Clarity [minor]: It's ok that authors have covered the details of the selection, expansion, simulation, and back-propagation in MCTS and it can be very useful for general audience with less context, however, I was hoping to read more details about how authors have implemented the expansion stage. Appendexi briefly touches base under \"Implementation details\" sub-section and mentions \"We sample 3 batches, and for each batch, we generate multiple new prompts\". It would be good if authors can further explain how they generate new prompts for each batch.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)