Automatic Engineering of Long Prompts (2024)

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in solving complex open-domain tasks, guided by comprehensive instructions and demonstrations provided in the form of prompts. However, these prompts can be lengthy, often comprising hundreds of lines and thousands of tokens, and their design often requires considerable human effort. Recent research has explored automatic prompt engineering for short prompts, typically consisting of one or a few sentences. However, the automatic design of long prompts remains a challenging problem due to its immense search space. In this paper, we propose an algorithm named Automated Prompt Engineering Xpert (APEX), a novel algorithm that automatically improves long prompts. Leveraging a greedy algorithm with beam-search for efficiency, APEX utilizes search history to significantly enhance the effectiveness of LLM-based mutation in its search process. Our results show that APEX achieves an average of 9.2% accuracy gain on eight tasks in Big Bench Hard and a consistent improvements on GSM8K with various models, highlighting the significance of automating prompt designs to fully harness the capabilities of LLMs.

heading: 1 Introduction
text: Large language models (LLMs) have exhibited remarkable capabilities when trained on large datasets, demonstrating the ability to comprehend complex and lengthy instructions for diverse tasks without the need for fine-tuning (Wei et al., 2022a; Brown et al., 2020; Chowdhery et al., 2022; Ouyang et al., 2022). Several prompt design principles have emerged in recent years, suggesting that incorporating more complex instructions, demonstrations, and chain-of-thought reasoning into prompts can boost the performance on challenging tasks (Brown et al., 2020; Wei et al., 2022b), including those involving mathematical
problem-solving (Cobbe et al., 2021) and reasoning (Suzgun et al., 2022; Srivastava et al., 2022). However, effective prompts for tackling complex tasks often contain thousands of tokens, posing challenges in designing and optimizing them. Figure 1 demonstrates a long prompt for a task in Big Bench Hard (Suzgun et al., 2022), which contains an instruction and several demos, each with humanwritten chain-of-thoughts.
Numerous studies have demonstrated the sensitivity of LLMs to prompts, revealing that minor modifications, such as adding or removing a few tokens or rephrasing the prompt, can significantly impact LLM performance (Liu et al., 2023; Zhu et al., 2023; Jiang et al., 2020). Therefore, prompt design has become a labor-intensive endeavor. Further complicating the matter is the rapid evolution of LLMs, rendering prompts crafted for a specific LLM ineffective when applied to a newer version of LLM. This highlights the need for automatic prompt engineering techniques.
In this paper, we consider the problem of automatic prompt engineering for black-box LLMs, where the tuning algorithms will improve prompts by querying the model. While automatic prompt engineering has been studied recently, existing research (Deng et al., 2022; Xu et al., 2022; Guo et al., 2023; Fernando et al., 2023) focuses on optimizing short instructions with one or a few sentences. These methods either evolve prompts by searching for word replacements or utilize LLMs to rewrite the entire prompt (Xu et al., 2022; Fernando et al., 2023; Guo et al., 2023; Yang et al., 2023). It is challenging to apply them for evolving a long prompt like Figure 1. For long prompts, word replacement-based search faces a large search space, while rewriting the entire prompt using a single LLM query is extremely difficult. Although it is possible to apply these methods to individual sentences, we have observed that tuning a single sentence may not sufficiently improve the perfor-
10672
mance of long prompts (see Section 4). Therefore, how to automatically find better long pompts and what’s the potential performance gain by tuning long prompts remain unanswered in the literature.
In this paper, we propose an algorithm named Automated Prompt Engineering Xpert (APEX). The main operation in APEX is to replace a sentence by a semantically equivalent sentence, which can be done by queries to LLMs. We adopt a greedy algorithm with beam search to make the prompt search more efficient. Further, for long prompt engineering it is important to identify which sentence to change, and to what direction. We thus propose two novel techniques that utilize search history to enhance the effectiveness of LLM-based mutation in our algorithm.
Our contributions can be summarized below:
• This paper presents the first formal discussion of automatic long prompt engineering, demonstrating substantial performance gains on multiple tasks. • We propose a greedy algorithm with beam search that can rapidly optimize prompts. Furthermore, a novel guided mutation method is introduced to enhance convergence. • We conducted experiments on the Big Bench Hard (BBH) benchmark, where prompts comprise thousands of tokens, including instructions and chain-of-thought reasoning. Our results demonstrate that the proposed auto-
matic long prompt engineering method significantly enhances performance. Notably, we achieved an average of 9.2% absolute accuracy improvements on 8 tasks selected from BBH, as shown in Figure 1, with only 50 evaluations on the training set. We further demonstrate the effectiveness of APEX on GSM8K across various LLMs.

heading: 2 Related Work
text: Designing effective prompts to fully exploit the capabilities of LLMs has become an important topic. Many principal ways for prompt design have been studied recently (Reynolds and McDonell, 2021; Brown et al., 2020; Wei et al., 2022b; Wang et al., 2022a, 2023a; Jiang et al., 2020). For instance, a well-designed prompt may include a system prompt, an instruction prompt outlining the task, multiple contextual examples, and a chain-ofthoughts reasoning section that explains the stepby-step thought process behind the examples. By incorporating these elements, effective prompts often have tens of sentences and thousands of tokens. Prompt engineering has been used to not only improving the performance of LLMs, but also to achieve certain goals, such as robustness testing (Wallace et al., 2019) and jailbreaking (Zou et al., 2023; Chao et al., 2023).
An orthogonal line of previous work has explored soft-prompt tuning, a technique that op-
timizes prompts within a continuous embedding space using standard continuous optimization algorithms (Lester et al., 2021; Zhang et al., 2021; Wang et al., 2022b). While capable of achieving satisfactory performance, soft-prompts lack interpretability and cannot be applied via LLM APIs. Additionally, these parameter-efficient fine-tuning methods demand large training sets, making them unsuitable for applications with limited data, such as those with only tens or hundreds of samples.
Given the limited availability of training data (e.g., < 1000 samples), our focus lies in exploring strategies for optimizing hard prompts, which are semantically equivalent to the original prompts but yield superior performance. The literature of automatic prompt engineering considers two primary settings. The first setting, which aligns with our work, assumes the existence of an initial humancrafted prompt and aims to refine or improve it to achieve enhanced performance. Several discrete search algorithms have been proposed for this setting: (Shin et al., 2020) searches for the trigger tokens that can improve the performance of language models on downstream tasks. (Xu et al., 2022) employs a genetic algorithm for prompt tuning, utilizing back translation, cloze tasks, and sentence continuation to mutate the initial instruction. (Pryzant et al., 2023) explored the utilization of input-output pair feedback to refine instructions in an iterative manner. More recently, (Fernando et al., 2023; Guo et al., 2023) proposed leveraging LLMs for mutation and crossover operations in evolutionary searches. Although they can also be applied to entire prompt, we show in the main experiment that the proposed algorithm is more effective in evolving an entire prompt. This justifies the proposed ideas (history-guided evolution/sentence selection) can effectively isolate the important part of long prompt to evolve.
In addition, (Yang et al., 2023) demonstrated the optimization capabilities of LLMs in generating improved prompt variations based on previous fitness scores. However, these prompt evolution techniques are designed for short sentences or paragraphs within a long prompt. For example, many of them try to evolve only the instruction part or the sentence “Let’s think step by step” in the prompt. Our work aims to provide complete freedom to evolve the entire long prompt, opening up more avenues for improvement but also introducing challenges in determining how and where to change the
original prompt. Other recent concurrent works have also used the idea of LLM as agents to optimize prompt. (Wang et al., 2023b) proposed to model prompt optimization as a state-transition process, where at each state the LLM uses the predicted error to generate feedback to and select next action. This work has a different search space with us, where they aim to correct existing prompts instead of rewriting it. Another work that published after this paper focuses on incorporating human feedback into prompt optimization (Chen et al., 2024).
Another setting focuses on automatic prompt generation without a pre-existing prompt. (Honovich et al., 2022) demonstrated the ability of LLMs to generate brief task descriptions based on input-output pairs. Building upon this technique, (Zhou et al., 2022) proposed an automatic prompt engineering (APE) algorithm capable of generating prompts from given pairs. Under the same problem setting, (Chen et al., 2023) developed a continuous relaxation approach. Beyond these settings, there exists some algorithms developed to automate the generation of chain-of-thought prompts (Zhang et al., 2022). Our work seeks to design a general prompt tuner, which is orthogonal and can be combined with those instruction learning or auto chain-of-thoughts methods.

heading: 3 Proposed Method
text: In this paper, we address the challenge of automatic long prompt engineering for language models. Given a language model L and an initial prompt pinit designed by human for a particular task, our goal is to refine the prompt to achieve superior performance. To enhance the prompt, we are provided with a limited training set (e.g., 100 input-output pairs) {(xi, yi)}ni=1 for performance evaluation. Specifically, for each sample we conduct prediction by L([p, xi]) and assess its agreement with the corresponding ground truth label yi. We define score(p) as the performance metric of prompt p on the training set. We consider the scenario where the number of available samples n is limited, which is a common situation where individuals lack sufficient data for model fine-tuning but can still utilize the data to design an improved hard prompt. We will evaluate generalization performance on a hold-out test set that is not used in prompt search.

heading: 3.1 Search space
text: Our goal is to generate a new prompt that is semantically similar to the original prompt while achieving enhanced performance. We avoid introducing non-interpretable tokens, such as adversarial triggers (Zou et al., 2023). The restrictions offer two advantages: 1) the prompts discovered by our algorithm are interpretable, facilitating the verification that the prompt still performs the intended task, and 2) since we are only provided with a limited set of training samples, constraining the search space can mitigate the issue of overfitting.
To conduct prompt search, we decompose the long prompt into m individual sentences:
p = [s(1), s(2), . . . , s(m)].
We then allow each sentence to be rephrased while preserving its semantic meaning. Since LLMs excel at sentence rephrasing, we adopt LLM to generate semantically equivalent variation of a given sentence and call it LLM-Mutator. The prompt we used for vanilla LLM mutator can be found in Figure 2. An improved version of LLM-Mutator will be introduced later in Section 3.3.

heading: 3.2 Search algorithm
text: A straightforward approach to conduct search using the LLM-Mutator is the following greedy algorithm. At each iteration, we randomly select a sentence, denoted by s(i), and utilize the LLMMutator to generate an alternative sentence s′(i). We then replace s(i) by s′(i) in the old prompt p if the new resulting prompt improves the performance. However, we observed that this vanilla greedy approach can be easily trapped in local optima. In our problem, the training set is so limited that sometimes a detrimental modification is not reflected in the training score. As a result, unfavorable edit are sometimes accepted due to insufficient evaluation, which hurts the prompt’s future improvement.
To address this issue, we propose conducting a beam search by maintaining a pool of k topperforming prompts, denoted as p∗1, p ∗ 2, . . . , p ∗ k. At each iteration, we randomly select one of these k prompts and refine the chosen prompt. We then evaluate this new candidate and maintain the top-k
prompt pool. This approach ensures that even with the introduction of detrimental edits, recovery from such errors is still possible as we retain not only the top prompt. Our experiments reveal that this approach leads to significantly improved training and test performance compared to the pure greedy algorithm, as demonstrated in Figure 3. Further discussions of the implementation details and comparison between beam search and greedy methods are in Appendix 7.1.
It is worth noting that our method is closely related to the Genetic Algorithm (GA). GA is a widely recognized method for discrete optimization involving black-box functions. Assuming P = {p1, . . . , pk} represents the solutions in the current pool, GA applies mutation and crossover operations to this pool to generate a set of newly proposed candidates P ′ = {p′1, . . . , p′k}. The fitness scores (performance of the solutions) of these newly proposed candidates are then evaluated. Subsequently, only the top k solutions in P ∪ P ′ are retained before proceeding to the next iteration.
While GA possesses a high exploration capability, it is slower in improving training accuracy in the first 50 iterations. However, given the vastness of our search space and the reasonably good quality of the human-written initial prompt, a majority of mutations and crossovers performed in the first few iterations yield poorly performing candidates. Evaluating each of these candidates is computationally expensive, even though it can enhance solution diversity.
Our algorithm is effectively a “greedy” version of GA. We maintain a candidate set of size k and update the candidate pool immediately upon generating each new candidate, rather than waiting for the evaluation of the entire generation of offspring before updating the pool. As a result, our solution pool remains more up-to-date, leading to faster convergence compared to GA. Figure 3 further verifies this observation, demonstrating the slower initial growth of GA’s learning curve.

heading: 3.3 History-guided search
text: Randomly rephrasing a sentence in a prompt is akin to conducting a random mutation within the space of semantically equivalent sentences, which is not very efficient. This subsection introduces our main innovation, where we show how the search history can be employed to guide the mutation in a more purposeful direction.
Guided mutation for a single sentence. At iteration T , we can denote the search history as {(sbeforet , saftert , rt)}T−1t=1 , where sbeforet is the sentence before mutation, saftert is the sentence after mutation, and the reward rt indicates the change in the score (rt is positive if the mutation enhances the performance, and negative otherwise). Assume sT is the current sentence that is selected for mutation, we can then use the history to guide the mutation towards more positive reward. For example, if we know rephrasing the sentence “So, it is true that Lesley is a great-grandfather of Leroy” to “Therefore Leslie is Leroy’s great-grandfather.” can improve the performance, then for another sentence “So, it is true that everyone who is an ancestor of Dana is a stepbrother of Brian” we may want to rewrite it as “Therefore everyone that is an ancestor of Dana is Brian’s stepbrother.”
It has been shown that LLMs are able to learn from in-context examples (Zhang et al., 2021). Therefore, we propose to let the LLM-Mutator incontext learn from the history. This can be done by listing history in the prompt: if rt > 0, we include an in-context example sbeforet ⇒ saftert , and if rt < 0 we include saftert ⇒ sbeforet . The prompt for LLM-Evolver is shown in Figure 4.
Since the history can be long and it has been shown that including too many in-context examples can be harmful, we only retrieve a small set of
relevant history entries when rephrasing a sentence. Specifically, when evolving sT , we compute the similarity between sT and each sbeforet , which is calculated by φ(sT )Tφ(sbeforet ) and φ is a sentence encoder (we use a T5 (Raffel et al., 2020) in our experiments). We then only select entries that pass a certain threshold to include as in-context examples.
Guided sampling for sentence selection. Long prompt typically contains tens or hundreds of sentences, posting a challenge to the search algorithm. Our experiments demonstrate that altering only a few sentences in a long prompt can significantly enhance its effectiveness, but the changes need to be at right places. It is thus beneficial to bias the sampling distribution towards selecting more impactful sentences for modification.
We model sentence selection as a contextual bandit problem (Langford and Zhang, 2007). In contex-
tual bandit, at each iteration a learner is faced with m arms associante with feature vectors x1, . . . , xm, and the learner is trying to pull an arm at each round to optimize the overall reward. In our case, m arms correspond to the m sentences s(1), . . . , s(m) in a prompt, and features can be obtained by φ(s(i)) where φ is a text encoder. Utilizing feature information is crucial because if modifying a particular sentence can lead to performance improvement, it is likely that modifying similar sentences will also yield positive results.
We then adopt the Lin-UCB algorithm (Li et al., 2010) to guide sentence selection. Given the history {(sbeforet , saftert , rt)}T−1t=1 , we compute a linear estimator of the underlying reward model
w∗T = (H TH + λI)−1HT r, (1)
where H is a (T − 1)-by-d matrix with each row being φ(sbeforet ), and r = [r1, . . . , rT−1]
T . This is simply a solution of a ridge regression with feature matrix H and reward r. The following UCB rule is then used to select the sentence for mutation:
argmax i φ(s(i))
Tw∗T + α √ φ(s(i))TA−1φ(s(i)),
(2) where A = HTH + λI . Since this Lin-UCB estimation may not be accurate, we let the algorithm has probability P to choose a purely random arm and 1− P to choose the sentence based on Eq. (2). The algorithm is summarized in Algorithm 1.
Algorithm 1 Automated Prompt Engineering Xpert (APEX) p0: initial prompt; k: beam size; θ: threshold. Initialize solution pool P as {p0} for iter = 1, 2, . . . do
Randomly select p from top-k solutions in P p← random number in [0, 1] if p < 0.5 then
Sample a random sentence s(i) Choose s(i) by Eq. (2).
end if Compute T = {t : φ(S(i))Tφ(sbeforet ) > θ} s′(i) ← Guided Mutation on s(i) with history T Evaluate p′ = [s(1), . . . , si−1, s′i, si+1, . . . ] Add p′ to P
end for

heading: 4 Experimental Results
text: In this section, we present empirical evidence demonstrating that APEX can significantly enhance performance on the Big-Bench Hard (BBH) benchmark (Suzgun et al., 2022; Srivastava et al., 2022). We then conduct ablation studies and test the performance on GSM8K dataset across various LLM models. Finally we show qualitative results on how APEX refines human-written prompts.

heading: 4.1 Experimental Settings
text: We consider the prompt developed in Suzgun et al. (2022) for the BBH tasks, where prompts consist of two parts: Task Description and Demos. The Task Description provides instructions describing the task, while each demo includes the question, a chain-of-thoughts demonstration illustrating the problem-solving process step-by-step, and the final answer. During prompt tuning, the format (e.g., “Question. ”, “Answer. ”) are retained, while the tuning algorithm is allowed to modify any other parts including the instruction part and the chainof-thoughts part. Table 1 summarizes the statistics of the datasets used in this study. The full prompts for each data can be downloaded online1.
Each task consists of 250 samples, randomly divided into 50% for training and 50% for testing. Only training samples are utilized for prompt tuning. All tasks are multi-class classification problems, and we report accuracy on training data, test data, and the combined dataset when comparing different tuning methods.
For most of the experiments, we evaluated the performance on text-bison model and utilized the instruction-tuned PaLM 2-L model as LLM mutator. Both models belong to the PaLM 2-model family (Anil et al., 2023). We also demonstrate that our method can be applied to other LLMs, where we conduct experiments on GPT-3.5 in Section 4.3.1. The GPT version used in our experiment is gpt-3.5-turbo-instruct-0914. Following the previous prompt tuning works, we set temperature as 0. To enhance the diversity of sentence mutation, we set the temperature of the LLM-Mutator to 0.5.
We compare the proposed algorithm with the following baselines:
• Original Prompt: Performance of the original human-designed prompt developed in (Suzgun et al., 2022), which also serves as the
1https://github.com/suzgunmirac/ BIG-Bench-Hard
initialization for other tuning methods. • Genetic Algorithm: An implementation of the
genetic algorithm for long prompt tuning. To facilitate a more comparable comparison with our method, we set the pool size to 4. At each step, 8 new candidates are generated by randomly mutating and performing crossover on the top candidates within the pool. • Evolve “step-by-step”: Several recent prompt tuning studies have explored the concept of evolving a prompt using a single sentence, such as the ’Let’s think step by step’ sentence employed in chain-of-thought prompts (Kojima et al., 2022). We adopted one of the state-of-the-art methods for single sentence optimization (Yang et al., 2023) to optimize all of the ’Let’s think step by step’ sentences within the chain-of-thought prompt. • APO (Pryzant et al., 2023) designed a prompt optimization algorithm which expands the instruction space by a feedback loop. We apply this algorithm to directly tune long prompts. • PromptBreeder (Fernando et al., 2023) uses a genetic algorithm to search for prompts. We incorporate LLM-based mutation and crossover without optimizing the meta prompt. • Greedy: A simple greedy approach mentioned in Section 3.2, where we only store the top performing candidate in the pool and generating new candidate on top of it.
For our method (APEX), we use the same hyperparameters for all the experiments. We set the pool size (beam size) to 4 for all experiments. When applying guided mutation in Section 3.3, we use a T5 encoder to encode both the current entry and the history and normalize the embeddings to have a unit `2 norm. We retrieve only the top 4 history entries and require the `2 distance between the encoded history and the current sentence to be below 0.5. When applying the sentence selection algorithm described in Section 3.3, we set α = 0.05 in Eq. (2) and set P = 0.5. Note that those hyperparameters are not well tuned.

heading: 4.2 Main Results: Results on Big Bench Hard
text: For this experiment, we limited the computational budget to 50 evaluations on the training set and reported the training, test, and combined accuracy achieved by each method. All the experiments are run three times and we report the mean and standard deviation. The results are summarized
in Table 2. APEX outperform all other methods, demonstrating significant improvements in accuracy across all tasks. We also compute the improvements of accuracy on the whole dataset (including both training and testing) and report the accuracy gain in Figure 1.
Across all 8 tasks, APEX achieves an average of 8.2% gain in test accuracy and 9.2% gain in the accuracy of full evaluation set (train + test). Among these tasks, we achieve the largest performance gain (18.45%) on the logical deduction task, and the smallest gain (2.45%) on reasoning about colored objects (Color Reasoning). One potential reason of the marginal gain on color reasoning is that the original prompt already achieves high accuracy, so there is not much room for improvements.
Comparing the baselines, it becomes evident that evolving a single sentence (Evolve ’step-by-step’) fails to achieve substantial improvements in long prompt tuning. This is mainly due to the fact that long prompt tuning typically involves over 20 sentences with detailed instructions and explanations, so naively modifying “Let’s think step by step” is unlikely to lead to significant enhancements. Furthermore, the Genetic Algorithm and Greedy Algorithm each exhibit their own limitations, as discussed in Section 3.2.
On the other hand, we show that conducting APO over the long prompt leads to suboptimal results, often lower than GA and Greedy (baselines for long prompt tuning). This is mainly due to the difficulty of rewriting the whole long prompt. For example, the original prompt of Hyperbaton contains 19 sentences with more than 500 words, but most of the prompt generated by APO has only <50 words. As a result, it couldn’t find a better prompt. This explains why evolving the whole long prompt together can fail.
Despite being able to significantly boost the per-
formance, we also observe some degree of overfitting in our search procedure. Although the proposed beam search method can partially mitigate overfitting (see Section 3.2), we still observe higher training accuracy than testing in most cases. However, as the performance gain is substantial, the prompts found by the algorithm are still significantly better on the test set.
Can previous methods handle long prompts? Although many previous methods can be directly applied to long prompts, our empirical finding in Table 2 demonstrated that they cannot achieve good performance by rewriting the entire prompt. In particular, the APO and PromptBreeder algorithms in our comparison directly rewrote the entire prompt and achieve lower performance than APEX. Another option is to use existing work and tune only a particular sentence. In Table 2, we show that tuning the “step-by-step” sentence by (Yang et al., 2023) is insufficient, and further, we conduct another experiment in Table 3 showing that only tuning the instruction part of the prompt is insufficient.
Can we directly use training samples for incontext learning? In our method and all the baseline prompt tuning algorithms, the training samples are used for obtaining a better prompt. However, another way to utilize training samples is to add all of them as demos in the in-context learning framework. To compare with this approach, we show the results of using all training samples in In-context Learning (ICL) and show the result in Table 3. We can see adding all samples in ICL improves the performance in some tasks (especially in Color Reasoning), but could also degrade the performances. Overall, APEX is still more effective in most of the cases comparing with ICL with all training samples.

heading: 4.3 Ablation Study
text: We study APEX under different LLMs, datasets, and by varying the components in APEX.

heading: 4.3.1 Different LLMs and datasets
text: To show the performance of APEX under different settings such as different datasets and LLMs, we conduct experiments on the GSM8K dataset (Cobbe et al., 2021) using different LLMs. We choose the “prompt_hardest” prompt2 as the initial prompt which contains 103 lines with 1,639 words. We conduct prompt tuning with 1,000 sampled training data for our method and evaluate the revised prompt on the full test set. For text-bison and Palm-2-L we use Palm-2-L as the mutation LLM, while for GPT-3.5 we use GPT-3.5 as the mutation LLM. The results are shown in Table 5. We observe that the proposed method consistently improves the performance with different LLMs including text-bison, GPT-3.5-instruct and Palm 2-L.

heading: 4.3.2 Different components of the algorithm
text: We conduct an ablation study on the two techniques introduced in Section 3.3: the history-guided mutation and the contextual bandit algorithm for sentence selection. To verify whether both techniques are essential in APEX, we consider two settings: APEX without history-guided mutation (during mutation, we randomly rewrite a sentence using the prompt in Figure 2) and APEX without sentence selection (randomly select a sentence to mutate, but using guided mutation Figure 4 at each iteration). All the hyperparameters are fixed in this experiment. The results are presented in Table 4. We can observe that both components are contributing to the final performance of the model.
2https://github.com/FranxYao/ chain-of-thought-hub

heading: 4.4 Qualitative results
text: One important benefit of automatic hard prompt engineering is that the resulting prompts remain interpretable by humans, allowing users to easily verify the modifications. We provide some successful examples found by our search in Table 6 and Table 7 (appendix). In each table, we show only the different parts of the initial prompt from BBH and the changes made by our method.
The first example demonstrated in Table 6 (appendix) is for the logic deduction task on five objects. The initial prompt achieves 38.8% accuracy while the revised prompt found at iteration 48 improves the performance to 57.9% train accuracy and 54.0% test accuracy. We observed that most of the changes involve minor revisions to the original sentence without altering its meaning. These seemingly insignificant modifications can lead to substantial improvements in LLM accuracy, showcasing the important of automatic long prompt engineering. We also include another example in Appendix 7.2.

heading: 5 Conclusions
text: We study the problem of automatic prompt engineering for long prompts, often comprising thousands of tokens. We investigate the performance of the standard greedy algorithm and genetic algorithm, and develop a search algorithm that yields superior performance. With only 50 evaluations on the training set, our method achieves an average absolute accuracy improvement of 9.2% across 8 tasks from Big Bench Hard. This demonstrates
the significant potential benefits of automatic long prompt tuning and underscores the importance of this emerging area.

heading: 6 Acknowledgements
text: We thank Ruochen Wang, Vineet Gupta, Kedar Dhamdhere, Daliang Li for valuable discussions and feedback.
Limitations
As the first paper focusing on automatic engineering of the entire long prompts, we identify several limitations of the current methods which can lead to interesting future research:
• The current algorithm relies on using another LLM to rephrase a sentence. As illustrated in Section 4.4, this LLM-mutator may introduce errors during sentence rewriting, particularly for intricate sentences (e.g., CoT in complicated logical deduction tasks). Therefore, improving the “correctness” of LLM-Mutator is an interesting future area of research, which has not been fully addressed in our work as well as other recent studies (Fernando et al., 2023; Guo et al., 2023). • In the current implementation, we break down the long prompt into individual sentences and modify one sentence at a time. However, it might be beneficial to manipulate multiple sentences simultaneously during mutation or consolidate multiple sentences into a single one. An automated mechanism for carrying out this process would be an interesting direction for enhancing our method. • Although our algorithm is able to find a good solution with less than 100 evaluations on the training set, the cost is still not negligible especially when tuning prompts using APIs with
cost or rate limits. Employing early stopping techniques, where the evaluation of poorly performing candidates are terminated early, could potentially reduce the number of queries. • Although automatic prompt engineering can achieve significant gain, the search space of hard prompt has limited representation power which hinders further performance improvements. It has been shown that soft prompt tuning has limited representation power (Wang et al., 2023c), and our search space is a small subset of soft prompts. Therefore, when provided with sufficient data, computational resources, and white-box access to the LLM, (parameter-efficient) fine-tuning may still achieve superior performance.

heading: 7 Appendix

heading: 7.1 Comparison between greedy, beam search and GA.
text: Here we include more implementation details of greedy v.s. beam search algorithm for reproducing Figure 3, and discuss why beam search performs significantly better.
In the implementation of greedy algorithm, we maintain the top candidate (based on training accuracy), and randomly mutate a sentence of the top candidate at each iteration. We then evaluate the new prompt on training data. If the new prompt achieves better training accuracy than the previous top prompt, we replace the top prompt with this new prompt. Otherwise we keep the original top prompt. The process is simple and there is no hyperparameter. On the other hand, the beam search algorithm maintains a beam size of k. At each iteration, it randomly selects a prompt from the beam, and randomly mutates one sentence to generate a new prompt. We then evaluate the prompt, and decide whether we need to update the candidates in the beam. Therefore, the only hyper-parameter is k and we set it to 4 in all our experiments. We found the algorithm is not very sensitive to k, as long as it is not too large.
We observe greedy search is very easy to saturate (accuracy not improving anymore). The main difference between greedy search and beam search is that the greedy search always through away all other candidates except the one with top training accuracy. However, the search will stuck when this top candidate has good training accuracy but bad generalization accuracy (so it achieves good training accuracy by “luck” and is actually a bad prompt). In this case, all the later prompts in the greedy algorithm are generated from this “bad” candidate which leads to bad performance, and it is hard to improve anymore. In contrast, the beam search algorithm always has a chance to recover from those “bad” candidates, since there are 4 candidates in the pool and we always have chance to derive later prompt from another candidate in the beam.
We also include more details for reproducing Figure 3. As described above, Greedy doesn’t have any hyper-parmaeter. Beam search has a hyperparameter k (beam size) which is set to 4 for all experiments in this paper. In order to make Genetic Algorithm (GA) comparable with beam search (see more details in Section 3.2), we maintain a pool
of size 4 for genetic algorithm. At each iteration, we randomly generate 4 candidates by randomly mutating from the pool, and another 4 candidates by cross-over. We have tried to further increase these numbers, and the GA will have even slower progress in the initial phase. For example, if we have 8 mutations and 8 cross-overs, then 1 iteration will require 16 evaluations (and it can be observe that beam search already achieves a high performance after 40-50 evaluations).

heading: 7.2 Additional Qualitative Result
text: The second prompt in Table 7 is for the Formal Fallacies task, designed to identify whether a given logic statement is valid. The initial prompt achieves 60% accuracy while the revised prompt found at iteration 91 improves the train accuracy to 92.1% and test accuracy to 83.1%. Similar to the previous case, most of the changes involve minor revisions. In this case, we also want to highlight a potential limitation of the proposed method that could be addressed in future work. In the sentence marked as *, the revised sentence is not semantically equivalent to the original one. The original and revised sentences represent different logical statements (original sentence: not A → B; new sentence: B → not A). However, the LLM appears incapable of detecting this subtle distinction, leading to an erroneous rephrasing. Although this change leads to improve training accuracy, it actually hurts test accuracy. Therefore, incorrect mutations can lead to overfitting, and developing strategies to mitigate these errors during the search process would be an interesting area of future research.

heading: 7.3 Experimental results with longer runs
text: In Table 2 we have demonstrated the results with 50 evaluations. Here we further show the performance of our algorithms versus several baselines when running with 100 iterations. The results are shown in Table 9. We can observe that APEX still outperforms other methods with 100 iterations.

heading: 7.4 Ablation study on in-context example selection
text: In this subsection, we conduct an experiment to demonstrate that having too many in-context examples is harmful for guided mutation, which justifies the proposed history selection algorithm. We compare the original APEX with two variants: APEX (random history) which has the same number of
in-context examples as APEX but select the examples randomly, and APEX (random 20 history) which randomly selects 20 in-context examples for guided mutation. The results in Table 8 demonstrates that (1) retrieval based sample selection as proposed in Section 3.3 leads to 1-2% improvements, and (2) selecting too many examples will hurt the performance significantly.

heading: Future Work
text: In this case, we also want to highlight a potential limitation of the proposed method that could be addressed in future work. In the sentence marked as *, the revised sentence is not semantically equivalent to the original one. The original and revised sentences represent different logical statements (original sentence: not A → B; new sentence: B → not A). However, the LLM appears incapable of detecting this subtle distinction, leading to an erroneous rephrasing. Although this change leads to improve training accuracy, it actually hurts test accuracy. Therefore, incorrect mutations can lead to overfitting, and developing strategies to mitigate these errors during the search process would be an interesting area of future research