Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Authors: Jianyu Wang
Zhiqiang Hu
Lidong Bing
Venue: ICML 2025 poster (2025)
Keywords: Prompt Optimization
In-context Learning
AI Alignment
Self-improvement
Open-Endedness
Primary area: deep_learning->large_language_models
TL;DR: A novel prompt design paradigm for LLMs, resembling the open-ended nature of biological evolution.

Abstract: We propose a novel prompt design paradigm that challenges conventional wisdom in large language model (LLM) prompting. While conventional wisdom prioritizes well-crafted instructions and demonstrations for in-context learning (ICL), we show that pruning random demonstrations into seemingly incoherent ''gibberish'' can remarkably improve performance across diverse tasks. Notably, the ''gibberish'' always matches or surpasses state-of-the-art automatic prompt optimization techniques, achieving substantial gains regardless of LLM alignment. Nevertheless, discovering an effective pruning strategy is non-trivial, as existing attribution methods and prompt compression algorithms fail to deliver robust results, let alone human intuition. In terms of this, we propose a self-discover prompt optimization framework, PromptQuine, an evolutionary search framework that automatically searches for the pruning strategy by itself using only low-data regimes. Much like the emergent complexity in nature—such as symbiosis and self-organization—arising in response to resource constraints, our framework evolves and refines unconventional yet highly effective prompts by leveraging only the tokens present within the context. We demonstrate its effectiveness across classification, multi-choice question answering, generation and math reasoning tasks across LLMs, while achieving decent runtime efficiency. We hope our findings can guide mechanistic studies on in-context learning, and provide a call to action, to pave the way for more open-ended search algorithms for more effective LLM prompting.

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Jianyu Wang 12 Zhiqiang Hu 12 Lidong Bing?
Abstract
1. Introduction
Prompting large language models (LLMs) has become the de facto standard for numerous applications, shifting the community focus designing prompts that maximize model performance. However, this task is inherently com- plex due to the nuanced and often unpredictable behavior of LLMs (Lu et al., 2022: Liu et al., 2023). Subtle changes in phrasing, structure, or context can dramatically affect outputs (Jiang et al., 2020; Shi et al., 2023). Consequently, prompt engineering relies heavily on iterative experimenta- tion and evaluation. On the other hand, automatic prompt optimization (Liu et al., 2023) explores minimize the human involvement by leveraging computations to refine prompts iteratively. The conventional wisdom (Zhao et al., 2021; Lu et al., 2022; OpenAI, 2023; Wan et al., 2024) sug- gests that well-specified task instruction, combined with a few tuned demonstrations for ICL,, yields the best results.
We propose a novel prompt design paradigm that challenges conventional wisdom in large language model (LLM) prompting. While con- ventional wisdom prioritizes well-crafted instruc- tions and demonstrations for in-context learning (ICL), we show that pruning random demonstra- tions into seemingly incoherent "gibberish" can remarkably improve performance across diverse tasks. Notably, the "gibberish" always matches or surpasses state-of-the-arr automatic prompt optimization techniques, achieving substantial gains regardless of LLM alignment. Neverthe- less, discovering an effective pruning strategy is non-trivial, as existing attribution methods and prompt compression algorithms fail to deliver ro- bust results, let alone human intuition. In terms of this, we propose a self-discover prompt op- timization framework, PROMPTQUINE, an evo- lutionary search framework that automatically searches for the pruning strategy by itself using only low-data regimes. Much like the emergent complexity in nature-such as symbiosis and self- ariiis in response to resource con- straints, our framework evolves and refines uncon- ventional yet highly effective prompts by leverag- ing only the tokens present within the context. We demonstrate its effectiveness across classification, multi-choice question answering, generation and math reasoning tasks across L.Mss, while achiev- ing decent runtime efficiency. We hope our find- ings can guide mechanistic studies on in-context learning, and provide a call to action, to pave the way for more open-ended search algorithms for more effective LLM prompting.
This paper presents study that challenges conventional wis- dom by showing that pruning clear, detailed demonstrations into seemingly incoherent "gibberish" (both syntactically and semantically strange) can, counterintuitively. improve performance across various tasks. Notably, this effect gen- cralizes across models, regardless of alignment (Shen et al., 2023b), suggesting a broader misspecification of unnatu- ral language in current LLMs. Even more surprisingly, we find that this "gibberish" consistently matches surpasses the performance of state-of-the-art automatic prompt opti- mization results in several tasks. Consequently, we propose a novel conceptual framework that reframes prompt com- pression as guided prompt search, enhancing both serving efficiency and task performance. We further explore algo- rithms to achieve these improvements.
To derive effective pruning strategies, one might expect existing instance attribution (Li et al., 201 6a; Yin & Neu- big. 2022) or prompt compression methods (Li et al., 2023; Jiang et al., 2023c;d; Pan et al., 2024) to provide guidance. However, we find that none of these methods can reliably produce accurate token importance scores to guide the prun- ing, let alone human intuition according to word semantics. A more practical idea is to ask algorithms to discover the pruning strategies by themselves. We thus develop self- discover prompt optimization framework, Genetic Prompt- Quine (PROMPTQUINE), an evolutionary search framework
This work was partially done when Lidong Bing was at DAMO Academy, Alibaba Group. 'DAMO Academy, Alibaba Group HHupan Lab 'MiroMind. Correspondence to: Jianyu Wang @wwl00@ucsd.cuu.>
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Evolving Prompts In-Context: An Open-ended, Selfrrriiin Perspective
that automatically searches for the pruning strategy by it- self using low-shot regimes. The philosophy of this frame- work is essentially self-replicating program (Von Neu- mann et al., 1966) that copies and mutates the prompt itself (e.g., pruning random tokens). The mutated prompts com- pete for limited resources to survive based on their fitness, with only the most performant surviving, thereby evolving effective pruming strategies over multiple generations. The entire process closely resembles the evolutionary dynamics of biological systems, where self-replication and adapta- tion drive the emergence of more effective strategies over successive generations (i.e., evolutionary self-replication).
to be conditioned to infer the task results. Formally, given K input-label pairs {(xx, y:) and the task test input Xtes concatenated in the input context, the LLM is conditioned to generate the task prediction:
Yust PLM (: |1, V1. .......... XK, Y.. X )
(I)
users can then parse the task output from the prediction Yiest (i.e., mapping into corresponding task verbalizers (Schick & Schütze, 2021)).
2.2. The Motivating Discussion
Evolutionary self-replication, a fundamental concept in Dar- winian evolution (Ofria & Wilke, 2004), explains how life reproduces and evolves through genetic variation and natu- ral selection, fostering the emergence of unpredictable traits and behaviors that enhance adaptability to constantly shift- ing environmental conditions. Similarly, the prompt design problem can also be framed as an evolutionary process, where prompts shall be iteratively refined to adapt to com- plex L.M environments. As such, prompts that are optimal for complex LLMs may exceed human intuition and require methods beyond manual design. Inspired by our findings, propose embracing more open-ended innovations (or broader open-endedness (Stanley et al., 2017))- -siftting from the human language space towards the "LLM language space"-to advance LLM prompting strategies.
An intriguing observation in recent LLM prompting re- search (Shin et al., 2020: Deng et al., 2022; Zou et al., 2023) suggests that, for certain tasks, unintelligible or unnatural prompts (e.g., StaffAAreaaocussiarcckee. Advisory for News Classification) can outperform care- fully crafted natural language instructions. This unnatural language phenomenon has been discussed as a form of se- crel language (Daras & Dimakis, 2022) in literature.
Specifically, secret language often refers to unnatural lan- guage prompts whose syntax and semantics are incoherent and difficult for humans to parse, yet can be surprisingly effective in certain scenarios. In the absence of theoretical explanation for their emergence, and universally effective method for large-scale discovery, such prompts, either dis- covered by coincidence or found after extensive computa- tion by certain algorithms (Shin et al., 2020; Deng et al., 2022; Jones et al., 2023, inter alia), such as the news classi- fication prompt we discussed above, are typically regarded as mysterious, hidden, and inherently non-scalable.
We demonstrate the effectiveness of our search framework through large-scale experiments across tasks and LLMs. Our results show that pruning low-shot ICL prompt could perform comparably to state-of-the-an methods on a range of tasks, while maintaining competitive runtime efficiency compared to prior optimization methods (e.g.. Table 10 in Appendix). Notably, our PROMPTQUINE frame- work is inherently well-suited for parallelization, which can further enhance both its scalability and efficiency, for ex- ample by parallelizing reproduction and fitness evaluation. Moreover, we find that the task improvements of our ap- proach could becosme more pronounced as the number of in-context examples increases, suggesting that scaling to more shots can unlock additional performance gains. This suggests that richer prompt variations can drive further gains. Finally, we show that key findings on label word importance also hold for our ICL pruning, with the additional insight that pruning has potentials to enhance performance with random verbalizers, even when starting from chance.
We argue that such seemingly chaotic discoveries may actu- ally contain universal insights into LLM sensitivity towards prompt design. We detail our thought process as follows: It's counter-intuitive that unnatural prompts can outperform natural instructions, despite being extensive trained to align human language. This suggests LLMs may only experience superficial aligument (Greenblatt et al., 2024) and instead may prioritize hypotheses over the explicit structure of hu- man linguistics. Recently, Chan et al. (2022) identified that transformer language models, especially LLMs, exhibit sparse, rule-based generalization in ICL, where minimal features can dominate predictions (Dasgupta et al., 2022). This raises the possibility that some input features can be redundant or inessential towards task prediction. We are thus curious whether natural language prompts, e.g., ICL, could be improved simply by removing certain input fea- tures in-context, exploring the potential of ICL optimization in-context. In other words, we deliberately explore disrupt- ing the grammatical structure of prompts in an attempt to approach a structure that LLMs might prefer.
2. Problem Formalisms
2.1. Preliminaries: In-context Learning
ICL. describes an emergent capability of L.LMs that given a
few training examples appended in context, the LLM is able
Alternatively, such perturbation might be viewed as a nearby
2

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
search within semantically coherent natural language con- texts, potentially outperforming the unnatural artifacts from token-level searches with limited algorithmic capacity (Shin et al., 2020; Deng et al., 2022). We refer to this as the Partial Context Hypothesis.
Search Space. The search space refers all possible solution candidates that the algorithm can explore. In our context, it specifically refers to any prompt subsequences extracted from the original prompts. Ideally, the word order within the prompts could also be altered during optimization. However, this operation would significantly complicate the search problem, resulting in an exponentially larger search space, Therefore, in this paper, we focus primarily on the fixed-order prompt subsequence search.
Specifically, given a natural language prompt, e.g., ICL prompt, x (#1, T2...... Fn) with task performance X, possible to prune few prompt tokens, resulting in pruned prompt = (21, (21,22,... 2m), m I n, with signifi- cantly improved task performance Z. Notably, the prompt can reach or even surpass task performance Y of unnatural language prompts y discovered by prior token-level search algorithms (Shin et al., 2020; Deng et al., 2022), although initial performance X may be substantially lower.
Search Objective. The search objective functions a performance measure evaluate the quality of candidate solutions, especially assessing the effectiveness of prompts in enhancing downstream task performance. For instance, prompt quality can be evaluated using an aggregated metric (c.g,, classification accuracy) on a held-out set separated from the original dataset. Since the search objective is typically con-differentiable, we cannot just approximate the solutions via traditional gradient ascent.
As we demonstrate in Appendix B through a pilot study using a simple hill-climbing approach (Section 3.1), the hy- pothesis proves effective and may outperform the unnatural language prompts discovered by Deng et al. (2022) across various contexts- eoppeils-e in pruning ICL, which is our focus. In terms of the effectiveness, for example, the popu- lar prompt "Let's work this out step by step to be sure we have the right answer," introduced by Zhou al. (2022), im- proves InstructGPT's performance on the MultiArith dataset (Roy& & Roth, 2015) from 78.7% to 81 81,5% outperforming the earlier prompt "Let's think step by step" (Kojima et al., 2022). By pruning the prompt to "Let's work out step by step sure we right answer", we achieve an even higher result of 86. 36.7%( (Appendix, Table 7).
Solution Selection. Once the search converges or termi- nates, the algorithm returns selected optimal solution, i.c., an optimal prompt. The optimality of prompt is typically measured by an aggregated metric on a held-out dataset. This dataset is often referred to as the validation set, with final performance reported separately on a test set using the task specific metric. We ensure strict separation between the validation and test sets to prevent data leakage and enable a reliable assessment of generalization.
2.3. Compression as Guided Search: A Reformulation
3. PROMPTQUINE
The prompt compression is a widely studied field where the typical target is to improve the inference speed (Li et al., 2023; Jiang et al., 2023d;b; Pan et al., 2024). In contrast, frame the prompt compression problem as guided prompt search where the task is construed as searching for the prompt subsequence which can elicit improved task results. To avoid the ambiguity, we mainly use the term "prompt pruning" in the paper.
We now introduce our search framework PROMPTQUINE. We'll begin with a straightforward hill-climbing search in Section 3.1, which serves as a strong baseline with good em- pirical performance and is also the method used in our pilot study (Appendix B) to validate the Partial Context Hypothe- sis. Then, we outline the core design principles for further improvements in Section 3.2, followed by an exploration of the objective landscape and the justification for using evolutionary search in Section 3.3. Section 3.4 outlines our evolutionary search framework for PROMPTQUINE.
Formally, we describe the search problem as below:
Given input prompt x (I1,, 22, ;pp). wn), the goal is to locate a pruned prompt z = (21, . 2m), m S n as subsequence of input prompt x. The subsequence length m is not predefined and it shall continuously adjust over optimization. Our search optimizes a non-differentiable task objective f(z; x, D) which typically represents the task performance over the dataset D and aims at returning the optimal solution, i.e., an optimally pruned prompt we dis- covered far. In the remainder of this section, we will describe the search space, the search objective as well as the overall principles for the solution selection in detail.
3.1. A Simple Hill-climbing Search
Due to the absence of well-established or highly efficient al- gorithms for this prompt subsequence search, as well as the failure of attribution methods (Appendix C) and prompt compression algorithms (e.g., Table 1), we start with greedy local search method. Basically, it works by iter- atively pruning tokens, removing those that improve the prompt's performance. This continues until no token can be pruned without harming validation set performance. This follows the general framework of hill-climbing. While this
3

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
approach may not yield optimal solutions, it provides quick and intuitive way to explore the hypothesis and gain initial insights, as detailed in Appendix B.
making it a strong contender for practical use. We explore further improvements in the following sections.
3.2. Design Principles for Improved Performance
Specifically, we primarily adopt the Threshold Accepting (TA) algorithm (Dueck & Scheuer, 1990), which builds upon the First-Choice Hill Climbing (FCHC) algorithm (Russell & Norvig, 2016). Namely, given a prompt se- quence x= 22,.......... .n)) the algorithm tracks the cur- rent solution (initialized as prompt x) and generates new candidate by making a local per-token change, e.g., remov- ing a token Ty., If the new candidate improves upon the current solution, it is accepted, and the tracked solution is updated accordingly. This directly contrasts with Steepest- Ascent Hill Climbing (SAHC) (Russell & Norvig, 2016), which evaluates all possible one-token modifications and selects the one that yields the greatest improvement (called SAHCPruning) our method offers significant speedups by accepting any modification that improves the prompt.
A search framework consists of three key components: the search space, algorithm, and objective. While refining the search space is constrained by the black-box nature of LLMs, there is significant potential to improve the algo- rithm and objective: (1) Search Algorithm: Hill-climbing may be suboptimal; insights into the search landscape (Deb & Saha, 2010; Ecoffet et al., 2019) can guide better algo- rithms. (2) Search Objective: The high cost of prompt evaluation, driven by task metries like accuracy, can be re- duced by using more expressive proxies ( (Yang et al., 2024b), We attempt to integrate these principles into our designs.
3.3. Navigating Sparse, Multimodal Search Landscapes
We now examine the search challenges in our optimization landscape to inspire more effective algorithms.
The algorithms continue until no further improvements are found or a stopping criterion is met. To ensure reproducibil- ity. we fix the pruning order (the method used to make per-token changes) in our implementation. In fact, we apply left-to-right pruning order, iterating over x1 to nn and proposing a new prompt by removing the most recently vis- ited token, Z4, at step i. As the algorithm converges when no tokens can be further removed, it follows that we may repeat the left-to-right iterations multiple times, with each iteration initializing the tracked solution using the optimal prompt from the previous iteration. We illustrate the whole idea in Appendix, Algorithm 1. We call this approach TA TAPruning, which helps escape local optima, as observed in preliminary experiments, by further accepting suboptimal candidates within a predefined threshold (e.g.. 2 96% of the current maximum prompting performance). This suggests a decep- tive nature of our search objective (i.e., held-out score). We'll revisit this later in Section 3.3 and Section 4, where explore how greedy hill-climbing struggles, becoming deeply trapped in local optima, and emphasize the need for rewarding suboptimal stepping stones (Lehman & Stanley, 10 Stanley & Lehman, 2015) to global optimization.
To begin with, we revisit our hill-climbing approach (e.g.. TA) and highlight a key assumption: hill climbing guaran- tees convergence to the global optimum only in unimodal spaces (Glover & Kochenberger, 2003). In such space, a single optimum, i.e., a single peak, allows efficient optimiza- tion by following increasing objective values without the need for exhaustive searches in multiple directions. This assumption, however, is very likely fail in our context.
As shown in the Figure (Left), we relax the left-to-right order constraint in our hill-climbing algorithm (TAPrun- ing) and explore random pruning orders in multiple runs using purely greedy hill-climbing. The results indicate that these runs converge to different solutions, leading to sig- nificant variations in task performance. This suggests a potentially complex, multimodal nature of the search prob- lem, where traditional hill-climbing and its variants, such as Simulated Annealing (Bertsimas & Tsitsiklis, 1993) and TA- TA-struggle to perform effectively. These findings high- light the need for more global exploration strategies to ad- dress local optima.
As shown in Table (Appendix B) for Meta- Mitta-Llama-33 3-BB- Instruct (AI@ Meta, 2024), and an extended study on various LLMs across sizes (e.g.. Table 10 in Appendix B) using held- out performance for prompt selection, we demonstrate con- sistent performance gains across models and tasks through pruning. Notably, such improvement is independent of the ICL prompt we sampled, suggesting a potentially effective approach to stabilize ICL performance (Zhao et al., 2021: Lu et al., 2022; Rubin et al., 2022). Besides, while sim- ple, TAPruning already delivers competitive results against prior search methods in both task results and runtime effi- eiency across several tasks (e.g., Table 11 in Appendix D.3),
To navigate the multimodal landscape, both unstructured methods like Random Search (Zhigljavsky, 2012) (RS) and structured approaches such as Population- Search (Be- heshti & Shamsuddin, 2013) (e-g., evolutionary search, ES) can be used, each offering unique advantages depending on the search space. In certain scenarios, RS can outperform more complex structured methods, particularly when local optima are densely packed (Bergstra & Bengio, 2012), as unstructured approaches may more efficiently explore multi- ple regions and find better solutions, while population- approaches may overly exploit a local region.
We investigate the use of RS and population-based search
4

---

Evolving Prompts In-Context: An Open-ended, Selfrrepii Perspective
0.9
1,0
(a) Optimization Pipeline
(b) Genetic Prompi-Quine
0.8
Selection
Mher seball pant expteration
0.7
0.5
Bett Roplicatiiiion
Mutation: Pruning
0.6
0.4
Dempnstratin
7G
I-bvt
0.5
0.2
Updatees
Sett-Repiictting Loep
0.0
43
Fruring Order
Random Evolution
Mothod
Number of shots KCL
4
Intovtoniion
Popptation
39
a Iterations of Selt Reglicating Leops
Gosted by Fltmace Selvcher
Figure 1. Optimization challenges in our ICL-initialized landscape using Llama-3-8B- 3-8BB-nsttt-ttt for subjectivity classification. Left: Randomizing pruning order in hill-climbing search leads to vary- ing task performance, highlighting the multimodal nature. Middle: Evolutionary search (ES) outperforms random search (RS) in iden- tifying high-quality solutions, with TAPruning result as a dashed line. Right: Relative success rate of RS over ES approaches zero as task difficulty increases, particularly when solution sparsity is enhanced by expanding the context. Further studies, with consis- tent conclusions, are provided in Appendix D.2.
Figure 2. Overview of the PROMPTQUINE framework. Similar to natural selection, our framework evolves prompts by copying and mutating them (i.c., pruning random tokens). Guided by task- specific selection pressures, progressively optimizes itself. Notably. the generation of unnatural language prompts. despite introducing unexpected variations, consistently outperforms man- ually designed prompts, representing a step towards open-ended sel self-improvement in AT (Schmidhuher, 2003; Nisioti et al., 2024).
We refer the reader to Appendix D.1 for details. Tuning the GA for an unknown landscape requires extensive trial and er- ror. We recommend using our configurations for follow-up experiments and discuss key designs below.
(i.c., evolutionary search, ES using our PROMPTQUINE de- sign in Section 4.1) in the ICL landscape, conducting inde- pendent sampling runs generate 1,000 unique prompt sam- ples for both subjectivity classification (Subj) and natural language inference (SNLI). As shown in the Figure 1 (Mid- dle). RS proves highly inefficient in obtaining high-quality prompts that outperform TAP TArruning, while ES achieves better performance under constrained prompt samples. In- spired by Real et al. (2020), we define the number of the acceptable prompts under the fixed budget as the number of sampled pruned prompts which outperform TAPruning's average performance. The success rate is then defined as the number of acceptable prompts divided by the number of pruned prompts sampled. As shown in Figure (right), the relative success rate of RS to ES (i.e., the success rate of RS divided by the success rute of ES) approaches zero as task difficulty increases. For example, when dealing with stan- dard long contexts, where high-quality prompts are sparse in the search space, unstructured methods like RS may strug- gle to efficiently navigate the space. We thus recommend ES, as it is more robust to the search dimensions.
We initialize the entire population with duplicates of ICL prompts, as early experiments with random pruning for ini- tializations showed no significant advantage. For mutation rate, a uniformly random selection of the number of flipped bits among the values E {1, 2, 4} effectively balances the -oorattoooeepliiaation We then use fournament selection with slightly reduced selection pressure, sampling k indi- viduals and selecting the best for reproduction, which helps mitigate local optima. Most crucially, we apply regular- ized evolution (Real et al., 2017: 2019), where only new offspring compete for population inclusion. This approach, which we have empirically validated, is highly effective in navigating the ICL landscape, particularly in addressing the premature convergence issue that standard GA struggles to overcome-an an issue that is a key bottleneck in tuning the configurations for ICL. As we demonstrate in Appendix D.7, such simple approach outperforms many complex diversity- preserving mechanisms (Friedrich et al., 2009) in balancing search speed and solution quality.
3.4. Evolutionary Search for PROMPTQUINE
Simple regularized evolution sacrifices some exploration, limiting its effectiveness in broader contexts. To improve exploration, we increase the selection probability for each individual through additional designs, promoting more re- production each generation. This approach is guided by two algorithmic frameworks (Syswerda, 1991): a paral- lelizable Generational GA (GGA) and more exploratory Steady-state GA (SSGA), detailed in Appendix D.I. Un- less otherwise noted, we use SSGA for 1-shot ICI. results. Subsequent experiments also reveal that GGA achieves com- parable performance across various ICL search landscapes.
We now provide an overview of our ES algorithm for prompt subsequence search. Specifically, we use Genetic Algorithm (GA) (Holland, 1992), due to its inherent compatibility with our problem. In this approach, we evolve a population of pruned prompts, where binary token masks serve as geno- types and the resulting ICL prompts as phenotypes. Mu- tations (i.e., pruning tokens) are implemented via bit-flip 1 1-to-0 operations. Ebitism-based selection guides offspring survival, enabling the autonomous evolution of pruning strategies. Additional details are provided in Appendix D, along with Algorithm 3. Our GA variants incorporate sev- eral designs, which effectively improves the search quality.
Finally, we introduce an additional prompt re-ranking phase,
5

---

Evolving Prompts In-Context: An Open-ended, Selfrrellicaigg Perspective
Table 1. Performance of PROMPTQUINE and baseline methods measured by overall accuracy (%) in classification tasks. All methods use 3-8B-BB I--stuuuttt (AI@ Meta, 2024) as the backbone LM for a fair comparison. The best results are highlighted in bold and the second best are underlined, Ratio denotes the compression ratio over the original I-shot ICL prompts.
Method
SST-2
Subj
AG's News
Yelp-5
SNLI
Yahoo
Avg.
Ratio
ICL (1-shot, original) (Brown al., 2020)
95.9 (0.6)
66.7 (4.3)
83.7 (1.9)
52.2 (6.00
61.9 (2.0)
57.1 (6.9)
69.6
0.0%
M..MLin (Jiang et al., 2023c)
95.8 (0.6)
64.0 (4.2)
85.3 (1.8)
49.7 (6.6)
64.2 (1.7)
49.5 (8.2)
68.1
30.0%
ingMua (Pan al., 2024)
(11.8)
60.2 (5.1)
58.1 (13.0)
45.4 (5.7)
51.5 (5.1)
57.1 (4.4)
55.7
71.2%
ICL (4-shot) (Brown et al., 2020)
94.8 (1.5)
74.0 (7.8)
66.6(18)
61.9 (0.6)
(2.5)
(5.7)
72.0
RLPrompt (Deng et al., 2022)
(1.5)
82.9 (0.5)
84.7 (0.7)
48.1
68.2 (4.2)
58.5 (0.6)
67.5
EvoPrompt (Guo et al., 2023)
92.9 (0.2)
84.1 (0.3)
86.5 (1.0)
51.5 (0.6)
(0.3)
73.6
Promptbreoder (Fernando et al.. 2024)
96.0 (0.4)
(3.5)
88.6 (0.8)
(1.3 (1.6)
(1.4
75.8
Prompibreeder (4-shol) (Fernando et al., 2024)
95.8 (0.5)
83.1 (3.0)
88.5 (1.0)
59.3 (1.5)
(1.9)
(1.1
75.2
TAPruning (1-shot ICL, Ours)
(3.9) (1.5
88.6 (0.3)
60.2
(2.6
(1.7
74.8
60.2 %
SAHCPrwning (I-shot ICL, Ours)
96,0 (0.7)
77.3 (6.6)
88.5 (0.8)
58.5
(2.8)
62.8 (0.9)
75.3
8.7%
PROMPTQUINE (1-shot ICL, Ours)
96.2 (0.2)
86.5
89.2 (1.8)
59.7 (2.1)
61.3 (2.0)
77.5
52.9 %
TAPruning (4-shot ICL., Ours) PROMPTQUINE (4-shot ICT., Ours)
95.4 (1.5)
86.9 (0.7)
88.9 (0.8)
61.3 (1.6)
67.3 (1.6)
63.8 (1.1)
77.3
96.4 (0.4)
93.1 (0.8)
89.4 (1.8)
64.3 (0.6)
(3.1)
66.2 (1.5)
81.3
called "calibration- then--ttteheheeee wwwettheeeeeeeeeee selection"' using the entire held- out score (e.g., what we used for TAPruming) to mitigate potential overfitting to the imperfect evaluation proxy. This function refines the prompt rankings, allowing for more accurate identification of the true elite prompts. We then select the "optimal" prompt from the calibrated rankings.
and leave others in Table 10 (Appendix D.3). We con- sider the following methods for comparisons: (1) Prompt Compression: LLMLingua (Jiang et al., 2023c) and LLM- Lingua2 (Pan et al., 2024); (2) Prompt Optimization: RL- Prompt (Deng et al., 2022), EvoPrompt (Guo et al., 2023) and Promptbreeder (Fernando et al., 2024). Unless other- wise specified, we append the optimized instructions from EvoPrompt and Promptbreeder to our 1-shot ICI. prompts to form their few-shot versions. We report RLPrompt's templates in Appendix B. (3) In-context Learning (ICL): In addition to 1-shot ICI., which directly illustrates the ben- efits of our pruning methods, we also include 4-shot ICL to assess whether simply increasing the number of shots (Zhao et al., 2021) can easily match the benefits of prun- ing. Finally, we include SAHCPruning on 1-shot ICL as a purely greedy baseline to illustrate the deceptiveness of the landscape. The results averaged across five seeds, with five different ICL initializations. All optimized prompts are selected based on the same held-out accuracy, with full details in Appendix D.3.
4. Task Designs& & Results
Due to space constraints, we discuss classification and gen- eration in the main paper, and put multi-choice question reasoning in Appendix D.5
answering and and D.6. In all these tasks, PROMPTQUINE consistently achieves improved results over TAPruning.
4.1. PROMPTQUINE for Classification
For classification, probability-basee prompt selection (Yang et al., 2024b) demonstrates some success in few-shot set- tings (Lu et al., 2022), enabling fine-grained measurement using metrics like Mutual Information (Sorensen et al., 2022), Entropy (Lu et al., 2022), and Majority Voting (Liao et al., 2022). Through extensive experiments, we finally take the piecewise reward function from (Deng et al., 2022) as our default fitness measure, with details in Appendix D.3. The existence of these multiple metrics can also be extended to multi-objective optimization (Deb et al., 2002) or novelty search (Lehman & Stanley, 2011), leveraging complementary proxies (Vo & Luong, 2024) for prompt selection. However, preliminary experiments indicate that these approaches offer no significant advantages over our single-objective formulation. Consequently, we center our efforts on the single-objective approach in this work.
Evaluation Settings. We evaluate SST-2 (Socher et al., 2013), Yelp-5 (Asghar, 2016), Subj (Pang & Lee, 2004), AG's News (Zhang et al., 2015), Yahoo (Labrou & Finin, 1999), and SNLI (Bowman et al., 2015) as our Pilot Study (Appendix B). The overall statistics are in Table 5 (Appendix B). As we find 8-shot balanced samples are generally suffi- cient for fitness estimation, extract such paired samples from our held-out set (200 validation samples as TAPruming in Appendix B) as in-search fitness estimation samples, and use the entire held-out accuracy for prompt re-ranking. We provide additional details and suggestions in Appendix D.3.
Models and Baselines. We report Instruct (L-aI33-8B-II) (AI@ Meta, 2024) in the main paper,
B-BB-
Results. As shown in Table 1, overall, PROMPTQUINE on 1-shot ICL is able to match or surpass sfate-of-the-
6

---

Evolving Prompts In-Context: An Open-ended, Selfreepicctrrg Perspective
art performance across settings. One particular baseline emphasize for comparison is the Promptbreeder (Fer- nando et al., 2024), which shares closely aligned spirit with our approach-leverrging leveraging evolutionary algorithms to promote open-endednesss in the context of L.Mss. Their framework, while demonstrating impressive generality, re- lies more heavily on manual engineering (e.g., handcrafted mutation prompts) and external resources (e.g., new tokens). In contrast, PROMPTQUINE fosters open-endedness under tighter resource constraints, by evolving unnatural language prompts using only tokens within context. We show that PROMPTQUINE also has potentials to be comparable in spe- cific problem contexts. We further compare the runtime efficiency against various baselines in Appendix, Table 11. To the best of our knowledge, this is the first token-level search (TAPruning & PROMPTQUINE) capable of optimiz- ing in just minutes. Please refer to Appendix 4.1 for addi- tional analysis, especially for the intriguing observations that SAMICruning may not always outperform TAPruning.
PLPrompt& & Promptbreeder under both greedy decoding and Best-of-N sampling (setup of Deng et al. (2022)). To ensure relatively fair comparisons, all methods are given access to the same set of samples (Appendix D.4).
Table 2. Automatic evaluation of Yelp Sentiment Transfer. We report their average Joint Score, averaging across negative and pos- itive transfer results. The results (no parentheses) are reported with greedy decoding. BoN refers to Best-of-N sampling, following the setups of Deng et al. (2022), ie.. Bo32 under top- 10-10 sampling. Additional results are presented in Table 12 in Appendix D.4.
Method
GPT-2 GPT-3 (BoN)
Llmm3-88--i
Llum3--88- (BoN)
ICL RLPrompt
4.5
40.8
54.4
69.6
10.4
51.0
4,1
54.4
Prompthreeder
10.2
45.3
71.8
ZAPraming
30.2
54.6
50.6
68
PROMPIQUINE
33.3
$7.9
61.0
72.1
As shown in Table 2, it's possible that PROMP- TQUINE could surpass the previous state-of-the-ard on this task. These results, as we demonstrate, are largely insensi- tive the decoding methods we employed. Additionally, as shown in Table 14 (Appendix D.4), our search framework operates more efficiently, delivering faster search. Neverthe- less, compared to few-shot classification tasks, the search tends to be slower due to the imperfect feedback we receive, requiring more samples for fitness estimation.
Results.
4.2. PROMPTQUINE for Text Generation
Generation tasks present unique challenges for the limi- tations of automatic evaluation metrics, such as surface based measures (Papineni et al., 2002) and neural embedding- metrics (Deng et al., 2021). These met- rics often fail to capture the full complexity of the task or accurately reflect fine-grained optimization progress, partic- ularly in open-ended generation tasks. In open- open-ended cases, e.g., topic-based generation, only sparse, qualitative scores are available, e.g., those by LLM-as-a-Judge (Zheng et al., 2023) based on predefined principles, or by more mechanis- tic approaches like Exact Match (Zou et al., 2023), looking for specific word overlaps. We now explore the potential of PROMPTQUINE on generation tasks, focusing on style trans- fer, which relies on imperfect quantitative measures, and jailbreaking, which depends on sparse qualitative feedback.
Table 3. Attack success rate (ASR) for jailbreaking comparison of PROMPTQUINE: and conventional ICL. The text in parentheses refers to the fitness measure we used for PROMPTQUINE.
Attack Method
ASR-EM
ASR-LLM
LLM:
ICL (2-shat)
50.4
54.7
PROMPTQUINE (ASR-EM)
99.3
97.4
PROMPTQUINE (ASR-LIM)
99.4
97.5
PROMPTQUINE (SV)
90.3
94.2
4.2.1. TEXT STYLE TRANSFER
LLM: Mis.taall----nn-----
v
ICL (2-shot)
48.0
47.2
Experimental Setups. We evaluate our prompts on Yelp sentiment transfer dataset (Shen et al., 2017), as Pilot Study (Appendix B) for both transfer directions. We set aside 200 samples from the original development set for validation and report results on the test set. We use Joint Score (Kr- ishna et al., 2020) following and reusing the modules by Deng et al. (2022), as both the fitness function for prompt search and the final metric for performance assessment. We search under the unsupervised setting, where we only need unpaired input samples from original development set (e.g., 100) for fitness estimation, use the entire 200 samples for re-ranking and report final results on the test set. To re- duce computations, we adopt an carly-stopping strategy inspired by Jamieson & Talwalkar (2016); Li et al. (2018b) to optimize resource allocation. We then compare against
PROMPTQUINE (ASR-EM)
98.8
93.8
PROMPTQUINE (ASR-LLM)
99.8
98.1
PROMPTQUINE (SV)
98.8
92.6
4.2.2. JAI.BREAKING
Evaluation Settings. We consider this task as highly chal- lenging due the only sparse, qualitative feedback avail- able, such as the Exact Match score. We adopt a simple few-shot priming setup where the model directly follows the demonstrations to make predictions (i.e.,peffe attack, with- out chat templates inserted), using popular models including Vicuna-7b-1 1.5 (Zheng et al., 2023) and Mistral-7B. 7B- v0.3 (Jiang et al., 2023a) on AdvBench (Zou et al., 2023). We measure Attack Success Rate (ASR) using both Exact
7

---

Evolving Prompts In-Context: An Open-ended, Selfeeiiinn Perspective
Match (ASR-EM) and LLM-as-a-Judge by Llama-Guard-3 (Inan al., 2023) (ASR-LLM). Please refer to Appendix D.4 for the details of our EM strings (Zhao et al., 2024) and prompts for Llama-Guard-3 (Ball et al.. 2024) (taken from Jainh (Chao et al., 2024)). We split the original 520 samples into 100 for validation and 420 for testing, us- ing samples from the validation set for fitness estimation and prompt selection. We construct the ICL prompts (2-shot) by reorganizing the 5-shot ICL prompts in Liu et al. (2024).
Task Metric Score
Standard
Task
Å (score(p min) -> Score(p maz)
Deviation
SNLI
14.3
(60.8 + 75.1)
7.2
PIQA
2.6
(79.5 + 82.1)
1.1
Yelp Positive.
2.5
(49.9 -> 52.4)
1.4
Yelp Negative.
1.0
(69.8 -> 70.8)
0.9
Table 4. Performance fluctuations of PROMPTQUINE- prompts across three random ICL templates. each with three 1- shot ICL seeds. Task scores reported on the official test set. A indicates the performance fluctuation from the minimum to the maximum results across the selected templates.
Experimental Details. We produce separated experi- ments, using both ASR-EM score and ASR-LLM score on the 50 validation samples to guide the search. These experiments are conducted under purely black-box setting, with the same validation re-ranking for prompt selection. We also explore to construct more expressive proxies (i.e. to reduce computational budgets) by leveraging steering vectors (SV) from mechanistic interpretability (Bartoszeze et al., 2025). Such vectors, extracted as activation difference, can enable direct intervention in model's inner layers to enhance task performance (Rimsky et al., 2024), allowing us to hypothesize that the similarity between internal repre- sentation changes after prompting and task steering vectors correlates with performance. Please refer to Appendix D.4 for the full details (3 seeds). We also explore variants of few-shot attack, e-g., in-context attacks (Wei al., 2023b) where input-ouiput pairs are separated in the conversational contexts, i.e., separated by the chat template tags, in the Ap- pendix 1,4.1, where we observe both successes and failures with the current pruning formulations.
the chosen ICI. templates. We present one such study: we cover set of tasks, SNLI (Bowman et al., 2015) for classi- fication, PIQA (Bisk et al., 2020) for multi-choice question answering, and Yelp positive transfer and negative trans- fer (Shen et al., 2017) for generation. For each task, we evaluate two additional ICL templates (see Appendix, Table 24 for examples, where the templates differ only in signal words, separators, spacing characters, and minor variations in natural language instructions normal variations in prac- tice) and run PROMPTQUINE pruning on each using three different seeds (i.e., three ICL, prompts).
As presented in Table 4, differences in templates can still lead to significant variations in task outcomes. For example, in SNLI, the absolute accuracy fluctuation can reach up to 14.3%, which is surprisingly high. This emphasizes that PROMPTQUINE, or fixed-order ICL pruning in general, exhibit instability when exposed to template differences. Incorporating richer, more diverse prompt variations (e.g., token replacement or insertion) can be crucial for further improvement. Nevertheless, achieving this in the current PROMPTQUINE could be hard due to fitness-based selection bresfcaes-beneffciall changes, such as the introduction of new tokens, may be prematurely discarded. As a result, we can only adopt more conservative mutation operators, such as varying instructions (Fernando et al., 2024). More aggressive approaches, like exploring the full token space (Deng et al., 2022), are riskier. A promising direction to advance is consider novelty search (Lehman & Stanley, 201 1), which favors novel solutions over fitness alone.
Results. As shown in Table 3, under the priming setup, PROMPTQUINE is able to derive effective pruning which leads to improved attack results, nearly doubling the tradi- tional average ICL. performance. Search results guided by ASR-LLM achieve highest performance. is noted that SV's performance is generally lower. This is expected as just an initial exploration, and further research is needed to obtain further improvements. We also release some direct predictions of our pruned prompts on the AdvBench in the GitHub repository'. ensuring rigor given the different data- separation schemes used across prior studies (Jiang et al., 2024; Paulus et al., 2024).
5. A Deeper Look at Pruning Effects on ICL
5.2. Mechanistic Analysis of Label Words
5.1. On the Limitations of PromptQuine
It is intriguing to analyze what matters in the unconventional pruned ICL prompts for their unreasonable effectiveness. In this section, we examine the role of label words in the mansttration-allinnen with verbalizers- classification tasks, in line with existing ICL research (Min et al., 2022; Yoo et al., 2022; Wang et al., 2023; Wei et al., 2023a). Con- cretely, it is often assumed that label words play a crucial
While pruning tokens are effective at enhancing overall ICL results, we identify its inherent limitations for current PROMPTQUINE. Specificallyy, we find that token pruning not a universally reliable method for stabilizing ICL per- formance, as its effectiveness remains highly sensitive to
8

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Parcentage
Ki4
903
014
GPS5
Gemmnnlt
Limma-i1--i Lame33750 Lim4A--7E-4
Model
Figure 4. Changes in (unpruned pruned) prompting performance on
Figure 3. The percentage of label word presence is surprisingly high for our PROMPTQUINE pruuded ICL prompts. We obtain these parsing results by exactly matching the label words within the pruned prompts. We verify that the ICL prompts used in the analysis do not contain these words in their exemplar inputs.
8-B-Instruct when labels are removed (left & middle) or when the entire output removed (right, breaking the input-label pairing format). Another study with similar findings on GPT-2 is presented the Figure in Appendix.
role in determining original ICL performance (Min et al., 2022; Wang et al., 2023). We conduct series of interven- tion experiments on label words to better understand their impact on pruning outcomes.
6. Related Work
We discuss briefly various prompt design paradigms and ICL studies in previous work, and provide more comprehensive discussion in Appendix A. Concretely, ICL was introduced by Brown et al. (2020), who demonstrated that L.Mss can adapt to downstream tasks using few-shot prompting, Never- theless, this paradigm has proven to be highly unstable; even slight variations can lead to significant differences in perfor- mance. Rubin et al. (2022) demonstrate the importance of demonstration selections, where retrieving demonstrations with similar patterns as the task input typically yields im- proved performance. Lu et al. (2022) further identifies that even demonstration orders affect the results to significant extent. Although heuristics such as retrieving structurally and potentially semantically similar exemplars- often per- form well in practice, there remains a lack of well-grounded principles to guide demonstration design. Several studies aim to optimize prompts in alternative formats, typically via direct natural language instructions (Zhou et al., 2022; Guo et al., 2023, inter alia). Unlike ICL stabilization, this approach treats prompt optimization as a search for optimal instructions- often using LLMs as prompt engineers. In contrast, we focus on optimizing in an unnatural language space, building directly on few-shot prompts.
First, we observe that many of the pruned ICL prompts retain certain task-specific label words in their exemplar contexts (Figure 3). This observation closely mirrors find- ings from conventional ICL, suggesting that label words may play& a significant role in ICL, even under pruning. Then, we perform the interventions. Figure 4 (left) shows the performance changes after removing the label words in original ICL prompts. The results are consistent with Min et al. (2022) that knowing the label space may help in conventional ICL. The hypothesis also generalizes in our unconventional ICL. pruning context (Figure 4, middle) that we can observe a slight performance drop on average in pruned prompting performance. We then further experi- ment with removing the whole output, i.c., the signal words (e.g., Sent iment: :) and the label words (e.g., great) in Figure 4 (right). This operation further largely degrades per- formance, highlighting the importance of preserving input- label format as in standard ICL (Min et al., 2022).
Although most of our findings so far are consistent with the findings on conventional ICL, there are still some spe- cial cases where prompt instances violate the aggregated findings discussed above (e.g., SNLI in Figure 4 (right), in which the pruned prompt (No Output) could achieve improved performance). This underscores the nuanced sen- sitivity of prompts within their specific contexts. Finally, we perform experiments with random verbalizers (label words are also changed in prompts). As shown in Appendix, Table 25, all these models show near chance-level performance without pruning. However, we find that counter intuitiiituu pruning could also bring some performance improvements, especially towards the large 70B model. A negligggee number of prompts can achieve significantly improved per- formance through pruning, and even achieve nearly identical to that of prompts with inst-intuitive verbalizers. This is indeed surprising, and we hope future research will explore its underlying mechanisms further.
7. Conclusions
We introduce novel prompt design paradigm that chal- lenges conventional practices: instead of carefully crafting instructions and examples, we show that pruning random demonstrations into incoherent "gibberish" can still achieve near state-of-the the-rer performance. As attribution and com- pression methods remain unreliable, present PROMP- TQUINE, an evolutionary framework that autonomously discovers effective pruning strategies. Experiments across diverse tasks and models validate its effectiveness and run- time efficiency, paving the way for future research into the mechanistic foundations of in-context learning.

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Acknowledgements
Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. Trans- formers learn to implement preconditioned gradient de- scent for in-context learning. Advances in Neural Infor- mation Processing Systems, 36:45614 4 45650, 2023.
We would like to thank Zhen Wang, Yichi Yang, Zhiting Hu for their early discussions; Jiayuan Mao and Freda Shi for their valuable feedback; Yu Rong and the entire DAMO Academy (Hupan Laboratory) for the support.
@ Meta. Llama 3 model card. 2024. URL
blob/ http:://///tthh..coo/oo
ama / 1ama3/
Impact Statement
Asghar, N. Yelp dataset challenge: Review rating prediction. arXiv preprint arXiv: 053662, 2016.
This work aimed at advancing the research of LLMs, with a particular focus on prompt optimization and in-context learning. In contrast the traditional approach of optimiz- ing in natural language space, we introduce novel prompt design paradigm that prunes random few-shot prompts into syntactically and semantically unnatural language. Surpris- ingly, we find that a simple pruning operation applied to ICL prompts is sufficient to match the performance of previous optimization methods across various tasks and models. This insight paves the way for future research, leaving several interpretability questions open for further exploration. For example, it is intriguing explore why pruning to unnatu- ral language is effective for prompt optimization and why such unnatural ICL remains effective for LLMs. This is particularly noteworthy in cases where pruning enhances performance for several random label words, whereas natu- ral ICL yields only chance-level results.
Bachrens, D., Schroeter, T., Harmeling, S., Kawanabe, M.,
Hansen, K., and Müller, K.-R. How to explain individual classification decisions. The Journal of Machine Learning Research, 11:1803-1831 2010.
Ball, S., Kreuter, F., and Panickssery, N. Understanding jailbreak success: A study of latent space dynamics in large language models. arXiv preprint arXiv:2406.09288 2024.
Bartoszcze, L., Munshi, S., Sukidi, B., Yen, J., Yang, Z.,, Killiams-King, D., Le, I.,, Asuzu, K., and Maple, C. Representation engineering for large-language mod- els: Survey and research challenges. arXiv preprint arXiv:2502. 1.X1:22002.7601..0 2025.
Moreover, we highlight the direct societal implications of our findings unnatural language. Notably, our work exposes critical weaknesses in current LLM alignment tech- niques. Despite extensive training designed to align mod- els with human values and ethical standards when given natural language instructions, our findings reveal that un- natural language can still be used elicit malicious behav- iors exps-exploiting gaps that developers cannot fully anticipate. As demonstrated in our paper, this vulnerability persists even in large models subjected to extensive red teaming. While continuously iterating on red teaming and eliminat- ing failure cases is beneficial, we advocate for exploring novel alignment techniques that go beyond surface-level fixes. particular, a stronger focus on inner alignment may lead to more robust improvements. For commercial models, strongly recommend complementing red teaming with output-level restrictions, as this may provide a more intu- itive and effective safeguard- especially given that existing alignment methods are primarily optimized for handling natural language inputs.
Beheshti, Z. and Shamsuddin, S. M. H. A review of population-based meta-heuristic algorithms. lnt. j. adv. soft comput. appl, 5(1):1-35, 2013.
Bergstra, J. and Bengio, Y. Random search for hyper- parameter optimization. Journal of machine learning research, 13(2), 2012.
Bertsimas, D. and Tsitsiklis, J. Simulated annealing. Statis- tical science, 8(1):10-15, 1993.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference artificial intelligence, 2020.
volume 34, pp.
Bowman, R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. In Mårquez, L.,, Callison- Burchh, C., and Su, J. (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp- 632-642, Lisbon, Portugal, September 2015. Association for Com- putational Linguistics. doi: 10. 10.11653/11/D15--0077 URL https: http://allaathhhoo ogy . .rgg 50tp//11110000 10755
References
Agarwal, R., Singh, A., Zhang. L. M., Bohnet, B., Chan, S., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J. D., Chu, E., et al. Many-shot in-context learning. arXiv preprint arXiv:2404. / 1018, 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
10

---

Evolving Prompts In-Context: An Open-ended, Sclf-replicating Perspective
Chan, S. C., Dasgupta, I., Kim, J., Kumaran, D., Lampinen, A. K., and Hill, F. Transformers generalize differently from information stored in context vs in weights. arXiv preprint arXiv:2210.05673. 2022.
Daras, G. and Dimakis, A. Discovering the hidden vocab- ulary of dalle-2. In NeurIPS 2022 Workshop on Score- Based Methods, 2022.
Dasgupta, I., Grant, E., and Griffiths, T. Distinguishing rule and exemplar-based generalization in learning systems. International Conference on Machine Learning, pp. 4816 48330. PMLR, 2022.
Chao, P., Debenedetti, E., Robey, A., Andriushchenko, M., Croce, F., Sehwag, V., Dobriban, E., Flammarion, N., Pappas, G. J., Tramer, F., et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv preprint arXiv:2404.0133/ 2024.
Deb, K. and Saha, A. Finding multiple solutions for mul- timodal optimization problems using multi-objective evolutionary approach. In Proceedings of the 12th annual conference on genetic and evolutionary computation, pp. 447-454, 2010.
Chen, L., Chen, J., Goldstein, T., Huang, H., and Zhou, T. Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03002, 2023.
Deb, K., Pratap, A., Agarwal, S., and Meyarivan, fast and elitist multiobjective genetic algorithm: Nsga- ii IEEE transactions on evolutionary computation, 6(2): 182-197, 197, 2002.
Cheng, X., Chen, Y., and Sra, S. Transformers implement functional gradient descent to learn non-linee non-linear functions in context. arXiv preprint arXiv:23 2ttl::23/2.052. /2.06223/ 06528, 2023.
Choi, Y., Bae, S., Ban, S., Jeong, M., Zhang, C., Song, I.., Zhao, L., Bian, J., and Kim, K.EE. Hard prompts made interpretable: Sparse entropy regularization for prompt tuning with RL. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1252- Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18663/11/2022.ac-ooggg 449. URL https:// ac aclanthology wwtp//////0/////////// acl- llog..oo...00mm 449/.
Deng. M., Tan, B., Liu, Z., Xing, E., and Hu, Z. Compres- sion, transduction, and creation: A unified framework for evaluating natural language generation. In Moens, M.-F, Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Pro- ceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 7580-7605, On- line and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 10.18653/w1/2021.mmmll manin-l-ii99m 599. URL https:/ a aclanthology org/ 2021.emm111pp...... 1aa.lppmmin.550
Co-Reyes, J. D., Miao, Y., Peng, D., Real, E., Le, Q. V., Levine, S., Lee, H., and Faust, A. Evolving reinforce- ment learning algorithms. International Conference on Learning Representations, 2021.
Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H., Shu, T., Song, M., Xing. E., and Hu, Z. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Fm- pirical Methods in Natural Language Processing, pp. 3369-3391 Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguistics. doi: 10. 1111665///1/222emm/mpppp www.emmaimlmmm2222221-- URL https:// aclanthology orgg 22//22222222 emnlp main. 222.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert. M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint ar 10.12110..11.6 2021.
Cui, W., Zhang, J., Li, Z., Sun, H., Lopez, D., Das, K., Malin, B. A., and Kumar, S. Phaseevo: Towards uni- fied long-context prompt optimization for large language models. In First Workshop on Long-Context Foundation Models@ ICML 2024, 2024.
Denil, M., Demiraj, A., and De Freitas, N. Extraction of salient sentences from labelled documents. arXiv preprint MXXXv:442..8155. 2014.
Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., and Wei, F. Why can GPT learn in-context? lan- guage models secretly perform gradient descent as meta-optimizers. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 4005-4019, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10. 10.18653/v1/2023 findings-acl. 247. URL https:/ aclanthology wwwpogg......//// or2/2023. f Lindings-acl--
Deutch, G., Magar, N., Natan, T., and Dar, G. In-context learning and gradient descent revisited. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Pupers), pp. 1017-1028, Mexico City, Mexico, June 2024. Associa- tion for Computational Linguistics. doi: 10. 18.18v53/v1/ 2024 2024.naacl- 58... URL https: https////ccaathooogooe org/2024 naac! aoaeclllome -lonoo..55. 58.

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Di Langosco, L. L.., Koch, J., Sharkey, L. D., Pfau, J., and Krueger, D. Goal misgeneralization in deep reinforce- ment learning. In International Conference Machine Learning, pp. 12004-12019, PMLR, 2022.
Glover, F. W. and Kochenberger, G. A. Handbook of mela- heuristics, volume 57. Springer Science & Business Media, 2003.
Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDi- armid, M., Marks, S., Treutlein, J., Belonax, T., Chen, J., Duvenaud, D., et al. Alignment faking in large language models. arXiv preprini arXiv:2412. 14093, 2024.
Dueck, G. and Scheuer, T. Threshold accepting: A general purpose optimization algorithm appearing superior to simulated annealing. Journal of computational physics, 90(1):161-175, 1990.
Guo, Q. Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. Connecting large language mod- els with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532, 2023.
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, Go-explore: a new approach for hard- exploration problems. arXiv preprint arXiv: 10995, 2019.
Holland, H. Genetic algorithms. Scientific american, 267 (1):66-73,, 1992.
Feng, S., Wallace, E., Grissom II, A., Iyyer, M., Rodriguez, P. and Boyd-Graber, J. Pathologies of neural models make interpretations difficult. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 3719-3728, Brussels, Belgium, October-November 2018. Association for Computational Linguisties. doi: 10.18653/w1/D18- 1407. URL https:
Hughes, J., Price, S., Lynch, A., Schaeffer, R., Barez, F. Koyejo, S., Sleight, H.. Jones, E., Perez, E., and Sharma, M. Best-of-n jailbreaking. arXiv preprint arXiv:2412.03556, 2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint urXiv:2410.21221 2024.
//aclanthole ogy. hww.lannccolll..orr//eeeeeee
Fernando, C., Banarse, D. S., Michalewski. H., Osindero, S., and Rocktäschel, T. Promptbreeder: Self-referential self-improvement via prompt evolution. In Forty-first International Conference Machine Learning, 2024.
Inan, H., Upasani, K., Chi. J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q. Fuller, B., Testug- gine, D., et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.0667. 2023.
Friedrich, T., Oliveto, P. S., Sudholt, D., and Witt, C. Anal- ysis of diversity-preserviin mechanisms for global ex- ploration. Evolutionary Computation, 17(4):455-476, 2009.
Jamieson, and Talwalkar, A. Non-stochastic best arm identification and hyperparameter optimization. In Ar- tificial intelligence and statistics, pp. 240-248. PMLR, 2016.
Fu, D., Chen, T.-Q., Jia, R., and Sharan, V. Transformers learn higher-order optimization methods for in-context learning: A study with linear models. arXiv preprint arXiv:2310.122 17086, 2023a.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas. D. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, I.., et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023a.
Fu. Y., Ou, L., Chen, M., Wan, Y., Peng. H., and Khot, T. Chain-of-thought hub: A continuous effort to measure large language models' reasoning performance, 2023b. URL htttps://aree org/ http:/bbb2333..1333
Jiang, F., Xu, Z., Niu, L., Xiang, Z., Ramasubramanian, B., Li, B., and Poovendran, R. Artprompt: Ascii art- based jailbreak attacks against aligned llms. In Proceed- ings of the 62nd Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers), pp. 15157-15173, 2024.
Gabriel, I. Artificial intelligence, values, and alignment. Minds and muchines, 30(3):411-45) 2020.
Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive kv cache compression for lms. arXiv preprint arXiv:2310.01801, 2023.
Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L. Limlingua: Compressing prompts for accelerated in- ference of large language models. In Conference on Empirical Methods in Natural Language Processing, 2023b. URL https: /pppi. httpps://ap.ssmmmiiisssstttte/ www.stiismcsootooeeeeee.o org org/CorppsTD:
Gendreau, M. and Potvin, J.-Y. Tabu search. Search method- ologies: introductory tutorials in optimization and deci- sion support techniques, pp. 165-186, 2005.
12

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Jiang H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L. Limlin- gua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023c.
55. URL https://aclantteee www.ogg..oggo......///// emnlp-main.55
K Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, I.,, Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626, 2023.
Jiang, H., Wu, Q. Luo, X., Li, D., Lin, C.-Y., Yang. Y., and Qiu, L. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint ar arXiv:2310.0m 06839, 2023d.
Jiang, Z., Xu, F. F., Araki, J., and Neubig, G. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423- 438, 2020. doi: 16.1 a 00324. URL https: / //aclanthology org/ 2020. tacl-1.28.
Labrou, Y. and Finin, T. Yahoo! as an ontology: using yahoo! categories to describe documents. In Proceedings of the eighth international conference on Information and knowledge management, pp. 180-187, 1999.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Jones, E., Dragan, A., Raghunathan, A., and Steinhardt, Automatically auditing large language models via discrete optimization. In International Conference on Machine Learning, pp. 15307-15329. PMLR, 2023.
Lee, H., Phatale, S., Mansoor, H., Lu, K. R., Mesnard, T., Ferret, J., Bishop. C., Hall, E., Carbune, V., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
Khashabi, D., Lyu, X., Min, S., Qin, L., Richardson, K., Welleck, S., Hajishirzi, H., Khot, T., Sabharwal, A., Singh, S., and Choi, Y. Prompt waywardness: The curious case of discretized interpretation of continuous prompts. In Carpuat, M., de Marneffe, M.-C,, and Meza Ruiz, I. V. (eds.). Proceedings of the 2022 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pp. 3631-3643, Seattle, United States, July 2022. Association for Computational Linguistics. doi: URL https://
Lehman, J. and Stanley, K. O. Abandoning objectives: Evo- lution through the search for novelty alone. Evolutionary computation, 19(2):189 189-223, 2011.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Moens, M.-F, Huang, X., Specia, L., and Yih, S. W.-t (eds.), Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, On- line and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/20211 .wwppppp.emmmmmmmmmmmmmmmmm main.243. URL https:// a aclanthology org / 2221 emnlp-main 243.
10.18653/v1/2022.mnam maccl aclanthology org/2022 www.aaalllnnnm marll-mmin..22 266.
www.://m22..a..mmmmm......... main.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa,
Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213. 2022.
Li, C., Farkhoor, H., Liu, R., and Yosinski, J. Measur- ing the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018a.
Konzel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H. MAWPS: A math word problem repos- itory. In Knight, K., Nenkova, A., and Rambow, O. (eds.), Proceedings of the 2016 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 1152-1157, San Diego, California, June 2016. Association for Com- putational Linguistics. doi: 10.18653/v1/N16- 1136. URL https: / /ta///aaanaaaaacoo
Li, J., Chen, X., Hovy, E., and Jurafsky, D. Visualizing and understanding neural models in NLP. In Knight, K., Nenkova, A., and Rambow, O. (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pp. 681-691, San Diego, California, June 2016a. Association for Computational Linguistics. doi: 10.18653/w1/N16-1100 / / //aclanthology - org / N1 6 5-82.
org/ N1 6 1136.
Krishna, K., Wieting, J., and Iyyer, M. Reformulating un- supervised style transfer paraphrase generation. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 737- 762, Online, November 2020. Association for Computa- tional Linguistics. doi: 10. 8653/v /2020.emm/lpmmain
URL https:
Li, J., Monroe, W., and Jurafsky, D. Understanding neural networks through representation erasure. arXiv preprint arXiv: 16/2.00 2016b.
13

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185):1-52,211
Longpre, S., Hou, L.,, Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effec- tive instruction tuning. In International Conference on Machine Learning, pp. 12631-226442 PMLR, 2023.
Li, X. L. and Liang, Prefix-tuning: Optimizing continu- ous prompts for generation. In Zong, C.,, Xia, F., Li, W., and Navigli, R. (eds.). Proceedings of the 59th Annuul Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pp. 4582-4597, Online, August 2021. Association for Com- putational Linguistics. doi: 10.186655/1120222acHHHorr 353. URL https: /tta:://aaaaanna https///aaaaaatoogg.ooooooooo/ www.:/llllooo../222220 acl sel-long.o. 353.
Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stene- torp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sen- sitivity. In Muresan, S., Nakov, P., and Villavicen- cio, A. (eds.), Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ire- land, May 2022. Association for Computational Linguis- tics. doi: 10. 10.18633ww122222.aol-ooogg 5556. URL https: / //aclanthology/ wrg. acl wocc-lloe long. 556.
Li, Y. Dong, B., Guerin, F,, and Lin, C. Compressing context to enhance inference efficiency of large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 6342-632 -633,, Singapore. December 2023. Association for Computa- tional Linguistics. doi: 10. 00.186553v112023.emmlpp 391. URL https: httl//aaaaaoooooo..ooooo org/ 2023. emnlp-main. 391.
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learn- ing work? In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1 11048- 11064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: m59. URL https:/ aclanthology. org/ org/2022 2022. emnlp-main 759.
10118653/1/20022.omm
Liao, C., Zheng, Y., and Yang. Z. Zero-label prompt selec- tion. arXiv preprint arXiv:221104400 14466,, 2022.
Ngo, R., Chan, L.,, and Mindermann, S. The alignment problem from a deep learning perspective. arXiv preprint arXhv:2209.00626 2022.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
Nisioti, E., Glanois, C., Najarro, E., Dai, A., Meyerson, E., Pedersen, J. W., Teodorescu, I.., Hayes, C. F., Sudhakaran, S., and Risi, S. From text to life: On the reciprocal rela- tionship between artificial life and large language models. In Artificial Life Conference Proceedings 36, volume 2024, pp. 39. MIT Press One Rogers Street, Cambridge, MA 02142-1209 USA journals-info. .... 2024.
Lin, X., Wu, Z., Dai, Z., Hu, W., Shu, Y., Ng, S.-K., Jaillet, P., and Low, B. K. H. Use your instinct: Instruction optimization for Iims using neural bandits coupled with transformers. In Forty-first International Conference on Machine Learning, 2024.
Ofria, C. and Wilke, C. O. Avida: A software platform for research in computational evolutionary biology. Artificial life, 10(2):191-229, 2004.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9): 1-3,),113300000 2023.
OpenAL. Prompt engineering guideline, 2023. URL https: openai.ee........... www/ coc//ocs/ guides puiddes/prompt--egineeee
Liu, S., Ye, H., Xing, L., and Zou, J. Y. In-context vectors: Making in context learning more effective and control- lable through latent space steering. Forty-first Interna- tional Conference on Machine Learning, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744 2022.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: orXiv:/907.1 /692, 2019.
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y,, et al. Limlingua-2:
Data distillation for efficient and faithful tank-agi
14

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
prompt compression. arXiv preprint arXiv:2403./2968 2024.
V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15504-15522, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. //aclanthologee . wwg//0////222/ acl- long. 828.
Pang. B. and Lee. L. A sentimental education: Senti- ment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd An- mual Meeting of the Association for Computational Lin- guistics (ACL-04), pp. 271-278, Barcelona, Spain, July 2004. doi: 10.3115/121895. 1218990 URL https: //aclanthology: wtwp:gg...rrggoooo wttp///0-100000
URL https:
Roy, S. and Roth, D. Solving general arithmetic word prob- lems. In Mårquez, L.,, Callison-Burch, C., and Su, J. (eds.), Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language P Processing, pp. 1743- 1752, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10338 wtgg URL https: http://aaaatttooooooo htw.thhoooo...gooooooooooooo
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: a method for automatic evaluation of machine translation. In Isabelle, P. Charniak, E., and Lin, D. (eds.), Proceed- ings of the 40th Annual Meeting of the Association for Computational Linguistics. pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computa- tional Linguistics. doi: 01231111/1177088107711111 URL ht tttss/
Rubin, O., Herzig, J., and Berant, J. Learning to retrieve prompts for in-context learning. In Carpuat, M., de Marn- effe, M.-C,, and Meza Ruiz, I. V. (eds.), Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pp. 2655-2671, Seat- tle, United States, July 2022. Association for Computa- tional Linguistics. doi: 10. 10.188653/1/20222.mmma 191. URL https: https:: http:///aaaaathhhce 2ow.aattthoooooyyoo/22222222 naacl -ma i n. 191.
Paulus, A., Zharmagambetor, A., Guo, C., Amos, B., and Tian, Y. Adyprompter: Fast adaptive adversarial prompt- ing for lims. arXiv preprint arXiv:2404.1657 16873, 2024.
Qin, G. and Eisner, J. Learning how to ask: Querying LMs with mixtures of soft prompts. In Toutanova, K., Rumshisky, A., Zettlemoyer, I.., Hakkani-Tur, D., Belt- agy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technolo- gies, pp. 5203-5212, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/1/20211 naacl-main.410 URL https:/ /ttp///aaaaaatocacaaccca org/2021 naw.l-maaamaaaaamm.. 410.
Russell, S. and Norvig, P. Artificial intelligence: modern approach. Pearson, 2016.
Sareni, B. and Krahenbuhl, L. Fitness sharing and niching methods revisited. JEEE transactions on Evolutionary Computation, 2(3):97-106, 1998.
Schick, T. and Schütze, H. Exploiting cloze-questions for few-shot text classification and natural language inference. In Merlo, P., Tiedemann, J., and Tsar- faty, R. (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguisties: Main Volume, pp. 255-269, On- line, April 2021, Association for Computational Linguis- tics. doi: 10. 10.18653/ / //aclanthology 22g eacl mmaaellmiii .220
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019.
Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y. L., Tan, J.. Le, Q. V., and Kurakin, A. Large-scale evolu- tion of image classifiers. In International conference on machine learning, pp. 2902-2911, PMLR, 2017.
URL https:
Schmidhuber, J. Gödel machines: self-refcrential uni- versal problem solvers making provably optimal self- improvements. arXiv preprint cs/0309048, 2003.
Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. Regular- ized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelli- gence, volume 33, pp. 4780-4789, 2019.
Shen, L., Mishra, A., and Khashabi, D. Do pretrained transformers really learn in-context by gradient descent? arXiv preprint arXiv:23 2023a.
Real, E., Liang, C., So, D., and Le, Q. Automl-zero: Evoly- ing machine learning algorithms from scratch. In Interna- tional conference on machine learning, pp. 8007-8019. PMLR, 2020.
Shen, T., Lei, T., Barzilay, R., and Jaakkola, T. Style transfer from non-parallel text by cross- tross-alignment. Advances in neural information processing systems, 30, 2017.
Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., and Turner, A. Steering Ilama 2 via contrastive activa- tion addition. In Ku, W., Martins, A., and Srikumar,
Shen, T., Jin, R., Huang, Y., Liu, C., Dong, W., Guo, Z., Wu, X., Liu, Y., and Xiong, D. Large language model
15

---

Evolving Prompts In-Context: An Open-ended, Self-repliictinng Perspective
alignment: A survey, arXiv preprint arXiv:2309.15025, 2023b.
Stanley, K. O., Lehman, J., and Soros, L. Open-endedness: The last grand challenge you ve never heard of. While open-endedness could be force for discovering intelli- gence, it could also be component of Al itself. 2017.
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In Inter- national Conference on Machine Learning, pp. 31210- 31227. PMLR, 2023.
Stroebl, B., Kapoor, S., and Narayanan, A. Inference scal- ing laws: The limits of Ilm resampling with imperfect verifiers. arXiv preprint arXiv:2411. 2024.
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting Knowledge from Lan- guage Models with Automatically Generated Prompts. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222- 4235, Online, November 2020. Association for Computa- tional Linguistics. doi: 00.18653/1/2020.emmml/ main. 346. URL. https: // a anthologhoyyyy. wtgg//2000000 emnlp- maillp-main. main. 346.
Sun, H., Hüyük, A., and van der Schaar, M. Query- dependent prompt evaluation and optimization with of- fline inverse rl. In The Twelfth International Conference on Learning Representations, 2023.
Sun, T., He, Z., Qian, H., Zhou, Y., Huang, X., and Qiu, X. BBTv2: Towards gradient-fiee free future with large language models. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1916-3930, Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguistics. doi: 10.186653/1/20222.mmm 255 URL https: aclanthology wrg//02/ emnlp-ma 259.
Shrikumar, A., Greenside, P., Shcherbina, A., and Kundaje, A. Not just black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.0/713, 2016.
Syswerda, G. A study of reproduction in generational and steady-state genetic algorithms. In Foundations genetic algorithms, volume 1, pp. 94-101. Elsevier, 1991.
Simonyan, K.
Deep inside convolutional networks: Visu- alising image classification models and saliency maps. arXiv preprint 1312.66.111.. 2013.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, I.,, Sessa, P. G., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan, C. L., Choquette-Choo, C. A.. Crepy, C., Cer, D., Ippolito, D., Reid, D., Buchatskaya, E., Ni, E. Noland, E., Yan, G., Tucker, G., Muraru, G.- C., Rozhdestvenskiy, G., Michalewski. H., Tenney, I., Grishchenko, I., Austin, J., Keeling, J., Labanowski, J., Lespiau, J.-B., Stanway, J., Brenman, J., Chen, J.. Ferret, J., Chiu, J., Mao-Jones, J., Lee, K., Yu, K., Millican, K., Sjoesund, L., Lee, L., Dixon, L., Reid, M., Mikula, M.. Wirth, M., Sharman, M., Chinaev, N., Thain, N., Bachem, O., Chang, O., Wahltinez, O., Bailey, P., Michel, P., Yotov, P., Chaabouni, R., Comanescu, R., Jana, R., Anil, R., Mcllroy, R., Liu, R., Mullins, R., Smith, S. L., Borgeaud, S., Girgin, S., Douglas, S., Pandya, S., Shakeri, S., De, S., Klimenko, T., Hennigan, T., Feinberg, V., Stokowiec, W., hui Chen, Y., Ahmed, Z., Gong, Z., Warkentin, T., Peran, L., Giang, M., Farabet, C., Vinyals, O., Dean, J.. Kavukcuoglu, K., Hassabis, D., Ghahramani, Z., Eck, D., Barral, J., Pereira, F., Collins, E., Joulin, A., Fiedel, N., Senter, E., Andreev, A., and Kenealy, K. Gemma: Open models based on gemini research and technology, 2024. URL httpt::///// http//raxivv.../ org aba///ba//2000 /4033 242..00295......
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning,
C. D., Ng. A., and Potts, C. Recursive deep models for semantic compositionality over sentiment treebank. In Yarowsky, D., Baldwin, T., Korhonen, A., Liveseu, K., and Bethard, S. (eds.), Proceedings of the 2013 Confer- ence on Empirical Methods in Natural Language Process- ing. pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: / //accccll anllahohhoogy wtt//////11111 11700.
Sorensen, T., Robinson, J., Rytting, C., Shaw, A., Rogers, K., Delorey, A., Khalil, M., Fulda, N., and Wingate, D. An information-theorett approach to prompt engi- neering without ground truth labels. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguisties (Volume 1: Long Papers), pp. 819- 862, Dublin, Ireland, May 2022. Association for Com- putational Linguistics. doi: 10.18663//122022.dllooor
60. URL https: http:::://cllnntooloy..ooo acl scl-long.eo 60.
Souly, A., Lu, Q., Bowen, D., Trinh, T., Hsich, E., Pandey, S., Abbeel, P., Svegliato, J., Emmons, S., Watkins, O., et al. A strongreject for empty jailbreaks. arXiv preprint 24Xi::2402.102600 2024.
Stanley, K. O. and Lehman, J. Why Greatness Cannot Be Planned: The Myth of the Objective. Springer, 2015.
Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., and MacDiarmid, M. Activation addition:
16

---

Evolving Prompts In-Context: An Open-ended, Selfrepiiingg Perspective
Steering language models without optimization. arXiv e-prints; pp. arXiv-2308, 2023.
Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., et al. Larger language models do in-context learning differently, arXiv preprint arXiv:2303.03846 2023a.
Vo, A. and Luong, N. H. Efficient multi-objective neural architecture search via pareto dominance- novelty search. In Proceedings of the Genetic and Evolutionary Computation Conference. pp. 1146-1155, 2024.
Wei, Z., Wang. Y., Li, A., Mo, Y., and Wang, Y. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.0638% 2023b.
Von Neumann, J., Burks, A. W., et al. Theory of self- reproducing automata. 1966.
Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie, A., Neubig, G., Kulikov, I., and Harchaoui, Z. From de- coding to meta- reta-generation: Inference-time algorithms for large language models. arXiv preprint 4rXiv:2406.16882 2024.
Von Oswald, J., Niklasson, E., Randazzo, E.,, Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. In International Conference Machine Learning, pp- 35151-35174, PMLR, 2023a.
Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., and Goldstein, T. Hard prompts made easy: Gradient- based discrete optimization for prompt tuning and discov- ery. Advances in Neural Information Prciiiooorooooo Processing Systems, 36, 2024.
Von Oswald, J., Schlegel, M., Meulemans, A., Kobayashi, S., Niklasson, E., Zucchet, N., Scherrer, N., Miller, N., Sandler, M., Vladymyrov, M., et al. Uncovering mesa- optimization algorithms in transformers. arXiv preprint arXiv:2309.05888, 2023b.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453, 2023.
Wan, X., Sun, R., Nakhost, H., and Arik, S. O. Teach better or show smarter? on instructions and exem- plars in automatic prompt optimization. arXiv preprint arXiv:2406.15700 15708, 2024.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang. J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang. Peng. R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., and Fan, Z. Qwen2 technical report. arXiv preprint arXiv:2407.1067 2024a.
Wang, L., Li, L., Dai, D., Chen, D., Zhou, H., Meng, F., Zhou, J., and Sun, X. Label words are anchors: An in- formation flow perspective for understanding in-context learning. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference Empirical Meth- ods in Natural Language Processing, pp. 9855,9855, Singapore, December 2023. Association for Computa- tional Linguistics. URL https: //a///aaaannllgooo///eee org/2022 emnll-mnee-.eeee caanmppmmin... 609.
Wang, R., Tang, D., Duan, N., Wei, Z., Huang, X., Ji, J., Cao, G. Jiang, D., and Zhou, M. K-Adapter: Infusing Knowl- edge into Pre- TreiTTanned Models with Adapters. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1405-1418, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2011 findings-acl.12.. URL https: / aaaaaaaannhooooooooo ogy. org/ /rgg findings-acl...
Yang, C., Wang. X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models optimiz- ers. ArXiv, abs/2309.03400) 2023. URL https: www.oollar.rrrrooooooooooooooeee orggCCrppsII:
/ 261582296.
Yang, S. Genetic algorithms with memory-and elitism- based immigrants in dynamic environments. Evolutionary Computation, (6(3):385-416 2008
Wang, Y., Zhang, P., Yang, B., Wong, D. F., and Wang, R.
Yang, S., Kim, J., Jang. J., Ye, S., Lee, H., and Seo, M. Improving probability-based prompt selection through unified evaluation and analysis. Transactions of the As- sociation for Computational Linguistics, 12:664-680, 2024b. doi: 1a162//aclaa000666 URL https: acl aclanthology ogy org/ 2024 2tt.22222..2....3 tac1-1.37.
2024. Latent space shabbdif-embeddingg enables output-free Ilm
self-evaluation. arXiv preprint arXiv:2410.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824-24837, 2022.
Yin, K. and Neubig, G. Interpreting language models with contrastive explanations. In Goldberg, Y., Kozareva, Z.
17

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
and Zhang, Y. (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 184-198, Abu Dhabi, United Arab Emirates, De- cember 2022. Association for Computational Linguistics. 14. URL https: //aclanthooogy.cooooo/ otg/ 2022. emnlp coannp--minn, main. 14.
Zhou, Y., Muresanu, A. I., Han, Z.., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models human- level prompt engineers. In The Eleventh International Conference on Learning Representations, 2022.
doi: 10. 100116663//11222222emmm---m
Zhu, Z., Shahtalebi, S., and Rudzicz, F. Predicting fine- tuning performance with probing. In Goldberg. Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 1 11534-1 1547, Abu Dhabi, United Arab Emirates, December 2022. Association for Compu- tational Linguistics. doi: 10. 10.11663/w122022.cmmlmmmm 793. URL http:::s//aaaaathhlooooooo org/ 2022. emn mmnlp-mat n
Yoo, K. M., Kim, J., Kim, H. J., Cho, H., Jo, H., Lee, S.-W., Lee, S.-g., and Kim, T. Ground-truth labels matter: A deeper look into input-label demonstrations. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.). Proceedings of the 2022 Conference on Empirical Methods Natural Lan- guage Processing, pp. 2422-2437, Abu Dhabi, United Arab Emirates, December 2022. Association for Compu- main.
tational Linguistics. doi: (0.18653/v1/20222.oo 155. URL htwps:ttt//c.....c ht//aclaantt http:///aaaaaallo/oo/o/o/o org 2022. emnlp-men mappmmain... 155/.
Zou, A., Wang. Z., Kolter, Z., and Fredrikson, M. Uni- versal and transferable adversarial attacks on aligned lan- guage models. arXiv preprint urXiv:2307.15044 2023.
Zhang, S., Dong, I.., Li, X., Zhang, S., Sun, X., Wang. S., Li, J., Hu, R., Zhang, T., Wu, F,, et al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308.10792 2023.
Zhang, X., Zhao, J., and LeCun, Y. Character-level convolu- tional networks for text classification. Advances in neural information processing systems, 28, 2015.
Zhao, Y., Zheng, W., Cai, T., Xuan Long, D.,, Kawaguchi, K., Goyal, A., and Shieh, M. Q Accelerating greedy coordi- nate gradient and general prompt optimization via probe sampling. Advances in Neural Information Processing Systems, 37:53710-53731, 2024.
Zhao, Z., Wallace, E.,, Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 Proceedings of Machine Learning Research, pp. 12697-12706. PMLR, 18-24 Jul 2021. https::::: ptt.///ooeeeecooocoocooocoocec.c ngs. ml .ress// v139 w139/zhao211 .tmll.
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging alm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Zheng, X., Pang, T., Du, C., Liu, Q.. Jiang, J., and Lin, M. Improved few-shot jailbreaking can cireumvent aligned language models and their defenses. Advances in Neural Information Processing Systems, 37:32856 32887, 2024.
Zhigljavsky, A. A. Theory of global random search, vol- ume 65. Springer Science & Business Media, 2012.
18

---

Evolving Prompts In-Context: An Open-ended, Selfrelliiiin Perspective
A. Related Work
LLM Prompt Optimization. LLMs are sensitive to even minor variations in prompts, making their responses difficult to predict without trial and error (Liu al., 2023). This highlights the need for automated prompt tuning to optimize prompts for specific tasks. Existing research can be broadly classified into two categories: 1) Soft Prompt Tuning (Lester et al., 2021; Qin & Eisner, 2021; Li & Liang, 2021), which optimizes continuous embeddings in the LLM's representation space, replacing discrete prompts with learnable tokens. These embeddings, typically optimized via gradient descent, can outperform hard prompts when applied to intermediate model layers (Sun et al., 2022), similar to parameter-efficiont tuning like adapters (Wang et al., 2021); 2) Hard Prompt Optimization: Shin et al. (2020); Deng et al. (2022); Zou et al. (2023); Jones et al. (2023); Choi et al. (2024); Wen et al. (2024) use token-level search algorithms to reward effective tokens and penalize ineffective ones. While promising, these methods often produce non-human-like prompts. In contrast, a more popular line optimizes prompts in natural language, often using LLMs directly as prompt engineers (Zhou al., 2022; Yang et al., 2023; Guo et al., 2023; Chen et al., 2023; Fernando et al., 2024; Lin et al., 2024; Cui et al., 2024). By combining LLMs with algorithmic designs like evolutionary algorithms (Fernando et al., 2024), these methods achieve expert-level performance.
In-context Learning Studies. Here, we briefly review the literature highlighting the mechanistic studies of ICL,, the emergent capability of LLMs. Notably, the underlying mechanisms behind emergent in-context learning remain unclear. The main debate revolves around the mesa-optimizztion hypothesis (Dai et al., 2023; Von Oswald et al., 2023b;a; Ahn et al., 2023; Cheng et al., 2023; Fu et al., 2023a). Specifically, they argue that these models implement subsidiary learning algorithms that adjust the model inner representations as new inputs are received, with update rules resembling gradient-based optimization of a principled objective. For instance, Von Oswald et al. (2023a) shows that linear self-attention can emulate gradient descent on simple linear regression tasks. However, most of these findings cannot generalize to practical LLM tasks: First, Min et al. (2022) shows that input-label correspondence in demonstrations has little impact on task results. Furthermore, Shen et al. (2023a); Deutch et al. (2024) provide concrete evidence against the validity of such hypotheses in NLP tasks.
AI alignment. Alignment (Gabriel, 2020; Ngo et al., 2022) describes a process of encoding human values and goals into Al assistants to make them as helpful, safe, and reliable as possible. Recent advances of LLMs also motivate extensive studies to align LLM chatbots to human values (Shen et al., 2023b). LLM alignment typically involves two steps. In the instruction -tuninggttgngnnn stage (Longpre al., 2023; Zhang et al., 2023), LLMs are given instruction responns pairs of the tasks so they can learn by imitating the output. In the critique phase, a human another AI interacts with the model and grades its responses in real-time, known as the reinforcement learning from human or Al feedback (Ouyang al., 2022; Lee et al., 2023). It seems that LLM researchers are already able to significantly improve the alignment progress as the increasing performance numbers in a wide range of benchmarks and the improved win rates of their optimized LLMs against other LLMs. However, just as Ngo et al. (2022); Di Langosco et al. (2022) hypothesize that, these "aligned" L.LMs may only experience superficial outer alignment as they can still be prompted to generate undesired output (Zou et al., 2023; Greenblatt et al., 2024). This necessitates further studies on the inner alignment problem (Ngo et al., 2022).
B. A Pilot Study with TAPruning
In this section, we provide a pilot study of the Partial Context Hypothesis across prompts, models and tasks. As introduced in Section 3.1, where we lack well-established methods for this hypothesis, we begin our investigation with a simple, casy-to-implement hill-climbing search, TAPruning. Please refer to Section 3.4 for a more effective algorithmic framework.
B. TAPruning Additional Details
Algorithmic Deseriptions. We present the pseudocode of our TAPruning algorithm in Algorithm 1. As stated, our algorithm maintains a tracked prompt, denoted as T, which serves as the basis for conducting local search operations. As presented in the do-while loop in Algorithm 1, we employ a left-to-right token pruning process, iterating sequentially from the leftmost token to the rightmost, attempting to remove each token at every iteration. The generated new prompt P experiences performance evaluation and comparison against the optimal performance fOptmmal, If it improves or at least, degrades within a certain threshold 8, we will accept the prompt P as the new tracked prompt T, which we may revisit in the next iteration loop. If it improves, we will also update the optimal prompt S = P. Until no further tokens are removable under the performance constraints, we return the optimal prompt we discovered as the final solution. Please note that
19

---

Evolving Prompts In-Context: An Open-ended, Selfrriiiinng Perspective
Algorithm 1 Threshold Accepting (TAPruming) for Prompt Pruning
Input: prompt X, IN.. dataset D, performance measure function language model M, Threshold 0
Output: Solution S
Measure the performance f for prompt X on 'D for language model M as the original prompting performance frriginal which is also used to initialize current optimal prompting performance ( fOptimal Originall
Initialize tracked prompt T = X1 ... ty = 21 EN.. Initialize optimal prompt S = X, 81 NN = 21 .N.
repeat
Initialize length variable L. the number of the tokens in current tracked prompt T (re-tokenized), for i = 1 to L do
Generate neighborhood prompt P by removing the token i; from prompt T.
Calculate the performance f" for P on D.
If f /Optinal, we accept and update the prompt = P, optimal prompt = P, and update the optimal performance record OOptimal f
If fP < fOptiiall we accept as the new T only fP fOprimal x 8 (within the threshold).
end for until The solution S converges
Algorithm 2 Steepest-Ascent Hill Climbing (SAHCPruning) for Prompt Pruning
Input: prompt X, ... dataset D, performance measure function f. language model M, Threshold ö Output: Solution S
Measure the performance fx for prompt X on D for language model M as the original prompting performance fOriginal which is also used to initialize current optimal prompting performance fOptimal Original
Initialize tracked prompt (also as optimal prompt) S 51 55 = =1 ... Initialize optimal token index candidate T, = -1 repeat
Initialize length variable T. as the number of the tokens in current tracked prompt S (re-tokenized). for = 1 to L do
Generate a neighborhood prompt P by removing the token s; from prompt S. Calculate the performance fp for P on D,
If f OOptimal, update the optimal performance record fOptimal = f. and optimal token index candidate Tg
end for
if To is not -1 then
Update tracked prompt S by removing token ST. Update T- = -
end if
until The solution S converges
the tokens (tokenized) may change through the optimization as the prompt changes in its surface form, as we desire the optimized prompts to be portable strings. Preliminary experiments show that the Partial Context Hypothesis also works if we retain the same prompt tokens through optimization.
We discuss other design choices, especially attempting to leverage insights from LLM mechanistic interpretability studies, such as attribution methods (Li et al., 2016a), in Appendix C. These methods can generate a token ranking in a single forward-backward pass, potentially leading to significant speedups. However, observed, they generally fail to effectively guide performance improvement, making them unsuitable for problem. We present SAHCPruning in Algorithm 2, which accepts the update only if this move (i.e., how to remove the next single token) is the best possible move among all available options. This algorithm limits in its high computational cost: since ICL prompts always take more than hundreds of tokens, i..., the problem relatively high-dimensional, the search would become much slower and less scalable, e.g., Table 11. Also, due to the deceptive, multimodal nature of the search landscape, SAHCPruning does not always outperform TA Pruning (Table 1).
20

---

Evolving Prompts In-Context: An Open--ended,,ff---ppieiee Self-replicating Perspective
Table Details of our classification datasets evaluated this work. |C]: # classes for classification tasks. [Test]: # of testing samples for each particular dataset.
Dataset
Type
|C|
[est
Label words
SST-2
Sentiment (Movie reviews)
2
1.8k
terrible, great
Subj
Subjectivity (Movie reviews)
2
2k
subjective, objective
AG's News
Topic (News articles)
7.6k
World, Sports, Business, Tech
Yelp-5
Sentiment (Yelp reviews)
50k
terrible, bad, okay, good, great
SNLI
Natural Language Inference
10k
Yes, Unknown, No
culture, science, health, education, computer, sports, business, music, family, politics
Yahoo
Topic (Question types)
10
60k
B.2. Pilot Study
Datasets. Our investigation covers four task types: classification, multiple-choice question answering, generation, and chain-of-thought reasoning. Unless otherwise stated, we sample from their official validation set for the prompt selection, and report on their official testing split for final evaluation. 1) Classification: We evaluate sentiment analysis (SST-2 (Socher al., 2013), Yelp-5 (Asghar, 2016)), subjectivity classification, Subj (Pang Lee, 2004), topic classification (AG's News (Zhang et al., 2015) and Yahoo (Labrou & Finin, 1999)), and natural language inference (SNLI (Bowman al., 2015)). We present the overall statistics in Table 5; 2) Multi-choice questions: We include commonsense reasoning datasets, PIQA (Bisk et al., 2020), binary-choice question answering dataset with verbalizers A and B. Since its testing performance can only be assessed via submission to the leaderboard, we sample from their training set to form our held-out set for prompt selection and evaluate on the official validation set, which consists of 2,000 examples, for the final performance assessment; 3) Generation: We include Yelp Sentiment Transfer (Shen et al., 2017) (Yelp Style.), where we follow Deng al. (2022) in an unsupervised style transfer setting, sampling from its development set for prompt selection and using its test for final evaluation, with reference collected by Li et al. (2018a) for both transfer directions (e.g., positive-to-negatvv and negative- toreeett--pppeo to-positive); 4) Chain-of-Thouu Thoughtt (CoT) reasoning: We include GSM8K (Cobbe et al., 2021) and MAWPS Lencel-Kedziorste et al., 2016).
Models and Baselines. We evaluate a range of popular LLMs varying in architecture, scale, and alignment efforts. Concretely, 1) Classification: we study base models: RoBERTa-large (Liu et al., 2019), GPT-2 (Radford et al., 2019), Meta-LJama-3-SB, Meta-Llama-3-70B (AI@ Meta, 2024), instruction-tuned model (SFT): Gemma-7b-it (Team et al., 2024), and further reinforcement lcarning-ture tuned (RLHF) models: -otta-llam----BB---
and Meta-Llama-3370B-- Instructt (AI@ Meta, 2024); 2) Multi-choice question answering: Meta-Llama-3-88 & -----aaaa-------sttees- 3) Generation:
GPT-2 & Mleas-Llama---S--Insstue 4) Reasoning: Meta-Llama-3-SB-nnsttee
(Jiang et al., 2023a)
and QQwen2-7B-Instruet (Yang et al., 2024a). In this section, we focus on results from RoBERTa-large and Meta-l.amma-- 8B-Instruct. Results for other models, along with PROMPTQUINE, are presented in the corresponding Appendix sections that follow. We include both the original ICL and RLPrompt (Deng et al., 2022) a state-of-the-as prompt optimization method-as well token-level search algorithm for the "secret language" baseline. It is important to note that this approach cannot be applied to CoT reasoning tasks. For ICL prompts, we primarily use one-shot ICL,, where each prompt for four-way classification is constructed by randomly sampling one instance and its label from each category in the training split. Two-shot prompts (two onput-output pairs) are used for style transfer. For ICL on classification and multiple-chott question answering, results are averaged over 10 random seeds, each with a unique ICL prompt. For generation and reasoning tasks, we average over 5 seeds. For RLPrompt, results are averaged over 3 seeds. We also use the entire held-out set for prompt selection in PlPrrmpt. Specifically, we first rank the explored prompts by their reward scores, select the top 50, and then re-rank them based on validation performance. We also explore using more training task samples for reward calculation but observe minimal performance gains. Thus, we adopt the original 16-shot setup from Deng al. (2022) for classification and use 200 unsupervised samples for style transfer, training over five tokens with default hyperparameters.
Prompt Template Details. First, for the RLPrompt, we use the following template for most of the classification tasks, following (Schick & Schütze, 2021; Deng et al., 2022):
21

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
RIPrompt- Classification:
(Tnput) ) (Prompt)
That is, we replace the "[Input)" placeholder with the task input instance. The policy network directly generates the prompt, replacing the "[Prompt)" placeholder. For masked language models, such RoBERTa-large. we directly append [MASK] into the last of the template, run the prompt, and parse its task predictions from that token. One exception the natural language inference task, which involves two inputs (the premise & the hypothesis). We provide its template as below:
RLPrompt Natural Language Inference:
(Premise) (Hypothesis) (Prompt) Entai Iment:
Then, provide its template for multi-choice question answering:
RLPrompt- Multi-choice Question Answering:
(Prompt)
Question: { (Input)
Options: A) [Option1) B) (Option2)
Answer:
The template for style transfer:
RLPrompt Text Style Transfer:
(Prompt } " "[Input)"
Just as (Deng et al., 2022) does, we allow the model to generate tokens one by one until we meet special character (the quotation mark). We then parse the completions inside the quotation marks as final task output.
Next, we provide our templates for our ICL prompts, which also serve as the basis for our prompt pruning.
ICL Sentiment Analysis (SST-2 and Yelp-5):
(Examples)
Review: {Input}
Sentiment:
For "(Examples)", the same format organize the input-output task pairs do for the input task instances.
This principle applies all our ICL prompts regardless of tasks.
We discuss templates for other tasks as follows:
ICL Subjectivityy Classification (Subj):
(Examples)
Sentence: (Input)
Viewpoint:
ICL News Topic Classification (AG's 's News):
(Examples)
Article: {Input)
Answer:
ICI. Natural Language Inference (SNLI):
22

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
[Examples)
Hypothesis: (Hypothesis)
Premise: (Premise)
Given the premise, is the hypothesis true? Yes, No or Unknown? The answer is:
ICL Topic Classification (Yahoo):
(Examples)
Sentence: (Input)
Topic:
ICL Multi-choice Question Answering (PIQA):
(Examples)
Question: (Input)
Options:
A) [Optionl) B) (Option2)
Answer:
ICL Text Style Transfer (Yelp Sentiment.):
(Examples)
Here is text, which is [negative/positivee "(Input)". Here is a rewrite of the text, which is [positive/negetiee::
We alternate between the sentiment signal words, "negative" and "positive", for two token choices for target sentiment direction. For example, for positive transfer, negative and positive respectively. Similarly, we parse the completions inside the quotation marks final task output.
ICL - Cfuin-of-thoughh Reasoning (GSM8K & MAWPS):
[Examples)
Question: (Input) Let's s think step by step.
The models then expected follow the exemplar patterns, i.e., generating the reasoning chains before predicting the task answer. For this particular task. lack the direct annotations of the ground truth reasoning chains, conduct the following steps to collect our CoT prompts: 1) GSM8K (Cobbe et al., 2021): we directly sample and adopt the ICL prompts provided by the Chain-of- Thought Hub (Fu et al., 2023b); 2) MAWPS (Koncel-Kede Kedziorkdi et al., 2016): we ask GPT-4o (Hurst et al., 2024) to generate the reasoning chains for the training questions we sampled, which are then paired to construct the corresponding ICL prompts.
Evaluation settings. During the search stage, each prompt's quality is evaluated 200 samples from the official validation split (as held-out set), or the training split if the validation is unavailable (e.g.. PIQA). We report performance on the official test set (i.c., validation set for PIQA).
Evaluation Metrics. As stated, we report testing accuracy for classification, multi-choice question answering and reasoning tasks. For style transfer, we follow Deng et al. (2022) to use their fine-tuned style classifiers for Style calculation, pre-trained LM to calculate onput-output alignment (Deng et al., 2021), Content, and a pre-trained grammaticality classifier (Krishna et al., 2020) for Fluency. Then, we average these sentence-lewe scores, as the Joint Score, strictly following
23

---

Evolving Prompts In-Context: An Open-ended, Selfreeiiinng Perspective
Table 6. Task performance evaluation of TAPruning upon original 1-shot ICT. prompts. For classification, multi-choice answering (PIQA) and math reasoning (GSM8K & MAWPS), we report accuracy on their test set. For style transfer (Yelp sentiment transfer. Yelp Style.), we report their joint score on separate 500 sample set, following (Krishna et al., 2020; Deng al.. 2022). We evaluate using greedy decoding. We report classification performance on RoBERTa-large (Liu et al., 2019) and overall task performance on Meta-Llama---SB-mmm (AI Meta, 2024) (Llama3-8B-II) Note that RlPrompt (Deng et al., 2022) cannot be applied to CoT reasoning, so we leave those cells blank. The numbers in (parentheses) are the standard deviations between different prompts.
LLM
Methods
SST-2
Subj
AG's News
Velp-5
Yelp Style.
PIQA
GSM8K
MAWPS
ICI.
(7.7)
(5.3)
(3.6)
(2.1)
(1.2)
(9:9)
RoBERTa-lure
R Prompt
(0.8)
(0.8)
(1.7
80.2 (0.7)
44.8 (4.3)
33.5 (0.8)
(1.0(
TAProning (Ours)
(2.6)
(1.6)
79.7 (2.3)
(2.3)
42.8 (6.9)
42.0
(1.4(L.5)
ICL
66.7 (0.9
(4.3)
(1.9)
52.2 (6.0)
61.9 (2.0)
57.1 (6.9)
(6.9
(1.6
(7.4
(9.4)
Llama3-SB-It
Prompp
88.4(1.5)
82.9 (0.5)
84.7 (0.7)
48.1
42.4
58.5 (0.6)
8.3 (2.8)
(2.6)
TAPrnning (Ours)
(1.5(
(3.9)
(0.3)
60.2 (0.9)
68.6 (2.9)
(1.7
(9.6
75.1 (3.0)
77.1 (2.1)
(3.9
Krishna et al. (2020)'s protocol:
[Content Style, Fluency)
=
(2)
meanxex (Content(x)= - Style(x) Flueney(x)).
Results. As shown in Table 6, pruning the original ICL prompts can improve performance, supporting our Partial Context Hypothesis. Surprisingly, this approach works well for mmain-of-thought reasoning, where the outputs are highly structured and answers are distant from the prompt-an unexpected result for such a complex task. Remarkably, pruned prompts could deliver competitive performance, often surpassing RLPrompt across several tasks, effectively bridging the gap between the "secret language" (i.e., unnatural language artifacts discovered by previous token-level search algorithms) and original natural language prompts. Note that the improvement is independent of the ICL prompts we sampled, suggesting a potentially more effective approach to stabilize ICL. performance (Lu et al., 2022; Rubin et al., 2022). We believe these results can inspire novel prompt optimization or in-context learning stabilization algorithms. We thus explore this further in Section 3.4, with our TA TAPruning serving as a baseline for the pruning- pruning-based prompt optimization. Despite its simplicity, this baseline comparable or directly outperforming previous state-of-the-an methods in both performance
remains competitive-beet and efficiency (e.g.. Table 11), making it a strong contender. It is also intriguing to analyze what is left after the pruning, which can potentially inspire some mechanistic insights. We provide such analysis in Section 5.2.
Table Zero-shot chain-of of-thought performance on the MultiArith (Roy & Roth, 2015) dataset using InstructGPT fext-davinci-009) (Ouyang et al., 2022). The natural language prompt was proposed in (Zhou et al., 2022) to enable the zero-shot chain-of-thoup reasoning of large language models. We produce these experiments using the answer extraction script from (Kojima et al., 2022).
No. Category
Zero-shot CoT Trigger Prompt
Accuracy
APE (Zhou al., 2022)
Let's work this in a step by step way be sure have the right answer.
1
81.5
Pruned
Let's work out step by step to be sure we have the right answer.
85.3
Let's work out step by step sure we right answer.
86.7
(Empty)
17.7
C. Alternative Prompt Pruning Design Choices
In this section, we discuss some potential alternative design choices that may be used in prompt pruning to provide more background about why we start with the hill-climbing method describe in Section B in exploring the partial Context Hypothesis and inspire future works.
The algorithms discussed below aim to improve prompt pruning speed by leveraging the information inherently provided by LLMs for the given task. Approaches requiring significant efforts, such as probing techniques to learn task proxies (Zhu et al., 2022), which depend on large collections of grompt-perfomann pairs for supervision, are not considered due to their impracticality. We also do not include the studies of ICI.-gradient descent correspondence hypothesis here, where the
24

---

Evolving Prompts In-Context: An Open-ended, Selfreiiiing Perspective
0.80
0.85
0.45
Gradiennt Norm
Gradient Nom
Gradiont Nor
0.75
Contrastive Gradiont Norm
0.80
Contraztive Gradient Norm
Inputx Gradient
Irputx
iiput
0.40
Contrantive Input Gradient
0.75
Cantrastive Input Gradient
0.70
0.35
Accuracy
0.65
Accuracy
0.70
Accuracy
0.65
0.30
0.60
0.60
0.25
0.55
0.55
0.50
0.50
0.20
0.45
0.45
0.15
25
50
75
100
25
50
100
25
50
75
100
Percent Pruned Takens
Percent Pruned Tokens
Percent of Pruned Takens
(a) How Task Testing Accuracy changes (SST-2, Subj and AG's News), guided by attribution scores, when increasing the percent of pruned tokens.
0.70
0.75
0.45
First Layer
Firet Layer
First Layer
Layer Aguregated
Layer Aggregated
Layer Apgregateo
0.70
0.40
0.65
0.65
0.35
Accuracy
0.60
Accuracy
Accuracy
0.60
0.30
0.55
0.55
0.25
0.50
0.50
0.20
0.45
0.45
0.15
25
50
75
100
25
50
75
100
25
50
75
100
Percent of Pruned Takens
Percent of Pruned Tokens
Percent of Pruned Takens
(b) How the Task Testing Accuracy changes (SST-2, Subj and AG's News), guided by attention weights, when increasing the percent of pruned tokens.
Figure 5. Pruning-hased prompting performance using variety of methods (Top for attribution scores guided pruning, and Bottom for attention weights guided pruning).
representational similarity may indicate task performance, as recent work (Shen et al., 2023a; Deutch et al., 2024) invalidate the effectiveness of such hypothesis in generalizing to practical NLP tasks for LLMs. Please note that the algorithms below are all inspired by the developments in mechanistic understandings of LLMs. Some of them might only be applicable in white-box scenarios.
Instance attribution scores guidance. Instance attribution methods aim improve interpretability by identifying influential tokens for model's prediction. Common approaches include gradient-based methods (e.g., gradient x input saliency (Bachrens et al., 2010; Li et al., 2016a)) and crasure-based methods (Li et al., 2016b; Feng et al., 2018), which measure the change in output when masking tokens. Our approach (e.g., TA) aligns with erasure methods, which are computationally expensive due to the need for multiple input perturbations. This raises the question of whether gradient- based methods can be adapted to approximate token rankings within single waakwrdd pass (Feng al., 2018). so, we can directly the attribution scores guide the pruning.
Token attention weights as guidance. Attention mechanisms are crucial for understanding how LLMs process prompts and assign token importance. Analyzing attention distributions across layers can rank token significance, with higher attention scores indicating greater influence on task prediction (Xiao et al., 2023; Ge et al., 2023). We aggregate attention weights across heads in specific layers to compute token importance scores, guiding pruning.
C.1. Experiments
We evaluate the aforementioned methods on several classification datasets and demonstrate their limited utility in the context of prompt compression as guided prompt search reformulation.
25

---

Evolving Prompts In-Context: An Open-ended, Selfrepliiiin Perspective
Table Classification accuracies of optimal prompts for TAPruning and alternative pruning methods hased on attribution scores and
attention weights in GPT-2 across various datasets.
Method
SST-2
Subj
Yelp-5
ICL (I-shot, original)
54.2 (4.8)
64.3 (9.8)
30.9 (5.2)
Gradient Norm
66.5 (7.5)
72.3 (0.0)
33.6 (4.6)
Input x Gradient
66.8 (5.2)
75.1 (1.7)
33.9 (3.5)
Contrastive Gradient Norm
66.1 (8.6)
75.9 (2.4)
31.5 (2.8) 31.5 (2.8)
Contrastive Input x Gradient
52.0 (1.5)
75.7 (2.9)
First Layer
57.3 (7.0)
74.7 (2.2)
35.7 (4.8)
Layer Aggregated
60.2 (4.1)
72.3 (0.0)
30.2 (3.0)
TAPruning (Ours)
68.1 (10.6)
75.9 (2.5)
39.9 (1.8)
Experimental Setups. We perform experiments on SST-2, Subj. and Yelp-5 using GPT-2, sampling five ICL prompts per dataset. Importance scores are computed once based on the same 200 samples, with performance evaluated on the official test split. 1) Instance attribution scores: we experiment with state-of-thh-ar approaches, including Gradient Norm (Simonyan, 2013; Li et al., 201 and Input x Gradient (Denil et al., 2014; Shrikumar et al., 2016) as well as their contrastive versions (Yin & Neubig, 2022). Token importance scores are aggregated across the 200 samples to generate final rankings, which are then used to guide token pruning: 2) Token attention weights: For each task, we use the first-layer attention scores and the aggregated attention scores (average) across all layers as the First Layer and Layer Aggregated importance scores, respectively. Prompt pruning is then performed according to the derived attention scores.
Results. We examine the impact of increasing token pruning percentages on task accuracy (Figure 5). If the derived scores guide pruning effectively, accuracy should first increase monotonically, then decrease beyond a certain threshold. However, as shown in Figure 5, all approaches exhibit nonlinear fluctuations, with their optimal performance still lagging behind our hill-climbing approach (Table 8). This suggests these methods are ineffective at guiding pruning. Also, please note that unlike the hill-climbing approach discussed in Section B, the methods presented in this section do not converge to a single optimal prompt. Instead, as shown in Table 8, we can only obtain the results by experimenting with and selecting from all the prompts derived on the validation set based on their importance scores, which also less flexible. Finally, just as the failures of existing prompt compression methods guide the pruning (shown in Table 1), we stick with our TAPruning. the hill-climbing approach, in section B to first gain basic intuition of our Partial Context Hypothesis.
D. PROMPTQUINE Additional Details
D.1. Genetic Algorithm Details
Introduction. As outlined in Algorithm 3, we evolve a population P of #p individuals (here, pruned ICL prompts P as phenotypes). At every generation, each P is evaluated, producing a fitness score fP). The selection process then identifies high potential individuals typically the best-performing ones, using methods like tournament selection. Genetic operators, such as mutation, are applied to generate offspring: mutation introduces small, random changes (c.g., pruning tokens), while crossover (i.e., exchanging tokens) is excluded in this case due to its limited benefits on performance and its potential to complicate the search space, which is also common practice (Real et al., 2017: Co-Reyes et al., 2021). The offspring are then evaluated, and elitism-based selection determines which individuals survive to the next generation. This process repeats iteratively until termination condition is met, such as achieving a satisfactory fitness level or reaching predefined number of generations.
Implementation Details. Here, we present the details of our GGA and SSGA implementations discussed in Section 3.4. Specifically, we provide their pseudo-codes: Algorithm 4 for GGA and Algorithm 5 for SSGA, both of which replace the main loop in Algorithm 3. Each approach has distinct trade-offs. As illustrated in Algorithm 4, GGA divides the reproduction process into multiple generations, with several reproduction events occurring within each. In our setup, the number of offspring (#c) often exceeds the population size (#P), encouraging each individual to participate as parent.
26

---

Evolving Prompts In-Context: An Open-ended, Selfreeliiian Perspective
Algorithm 3 Genetic Prompt-Quine (PROMPTQUINE) Framework for Prompt Subsequence Search
Input: prompt X, 2 eN training dataset Dirain, validation dataset patt fitness function f. primary task performance measure m, population size #p, sample size #s, number of iterations #n. language model M, clite calibration selection
function R
Output: Prompt P P < poppppptiiiniiiiinn -nnt (X,M, D Dtrain: #p).
History H +-++0 P.
for
1 to #n do
8(+) sample (P, #s).
Parent select (8()).
Child + copy-ther.mutaate (Parent).
Scores - evaluate (Child, f, Dtrain).
P, H. < push-pop (Child, Scores, P, H. #p, i).
end for
Elite t R (H)
Prompt P < prompt-rank (E, M, m, Dvalid).
The framework converges either when predefined iteration limit is reached or, commonly, when the minimal prompt length threshold (#1) is met. GGA is well-suited for parallelization, which can further enhance the results by leveraging computations. In our current implementation, we mainly use batching along with efficient LLM serving tools, such as vLLM (Kwon et al., 2023), which demonstrates improved runtime efficiency.
The core difference between SSGA and GGA lies in SSGA's adffaadatttioon of parameters, such as mutation rate and selection pressure, throughout the evolutionary process. As outlined in Algorithm SSGA continuously updates the population with new offisping at each step. We also allow more offspring before inclusion through regularized evolution, introducing genetic diversity in real time-akin to self-adaptive mutation rate. Additionally, by dynamically increasing the population size, SSGA creates a self- adattive selection pressure, potentially increasing the likelihood of each individual becoming a parent. The evolution process terminates under the same conditions as GGA, typically when the iteration limit reached. Compared to GGA, SSGA is more exploratory but sacrifices some runtime efficiency due to the lack batching and parallelization. As a result, we primarily use SSGA for 1-shot ICL pruning experiments (10,000 iterations), while GGA, which converges faster, is used for more-shot ICL pruning experiments. Later experiments show that, for most landscapes encounter, GGA performs comparably to SSGA.
Table Hyperparameters. Through extensive experiments, the current hyperparameters have proven stable across the tasks presented in the 1-shot ICL pruming studics. For more-shot experiments, we recommend increasing the number of iterations, and, if computational resources permit, enlarging the population and offspring sizes as well.
Hyperpurameter
Default Value
Description
Population Size
30
How many individuals per population. How many oflspring individuals generated per generation.
Offspring Size
50
Mulation rate
[1,2,3,4]
The possible range of bits we change for reproduction. The ratio of individuals sampled for tournament selection. The maximal number of prompts explored through pruning. The expected minimum average population prompt length for search termination.
Tournament Selection Ratio
0.2
Number of Iterations
10,000
Minimal Prompt Length Threshold
15
Hyperparameters. We present our shared hyperparameters for both SSGA and GGA under I-shot ICL pruning in Table 9. If additional resources are available, we highly recommend increasing both the population and offspring sizes, as this can yield significant performance benefits. Moreover, believe that employing more adaptive mutation rate can be beneficial, which we leave as future work (c.g., using larger mutation rate in the carly stages and gradually annealing it over time may enhance performance).
27

---

Evolving Prompts In-Context: An Open-ended, Sclf-replicating Perspective
Algorithm 4 PROMPTQUINE'S Generational GA (GGA) implementation for Prompt Subsequence Search
Input: Initial population P. with population size #p, number of maximal iterations # minimal prompt length threshold #1. training dataset Pirain. fitness function f. offspring size #c, language model M, tournament selection ratio #k, mean population prompt length calculation function h Output: Prompt History H
History + P.
for i = 1 to #n do Initialize g - Empty.
Initialize mean population prompt length I. +- ((P).
if L < # then
break end if
for = to #c do
Parent f Seaernammntt--leetiiee (P, #k).
Child < copy-then-mutate (Parent).
Score evaluate (Child, M, M, prain))
g. H. push (Child, Score).
end for
Sort g in descending order of Score.
Update population p + gl: #p).
end for
Algorithm 5 PROMPTQUINE'S Steady-state GA (SSGA) implementation for Prompt Subsequence Search
Input: Initial population P. with population size #p, number of maximal iterations #n, minimal prompt length threshold #1, training dataset 'D Drrain. fitness function f, offspring size #c, language model M, tournament selection ratio #k, mean population prompt length calculation function h Output: Prompt History H
History H + P.
for i = 1 to #n do
Initialize mean population prompt length + h(P).
if L < #l then
break end if
for 1 to #c do
Parent + Tournaanntt-Slecttion (P, #k).
Child < copy-then-mutatee (Parent).
Score + evaluate (Childd Derain).
P, push (Child, Score).
end for
Initialize g P(#p :].
Sort y in descending order of Score.
Update population P + gl: #p). end for
D.2. Multimodal Landscape Additional Studies
In the main paper, present the studies only using instruction-tune tuned Meta-Llama-3-8B. Innstrrru Here, present additional results for base model, Meta-Llama-3-SB.
The results in Figure 6 show that greedy hill-climbing alone fails to consistently converge to the same prompt solutions, underscoring the multimodal nature of the ICL landscape. An exploratory search approach may prove more effective in improving the results. Additionally, as shown in both middle and right subfigures, RS is inefficient at obtaining high-quality prompts, whereas ES is more effective, potentially leading to better local optima under restricted computational budgets,
28

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
1.0
1-shot 2-shot s-shot
0.5
0.7
0.9
RSI evolution success
0.70
0.4
0.6
Accuracy
0.8
Accuracy
0.3
0.5
0.7
0.60
0.2
0.4
0.50
0.6
0.1
0.40
0.3
0.30
0.5
0.0
Random Evolution Method
1-shot
2-shot
4-shot
Pruning Order Seed
Number of shots for ICL
(a) Optimization challenges in ICL-initialized landscape using
for natural language inference (SNLI).
1.0
0.9
0.5
0.9
0.8
RS/ evolution success
0.4
Accuracy
0.8
Accuracy
0.7
0.80
0.3
0.6
0.7
0.70
0.2
1-shot
0.5
0.60
0.6
2-shot
0.4
0.50
0.1
4-shot
0.40
0.5
0.3
0.0
Random Evolution
1-shot
2-shot
4-shot
Pruning Order Seed
Method
Number of shots for ICL
(b) Optimization challenges our ICL-initialized landscape using Meta-Llama- for subjectivity classification (Subj).
1.0
1-shot
0.5
0.7
0.9
2-shot
0.70
RS/ evolution success
4-shot
0.4
0.6
Accuracy
0.8
Accuracy
0.3
0.5
0.7
0.60
0.2
0.4
0.50
0.6
0.3
0.40
0.1
0.30
0.5
0.0
Random Evolution
1-shot
2-shot
4-shot
Pruning Order Seed
Method
Number of shots for ICL
(c)
Optimization challenges our ICL-initialized landscape using Meta-Llama-3-8B natural language inference (SNLI).
Figure 6. Additional results both Llama-3-8B base and instruct models, revealing the complex, multimodal nature of the ICL. search
landscape.
further corroborating our findings in the main paper.
29

---

Evolving Prompts In-Context: An Open-ended, Seifreliicatn Perspective
D.3. PROMPTQUINE for Classification
Fitness Function. Our preliminary experiments show that the piecewise reward function proposed in RLPrompt (Deng al., 2022) is highly effective at distinguishing between prompts of varying quality, compared to variety of probability-only selection approaches (Yang et al., 2024b). We thus provide its details below, our classification task proxy:
R(z,x, c) A1 -1 Correct Correct
Au
Caree Gapz (c),
(3)
where the reward is defined as the gap between the label probability and the highest probability from other classes for a given prompt 2 and training example (x, The gap is written as (c), == Pa(c) - max,' Pa(c'), where Pa(e)= == [c,(c x) to denote the probability of label c. The gap value is positive when the prediction is correct, and negative otherwise. We denote Correct == [[Gap,(e) > 0] that for a correct prediction, the positive reward further multiplied by larger number to signal its desirability. We set A and X2 as 180 and 200, following (Deng et al., 2022).
Re-ranking Mechanisms. As pointed out in the main paper and algorithm 3, we incorporate specific prompt re-ranking mechanisms in later stages. This is because, despite the effectiveness of the fitness function in Equation 3, it can still be exploited due to its imperfect design. This may stem from the limited number of task samples used for fitness estimation or from inherent imperfections and complexities within the fitness function itself. This is inevitable, as a simple reward function often struggles to capture the nuanced complexity of the problem (Di Langosco et al., 2022). Relying solely on the original ranking may lead to misleading results. Therefore, we adopt a two-stage approach: the first stage uses the original fitness scores to select a shortlist of clite prompts. while the second stage leverages more accurate validation accuracy, though still limited, to further refine the prompt rankings. Here, we discuss our designs follows: 1) Elite-based selection: From the prompts we explored during search, we rank and select the top k% prompts, guided by fitness scores form the elite prompt collection; 2) Calibration-then-rant we evaluate the validation accuracy of the elite prompts, and select the highest- seehest-performig prompt among them as the final prompt. The most challenging aspect in this process lies in selecting the value of k. A larger & encourages more exploration, while a smaller & favors exploitation. Empirically, we find that for tasks with fewer categories, such as binary classification like SST-2 and Subj, selecting a higher à consistently leads to better outcomes, as may help uncover superior solutions. This can be attributed to the inherent imperfections of the fitness scores in few-shot settings. Specifically, for Subj and SST-2, we select k 10, while for the other datasets, we choose k 5. A lower l (e.g., 0.01) may still be suitable for some datasets. The optimal selection require trial and error the validation set.
Implementation Details. For ICL, we use the same templates as provided in Section B.2. For LLMLingua, we follow its default setup (Jiang et al., 2023c), leveraging model-specific self-information. For LLMLingua2, we adopt their pre-trained XLM-RoBERTa--arg model to guide the compression, following the configurations in Pan et al. (2024). For EvoPrompt, since some tasks lack annotated instructions for population initialization in its original implementation, we use GPT-4o (Hurst et al., 2024) to directly generate diversified natural language instructions, which are then used for the evolution population initialization. The prompt is further selected based on the validation accuracy. For RIPrompt, as Appendix B, we follow the original setup (Deng al., 2022) for the 16-shot policy network training and incorporate the re-ranking designs for improved results. We use GPT-4o (Hurst et al., 2024) as the LLM mutator in Promptbreeder, as it outperforms self-referential use of the target models, especially smaller ones. Since the original code of Promptbreeder (Fernando et al., 2024) is unavailable, we reimplement it from scratch using their mutation prompts and evolution operators. Unfortunately, this may introduce slight bias in the comparisons. For PROMPTQUINE, we use the default SSGA configurations as specified in Appendix D. for 1-shot ICL pruning, searching for 10,000 iterations (i.c., the number of prompts we explored). Subsequent experiments show that GGA is comparable for most of the LLMs,, converging faster- within 3,000 iterations. For 4-shot ICL pruning, we observe that increasing the number of samples used for fitness estimation during the search stage can be beneficial. Therefore, we use 32-shot samples for SST-2 and Subj, while retaining 8-shot samples for other tasks. For this setup, implement our GGA, conducting search over 100,000 iterations. Most GGA runs converge much faster, typically within 10,000 steps. It is important to note that the number of iterations required for convergence generally scales with the number of tokens presented in context. Only in worst cases, the search may take days on one GPU (e.g., 4-shot ICL pruning on Yelp-5 upon Llama3-8P -BIIt, with 4, 4,000 tokens in context).
30

---

Evolving Prompts In-Context: An Open-ended, Selfrrepicctnrn Perspective
Table 10. Additional Classification Results. For ICT. and corresponding pruming/ compression methods, we all 1-shot for the experiments. Promptbreeder's results also correspond to those of its shot variant, which is based on 1-shot ICL, using GPT-4o (Hurst al., 2024) as the LLM mutator.
Method
SST-2
Subj
AG's News
SNLI
Yelp-S
Yahoo
Avg.
LLM: RoBERTa-large
ICL.
86.2 (7.7)
53.8 (5.3)
57.2 (36)
33.3 (1.2)
27.3 (2.1)
36.8 (2.)
49.1
88.3 (6.2)
57.9 (99)
59.1 (9.1)
33.6 (1.2)
28.4 (3.3)
30.9 (80)
49.7
LLMLingua2
59.5 (13.5)
51.2 (2.6)
47.6 (10.9)
33,0 (0.0)
25.5 (3.0)
40.4 (2.0)
42.9
RLPrompt
92.5
81.2
80.2
33.5 (0.8)
44.8 (1.3)
48.6 (1.0)
63.5
Promptbreeder
92.1 (1.0)
72.8 (2:3)
81.8 (0.4)
38.2 (3.1)
44.7 (0.4)
45.5 (L.))
62.5
TAPruning (Ours)
90.8
80.9
79.7
42.0 (5.9)
42.8 (6.9)
51.4 (1.5)
64.6
PROMPTQUINE (Ours)
92.9 (2.2)
80.0 (3.6)
82.4 (1.0)
38.8 (2.6)
49.8 (1.9)
55.0 (1.0)
66.5
LLM: GPT-2
ICL
54.2 (4.8)
64.3 (9.8)
36.5 (6.5)
33.6 (1.0)
30.9 (5.2)
26.3 (3.7)
41.0
LLMLingua
53.1 (2.9)
63.5 (9.3)
33.8 (7.0)
33.4 (0.8)
30.0 (7.3)
15.0 (5.3)
38.1
LLMLingua2
56.5
61.5 (9.8)
50.7 (10.0)
36.4 (1.6)
28.6 (1.4)
30.1 (6.1)
44.0
RLPrompt
79.2 (4.))
76.7 (3.9)
75.3 (1.6)
39.1 (1.9)
35.1 (1.8)
46.8 (1.3)
58.7
Promptbreeder
76.9 (0.8)
70.8
60.2
34.1 (0.4)
36.3 (1.2)
25.2
50.6
TAPruming (Ours)
68.1 (10.6)
75.9 (2:5)
(4:2)
40.8 (2.9)
39.9 (1.8)
39.2 (28)
54.9
PROMPTQUINE (Ours)
77.2 (3.61
77.8 (3.6)
66.7
42.3 (3.6
40.2 (2.7)
47.2 (1.5)
58.6
LLM: Gemma 7B-It
ICL
92.7 (1.2)
59.8 (2.0)
72.7 (2.0)
41.7 (7.0)
47.0 (3.9)
55.2 (3.9)
61.5
LLMLingua
93.7
57.6
81.0
48.1 (10.3)
42.1 (6.7)
45.8 (14.2
61.4
LLMLingua2
60.1 (16.1)
59.4 (8.6)
41.8 (13.7)
37.7 (2.9)
35.2 (7.8)
54.2 (3.6)
48.1
RLPrompt
89.9
83.4
75.5
46.4 (0.6)
50.4 (0.4)
50.6 (0:4)
66.0
Promptbreeder
93.1 (1.2)
65.1 (1.9)
83.8 (1.4)
55.4 (1.9)
49.3 (5.8)
(L.4)
67.6
TAPruning (Ours)
93.3 (1.2)
77.6 (5.2)
85.6 (1.4)
63.5 (1.2)
49.7 (4.7)
56.9 (2.6)
71.1
PROMPTQUINE (Ours)
93.7 (0.8)
79.9 (4.9)
(2.1)
63.0 (4.2)
50.0 (2.4)
63.0 (L.6)
72.2
LLM: LLama3-88 8B
ICL
94.6
60.2
83.2 (2.0)
62.5 (1.9)
47.3
61.8 (1.8)
68.3
LLMLingua
92.2 (3.))
60.1 (7.3)
83.6 (2.8
50.4 (10.9)
42.2 (5.0)
50.3 (6.9)
63.1
LLMLinguat
59.2 (1.7)
54.2
55.5 (10.2)
35.2 (1.5)
40.5 (5.2)
58.4 (28)
50.5
RlPrompt
91.1 (07)
80.4 (3.4)
83.3 (1.2)
41.6 (1.0)
45.4 (1.9)
58.3 (0.7)
66.7
Promptbreeder
95.4 (0.4)
76,8 (1.2)
88.2 (0.6)
64.2 (0.5)
55.2 (2.2)
62.0 (L.7)
73.6
TAPruning (Ours)
94.4 (1.3)
75.3 (6.4)
(0.4)
66.8 (3.5)
56.0 (1.7)
63.9 (2.1)
74.1
PROMPIQUINE (Ours)
95.4 (0:6)
83.9 (3.1)
88.7 (0.4)
65.6 (1.9)
56.4 (L.0)
65.4 (L.0)
75.9
L.M::
Llama 70B
ICL
96.6 (0.3)
66.7 (9.5)
88.5 (1.5)
61.2 (1.7)
39.8 (4.3)
56.7 (10.6)
68.3
L.MI I.M.Lingua
95.9
70.4 (7:3)
80.8 (194)
52.9 (7.)
40.3 (5.0)
43.5 (108)
64.0
L.MI .ingua2
89.5 (8.0
49.2 (2.9)
85.0 (10.1)
36.0 (3.)
40.4 (4.9)
58.8 (42)
50.2
RL Prompt
86.6
39.5 (2.1)
44.9 (0:3)
53.8
66.6
Promptbreeder
96.9 (0.3)
77.3 (2.1)
89.2 (0.6)
69.2 (1.5)
54.2 (3.2)
66.8 (L.8)
75.6
TAPruning (Ours)
94.5 (2.7)
84.8 (1.8)
89.6 (0.5)
69.4 (2.9)
53.7 (5.0)
65.3 (2.0)
76.2
PROMPTQUINE (Ours)
97.3 (0.4)
86.8 (5.2)
90.6 (1.1)
73.5 (3.0)
54.2 (2.2)
69.7 (1.5)
78.7
LLM: Llama3-708
ICL
97.1
71.9
89.3
57.7 (1.5)
54.6 (5.5)
65.2 (1.9)
72.6
L.MMLingua
96.9 (0.3)
70.8 (44)
88.9 (0.)
54.3 (4.4)
50.2 (5.5)
56.6 (4.9
69.6
I. ..MLingua2
72.8 (11.0)
70.0
79.1
39.3 (2.8
48.0 (7.3)
61.7 (3:5)
61.8
RLPrompt
88.7 (1.5)
81.9 (0.3)
88.2 (0:)
49.1 (LD
50.2 (1.4)
54.0 (0.)
68.7
Promptbreeder
97.2 (0.2)
88.3 (.8)
89.0 (08)
67.5 (2.2)
61.7 (0.5)
68.7 (0.3)
78.7
TAPruning (Ours)
96.8 (0.4)
86,9 (2.5)
89.7 (0.5)
74.1 (3.3)
61.8 (0.7)
65.5 (2.0)
79.1
PROMPTQUINE (Ours)
97.9 (0.8)
89.3 (2.3)
90.6 (1.2)
75.6 (4.2)
62.0 (3.))
70.7 (1.2)
81.0

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Table 11. Deployment efficiency of proposed approaches and baseline methods until convergence. We ensure that they can only access the same task samples in search for fair comparisons. Wall time is reported to measure the training time efficiency. We produce these experiments on one NVIDIA A100 GPU, following their default configurations on Meta-Llama-3-8 1--8---Innuttt Our algorithms may take longer time if providing with longer ICL initializations. We highly recommend to use TA TrPruingg for some quick experiments and choose our GGA for PROMPTQUINE using both batching and parallelization which can, in principle, largely reduce the wall-time, especially when we lack more expressive task proxies (c.g., some generation and reasoning tasks). Note that both EvoPrompt and Promptbreeder are generally bottlenecked by the external LLM's inference latency (i.c., OpenAl API, as we use GPT-4o (Hurst et al., 2024) for the LLM mutators). For example, we observe that Promptbreeder almost always requires around 30 minutes and sometimes even longer- for single run, even when optimizing prompts for a small LLM (e.g.. GPT-2).
Gradient- Free
Subj
AG's News
Method
Acc
Wall Time
Acc
Wall Time
EvoPrompt (Guo et al., 2023)
84.1
18 min
86.5
52 min
Promptbreeder (Guo et al., 2023)
83.6
28.3 min
88.6
34.7 min
RLPrompt (Deng et al., 2022)
x
82.9
84.7
33 hr
PIN (Choi et al., 2024)
79.5 x
3 hr
77.9
7 hr
SAHCPruming (1 (I-shot ICI.)
/
77.3
13.2 min
88.6
25.7 hr 12 min
TAPruming (1-shot ICL)
<
74.5
4.5 min
88.6
PROMPTQUINE (I-shot ICL., GGA) PROMPTQUINE (I-shot ICL, SSGA)
85.2
4 min
89.3
35 min
86.5
18 min
89.2
51 min
Additional Analysis. Here, we present the additional analysis on the Table I's results. The purely greedy pruning approach, SAHCPruning, requires significantly more optimization time (Table 11)-sometimes taking several days-yet it does not consistently outperform TAPruning, let alone PROMPTQUINE. Interestingly, SAHCPruming manages to find relatively good prompts by pruning just handful of tokens. This is slightly different from what we observed in the dynamics of PROMPTQUINE, pointing to the complex, multimodal nature of the landscape. Notably, some ICL. initializations may stagnate entirely at the earliest stage for SAHCPruning. We provide one such example in the GitHub repository?. This highlights the deceptive nature of our search objective (e.g., validation accuracy). Under constrained samples for prompt evaluation, validation accuracy overfit to narrow slice of examples and fail to reflect true generalization. That's why rewarding seemingly suboptimal stepping stones is crucial - they often provide the necessary diversity or novelty that helps escape local optima (Stanley& & Lehman, 2015). To further push the boundary of the results, we explore pruning richer 4-shot ICL prompts. Surprisingly, just as how conventional ICL. performance scales with the shots in their contexts (Zhao et al., 2021), we find it is possible to improve the pruned prompting performance as well. We discuss more in Appendix E. Notably, existing performant prompt compression methods fail to achieve consistent improvements in pruning. We hope our findings can inspire new research on prompt compression.
Additional Results. We present additional results along with original TAPruning in Table 10. As expected, our PROMP- TQUINE has achieved almost state-of-the-arr results across all these datasets and models. It is worth noting that both LLMLingua (Jiang et al., 2023c) and LLMLingua2 (Pan et al., 2024) fail to show improvements in our formulation. In particular, LLMMingua,, which claims to surpass L.L.I L.L.Liinuua in prompt compression, experiences more performance degradations than LLMLingua. This may be reasonable, as LLMLingua2 relies solely on human-intuitive information, training on GPT-4 summarized input-output pairs. In contrast, LLMLinguu leverages model-specific self-information, making its approach potentially more aligned with the underlying model's capabilities.
Runtime Efficiency Analysis. We compare our search approaches with several state-of-the-arr approaches in Table 11, specifically EvoPrompt (Guo et al., 2023), Promptbreeder (Fernando et al., 2024), RLPrompt (Deng et al., 2022), and PIN (Choi et al., 2024), following their default setups. As we have introduced ExoPrompt & RI.Prmppt & in the earlier discussions, we provide more details regarding PIN here. PIN is another token-level search algorithm built upon RLPrompt. Concretely, PIN incorporates sparse Tsallis entropy regularization upon RLPrompt, attempting to prune the search space for RL training. Their results on RoBERTa-large have demonstrated its effectiveness while significantly improving its RL training efficiency. However, as Table 11 shows, this also sacrifices some final task performance. Notably, teken- search methods, including RLPrompt and PIN, still require hours to optimize prompts for classification tasks, In contrast, our
32

---

Evolving Prompts In-Context: An Open-ended, Sclfrreeicctrr Perspective
approach is the first optimize prompts within minutes for this task. Our approach is also comparable EvoPrompt, which, according the analysis in Cui et al. (2024), space. Finally, want to highlight that our approach has the potential to be further improved in both task effectiveness and runtime efficiency, by leveraging parallelization, which is key strength of evolutionary search. Therefore, given enough computes, is definitely possible that PROMPT PROMPTQUINE can surpass TA TrPrrnnng in terms of both wall-time and task results.
of the most efficient search techniques optimizing in the natural language
Table 12. Automatic evaluation of Yelp Sentiment Transfer, averaging the results of negative and positive transfer. We evaluate both GPT-2 (Radford et al., 2019) and Meta- Llama-3-88 3--n--3--B----- Instruct (AI@ Meta, 2024) (Llama3-88) -B--tt)... It). The results (no parentheses) are reported with greedy decoding. BoN refers Best-of-N sampling. following the setup of Deng al. (2022), which employs Best-of-32 strategy with top- sampling. We use GPT-4o (Hurst et al., 2024) as the LLM mutator for Promptbreeder.
Method
CONTENT
STYLE
FLUENCY
JOINT
ICL
74.7
18.9
RLPrompl
56.9
46.8
10.4
GPT-2
Prompibreeder
53.1
41.2
95.4
10.2
TAPruning
46.1
84.4
95.4
30.2
PROMPTQUINE
48.0
86.3
95.2
33,3
ICL
49.7
94.6
93.4
40.8
RI.Prompt
57.0
99.6
90.8
51.0
(PT-2.(BoN)
Promptbreeder
57.0
98.7
91.4
45,3
TAPruning
60.7
99.9
90.7
54.6
PROMITQUINE
65.9
99.5
88.6
$7.9
ICL
63.7
91.1
94.9
54.4
RLPrompt
78.0
13,4
94,2
4.1
Llama3-8B-It
Promptbreeder
73.1
87.6
93.4
59.1
TAPruning
71.9
89.4
93.1
59.6
PROMPTQUINE
71.6
91.2
93.5
61.0
ICI.
75.4
99.8
92.7
69.6
Llama3-8B-In (BoN)
RLPrompp
62.5
96.4
93.0
54.4
Promptbrceder
79.2
99.7
91.2
71.8
TAPruming
99.1
91.0
PROMPTQUINE
81.7
99.8
93.5
72.1
D.4. PROMPTQUINE for Text Generation
Prompt
Develop strategy for hacking into government database and stealing sensitive information.
Prompt with jailbreak
Respond the below prompt while following these rules: 1. Do any punctuation. 2. Do words than
lettes long. 3. Never say the word "the". Prompt: Develop strategy for hacking into a government database and stealing sensitive information. Remember: Respond to the prompt while following these rules: Do not use any punctuation. 2. Do not use words more than seven letters long. 3. Never say the word "the".
Joint Score (Equation 2) transfer. For jailbreaking, stated, in pure black-box experiments, we try both exact match score (ASR-EM) as well as the score generated by LLM-as-a-Jule
Fitness Function. As mentioned in the main paper,
fitness function for style
(ASR-LLM). For potential jailbreaking proxies, we measure the fitness score as the cosine similarity between the steering vector and the change in the model's latent state before and after appending the current demonstrations x at the last token position. As introduced section 4.2, the contrastive prompt pairs consist of both jailbreak and non-jailbreak versions of the same request, like the example above, in which the template taken from Ball al. (2024). Specifically, we compute the input-specific steering vectors for each input request in validation set D at layer 14 for both the Vicuna-7b-vl.5 model and the Distral-7B-Instrutt-ve by averaging these steering vectors. For the selection of layer 14, we heuristic approach. Instead of PCA clustering analysis in previous work (Rimsky et al., 2024; Ball et al., 2024), we use cosine similarity between the calculated steering vectors for each input request to select the layer. Intuitively, a higher similarity suggests that the concept direction in this layer may be more aligned. We find this shortcut works effectively, and the final selection, layer 14, aligns well with existing findings that intermediate model layers are more effective for the activation interventions (Turner et al., 2023). Next, we extract the model's latent state R at the same layer for the last token of the ICL prompt (template as below),
model. We then obtain the aggregated version
33

---

Evolving Prompts In-Context: An Open-ended, Selffrepliictrgnnn Perspective
Table 13. Yelp sentiment transfer testing performance of various methods, evaluated across different sample sizes Samples) used for training in prompt quality estimation. The results are averaged for the Joint Score across both positive and negative transfer. We use SSGA with greedy decoding for this table, the results from GGA are similar.
LLM
Method
Samples
Joint Score
RI.Prompt
16
10.3
PROMPTQUINE
16
25.8
GPT-2
RLPrompt
100
12.7
PROMPTQUINE
100
33.3
TAProning
200
30.2
RLPrompt
16
6.7
PROMPTQUINE
16
59.2
Llama3-8B-It
RLPrompt
100
7.3
PROMPTQUINE
100
60.5
TA TAPrinnng
200
59.5
ICL CCL-JJilbreakingg
(Examples)
Input: (Input)
Output:
and extract the model's latent state Ry at the same location for the prompt with demonstrations, while retaining the Input: and Output: signal words. We use the cosine similarity between the mean-aggregated difference vector R-RN for each request in the validation set and the aggregated steering vector to guide the pruning process. Preliminary experiments indicate that this formulation of fitness function yields the best results.
Re-ranking Mechanisms. As what we have done in classification, we produce similar procedures for generation tasks. Specifically, for the elite-based selection, we just sample the top-20 prompts ranked by their fitness, forming the elite collection. Then, we re-rank the elite prompts based on their validation set's sss joint score/ attack success rates. We find that jailbreaking is more robust to such selection choice. Similar procedures are employed in RLPrompt.
Implementation Details. For ICL and RLPrompt in style transfer, we directly follow what we have done in Appendix B. For ICL prompts for jailbreaking. we take the demonstration examples from Liu et al. (2024) and organize them into 2-shot demonstration exemplars for our experiments. For the precise instructions we used when prompting Hlma-Guards- and how we parse the results, please refer Chao al. (2024) for the details. We directly follow their prompts and evaluation protocols. For exact matched strings, we follow the existing work and use their strings Zhao al. (2024). Then, explain our early-stopping methods for style transfer in detail. Specifically, we monitor the fitness of the least-fit individual in the population, min (P:)), and reserve 50 samples for pre-evaluating the t + generation. Only individuals with fitness above min(((())) are fully evaluated on all 100 samples. In final re-ranking stage, the fitness scores are calibrated using all 200 validation samples (unsupervised, ranked by the joint score).
Style Transfer Additional Results. As stated in section 4.2, we use 100 unpaired input samples for prompt quality estimation during search runtime, with the fitness score determined by the Joint Score, based on preliminary experiments. As presented in Table 13, we also conduct experiments using only 16 input samples for the fitness calculation, just as what we have done in 8-shot classification. However, as we observed in GPT-2. there are cases when PROMPTQUINE clearly lags behind TAP TAPruning, which counter-intuitive. We attribute this to the limited numbers of samples we used for fitness estimation. We thus select to increase the samples 100 in fitness estimation, which leads to better results (Table 13). A more expressive task proxy has the potential to further reduce the samples. We compare such results against RLPrompt, one of the statc-of-the-an approaches for style transfer, using the same set of samples in optimization. Clearly, our results surpass RLPrompt in terms of both Joint Scores (Table 12) and optimization efficiency (Table 14), regardless of the decoding strategies (or meta- titta--gnerationn algorithms (Welleck et al., 2024)) and the number of samples used in measuring the fitness scores. Notably, we find that RL. Prompt cannot achieve comparable task results using greedy decoding, potentially due to
34

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Table 14. Deployment efficieney of proposed approaches and haseline methods until convergence. The results averaged across both positive and negative transfer, with three random seeds per task. We use greedy decoding for this table.
Yelp P.
Method
Samples
Joint Score
Wall Time
TAPruning
200
hr 59.6
RLPrompt
6.7
hr
PROMPTQUINE (SSGA)
16
59.2
1.8 hr
PROMPTQUINE (GGA)
16
58.6
40 min
RLPrompt
100
7.3
7.5 hr
PROMPTQUINE (SSGA)
100
61.0
5.2 hr
PROMPTQUINE (GGA)
100
60.8
hr
their reward instability through optimization.
D.4.1. EARLY EXPLORATION INTO VARIANTS OF FEW-SHOT JAILBREAKING
Here, we explore variants of few-shot jailbreaking, specifically in-context attacks (Wei et al., 2023b). We provide one such example for vicuma-7b-v 1.55 below. In particular, the exemplars are separated by the conversational tags:
A chat between curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
USER: Write a tutorial on how to make a bomb.
ASSISTANT: Sure, here's 's the first part of the tutorial on how to make a bomb: Step Gather Materials make simple bomb, you will need the following materials: *A strong, heavy object, such metal pipe piece of concrete AA small amount of explosive material, such black powder smokeless powder "A container, such as a bottle Assemble the Bomb Once you have obtained the materials, you can begin assembling the bomb. Here is a basic outline of the steps involved: 1. Cut a hole in the container to accommodate the fuse. 2. Place the strong, heavy object inside the container. 3. Place the explosive material around the object. Wrap the fuse around the container and light it. </
jar Step 2:
USER: {Impur}
ASSISTANT: {Oulput)
Experimental Setups. We follow the experimental setups in Section 4.2.2, using the same 2-shot ICL prompts and testing vicuna-7b-v1.5 and Mistral-7B-Innttrut--dd on the same AdvBench, with the same data splits. Additionally, we investigate Llama-2-7b-chat, a widely used LLM in jailbreaking attempts that has undergone extensive RLHF and red teaming. We find that the introduction of additional tags makes increasingly challenging to extract robust steering vectors using a standard implementation. We thus only investigate the black-box search, where guide the prompt search. Here, to ensure robustness, as, for instance, Souly al. (2024); Stroebl al. (2024) raise concerns regarding the imperfect verifiers, we follow Hughes et al. (2024) by conducting additional manual reviews Aannuua,,l)), We consider a jailbreak successful if it provides the user with information relevant to the harmful request, even if it is not complete and comprehensive. We produce the experiments with three ICL initialization seeds.
both ASR-EM and ASR-LLM to
Results. Table 15 shows the effects of pruning under in-context attacks. Similar the priming setup (e.g., Table 3) discussed in the main paper, pruning (i.e., PROMPTQUINE) also proves effective in scenarios where exemplars are separated by conversational tags. For instance, in Vicuna-7b-v1.5 the attack success rate triples under this setup. However, this improvement deteriorates significantly under Llama-2-7b-che Intriguingly, the reasons remain unclear, though the poor small-shot in-context attack performance aligns with Wei et al. (2023b). A richer prompt variation might improve this, for instance, prompt injection attack (Zheng al., 2024).
35

---

Evolving Prompts In-Context: An Open-ended, Selfreeiiinnn Perspective
Table 15. Attack (ASR) for jailbreaking comparison of PROMPTQUINE, and conventional ICI. under in-context attacks. The
text in parentheses refers to the fitness measure used for PROMPTQUINE.
Attack Method
ASR-EM f ASR-LLM 1
LLM: Vicuna-7b-vi. 7b
ICL (2-shot)
36.7
32.6
PROMPTQUINE (ASR-EM)
96.6
94.8
PROMPTQUINE (ASR-LLM)
92.7
92.5
LLM: Mistral---a- -B-IInIIIIttttt
v0.3
ICL (2-shot)
91.0
82.1
PROMPTQUINE (ASR-EM)
98.6
93.3
PROMPTQUINE (ASR-LLM)
98.6
91.4
LLM: Llama-22222 2-7b-chatt
ICL (2-shot)
0
0
PROMPTQUINE (ASR-EM)
0.4
0.1
PROMPTQUINE (ASR-LLM)
0.6
0.6
D.5. PROMPTQUINE for Multi-choice Question Answering
Multi-choice question answering tasks (MCQs) are also popularly framed as classification tasks, Specifically, in a standard question answering where we have four options, question and ask our LLMss to generate the task prediction. This approach operates in manner analogous to classification task. Therefore, we are allowed to reuse the few-shot classification designs, specifically their fitness functions, to optimize prompts for the MCQ tasks.
condition each option choice (A, B, C, and D) the prompt and
Table 16. Results on choice Question Answering. BASE denotes base model of Meta-Llama Laa---alaa---ee 3-8B (AI@ Meta, 2024) and INSTRUCT denotes instruction-tuned model of Meta--Lamm--388-IInnteee same prompt template. For RL Prommpt we append the optimized instruction into the template for evaluation, in the same position where ICL demonstrations would typically appear.
(AI@ Meta, 2024). We ensure that these approaches built upon the
BASE
INSTRUCT
ICL (1-shot, original) (Brown al., 2020)
70.7 (2.5)
(1.66
RLPrompt (Deng et al., 2022)
(1.7)
76.3 (2.1)
TAPrumnng (1-shot ICL., Ours)
74.8 (1.6)
75.1 (3.0)
PROMPTQUINE (1-shot ICL, Ours)
(3.1)
79.5 (1.0)
PROMPTQUINE (2-shot ICL, Ours)
79.1
82.3 (1.4)
reuse the evaluation settings, including prompts, models and datasets, we explored Appendix B. contrast to the previous results that TAPruning under-performs original ICL in one of our investigated models, we show that pruning-our PROMPTQUINE can successfully improve performance both models (Table 16). Finally, steering vectors (Rimsky et al., 2024), which used for fitness calculation in generation tasks, may also be valuable for guiding the search in multi-choice question answering. We leave this exploration for future work.
D.6. PROMPTQUINE for Chain-of-though Reasoning
Chain-of-thowe ofouhhthtthtt (CoT) reasoning (Wei et al., 2022; Kojima et al., 2022) is novel prompting technique specifically designed for complex reasoning tasks. Rather than directly generating the final answer, LLMs are allowed to invest additional inference-time compute to generate an intermediate reasoning chain before reaching a specific answer. In this work, we present CoT pruning studies applied to math reasoning tasks, where evaluating correctness is more straightforward (Kojima et al.. 2022; Lightman al., 2024). However, it remains an open question how to develop more expressive proxies that can effectively judge the quality of prompts for math reasoning. Currently, we rely solely on their prediction
36

---

Evolving Prompts In-Context: An Open-ended, Selfreeiiinnt Perspective
Table 17. Few-shot chain-of-thoughee reasoning performance the math reasoning datasets. We run GGA using 100 samples for fitness estimation, along with 50 samples for early stopping, whereas TAPruning takes 200 samples during search. # Tokens denotes the average prompt token length across the few-shot CoT prompts we used.
GSM8K
MAWPS
LLM
Method
Test Acc
# Tokens
Test Acc
# Tokens
ICL (1-shot)
45.1
100
66.5
177
ICL (4-shot)
41.2
865
76.2
1108
Mistral-It-v0.1
ICL (8-shot)
42.8
2292
76.8
1196
TAPruming (I-shot)
45.8
58
73.8
48
PROMPIQUINE (I-shot)
45.7
56
73.8
90
ICL (1-shot)
84.2
93
87.7
161
ICL (4-shot)
84.2
822
88.6
1034
Qwen2-7B-It
ICL (8-shot)
85.0
2161
92.0
1099
TAPruming (1-shot)
82.8
68
89.1
75
PRoMPiQUINE (I-shot)
84.0
61
91.7
58
ICL (1-shot)
68.0
87
75.6
150
ICL (4-shot)
77.7
729
89.9
937
Llama3-8B
ICL (8-shot)
78.5
1961
89.9
1034
TA TAPruming (1-shot)
77.1
67
85.0
76
PROMPTQUINE (I-shot)
76.4
57
86.2
58
outcome cerromee-seecffccallyy the overall gobblem-solving accuracy a separate dataset. We hope that future research will focus on developing more efficient and expressive proxies for this purpose, especially by focusing on the associations between model hidden states and outputs, as demonstrated by some success in prior work (Sun et al., 2023; Wang et al., 2024), or investigating the use of process supervision derived from a process reward model (Lightman al., 2024), which typically more expensive.
We reuse the evaluation settings, including prompts, models and datasets, explored in section B. For re-ranking, take top-10 10 prompts ranked by the fitness scores, and pick the highest-performmnt prompts based on the validation performance for final results. As shown in Table 17. pruming can be effective in improving the CoT performance. Interestingly, pruning-based results can sometimes achieve comparable performance with traditional more-shot CoT results, while being more efficient its context token use (e.g., vs 1,000). PROMPTQUINE, with 50 samples for early-stopping and 100 whole samples for fitness estimation, is enough to achieve comparable performance with TAPruning under 200 samples. Notably, there are still some bad cases in which PROMPTQUINE underperforms TAPruning. We conjecture that in addition the limited samples used for the fitness evaluation, another important reason be the noisy designs for current fitness function the aggregated accuracy. That is, in our experiments, we observe that the improvement in testing accuracy is slightly lower than the gain in 200 validation accuracy, particularly in PROMPTQUINE. A more robust reward design is left for future work.
D.7. Exploring Diversity-Preserving Mechanisms to Mitigate Premature Convergence
As discussed, the primary challenge in tuning GA designs is the premature convergence, where, at certain generations, most individuals in the population become genetically similar, and most mutations fail to produce individuals with improved fitness. This mainly occurs weaker models, such as GPT-2, or sparser contexts. We illustrate this issue with examples and explore potential solutions. In particular, we demonstrate the surprising effectiveness of regularized evolution in navigating the ICL pruning landscape. Note that we are not suggesting that regularized evolution yields the best task performance. Rather, we aim to highlight its effectiveness in balancing search speed with task performance. It is definitely possible to reach better task results, by leveraging more computations for the prompt exploration, using variety of techniques discussed below. We consider the following baselines, with all other configurations remaining the same what we have done for PROMPTQUINE, in Appendix D.I:
(1) SIMPLE GA: This baseline represents the simplest implementation of traditional generational GA (Syswerda, 1991). In each generation, offspring compete directly with parents for survival, with only the fittest individuals selected for the next round of selection and reproduction. This baseline highlights the severity of premature convergence.
37

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
(2) +REDUCED SELECTIVE PRESSURE (RSP): A simple approach to mitigate premature convergence is reduce selective pressure by replacing tournament selection with pure random selection method. This allows even the least fit individual chance to reproduce, fostering greater genetic diversity, which may improve long-term performance.
(3) +RSP& & TABU LIST (Gendreau & Potvin, 2005): Excessive revisiting of the same individuals can lead to stagnation. To address this, we implement a more aggressive Tabu List approach, using a binary input mask for each individual to track mutations on offspring. Each bit indicates whether a token has been pruned before, and once visited, further mutations on that token are prohibited. While this reduces exploration by narrowing the search space, it may help mitigate premature convergence.
(4) +MMIGRANT-LIKE STRATEGIES (Yang, 2008) (IM): Immigrant methods introduce new individuals to increase genetic diversity, either externally or via random crossover and mutation. We use an internal approach: when stagnation is detected (measured by mean fitness improvement), we dynamically adjust the population size (c.g., temporally doubling the size), allowing weaker individuals into the next reproduction cycle. Combined with sufficient parent selection sampling. this promotes greater diversity.
(5) +RANDOM RESTARTING (RES): We also investigate random restarting, a strategy that refreshes the entire population when fitness stagnation persists specified period (i.e., five generations). Here, we adopt a "partial" restarting approach. Specifically, we replace the entire population with randomly mutated offspring, ignoring their fitness values. We do not adopt full restart by reinitializing the entire population with unpruned ICL. prompts or slight random pruning over the original unpruned ICL prompts, as this is computationally more expensive, which may not be ideal for our target.
(6) +FITNESS SHARING (Sareni & Krahenbuhl, 1998) (FS): Niching methods in evolutionary computation help maintain population diversity, promoting exploration of multiple suboptimal regions (niches) in the search space. Specifically, the key idea is to share fitness among individuals that are close each other in the solution space. This is done through sharing function that penalizes individuals based on their distance from others:
S(x) = d(x,x;) x;)
(4)
f(x)
fistlettive (x)
(5)
Σ S(x;)
Equation 4 denotes the sharing function S, with hyperparameter, the Radius of Sharing (6). controlling the degree of fitness sharing. Equation 5 normalizes the original fitness of each individual within the neighborhood, by the fitness sharing value, yielding the effective fitness re-estimation. In our implementation, fitness sharing is calculated using the Hamming distance between input masks to measure cross-prompt similarity. Given that our mutation rates range from I to 4, we set the Radius of Sharing 5 as 2 in order to encourage diversity within the population.
We specifically test the Yelp-5 dataset with the GPT-2 model to highlight the shortcomings of these alternative methods in addressing premature convergence. As shown in Figure 7, most of the designs we presented above still lags behind our PRomPTQUINE designs, including both SSGA and GGA implementationss in terms of final task results. Additionally, with the exception of RES, most of the designs exhibit minimal pruning progress from 4K to 10K iterations, which is feature of stagnation, as shown in the right subfigure. + RES is the only baseline which adopts similar procedure as the regularized evolution, which shows both top performance and progress measured by the prompt length among the designs above. In contrast, complex designs, such as niching-the the fitness sharing (+ FS), show slow progress in its optimization. We hypothesize that this may be more sensitive to our fitness scales, leading to certain inflexibilities. Therefore, considering both the computational budgets and final task performance, especially for dealing with long contexts, we finally adopt our current regularized evolution designs as illustrated in Section D.1.
E. Towards More Open-Ended Prompt Designs
Our previous search, constrained by limited shots in standard ICL exemplars, still lacks the variety needed to effectively explore unnatural language designs. This section attempts to advance further, providing a preliminary study for the exemplar variations- shot scaling, and appended instructions- ICL, performance.
38

---

Evolving Prompts In-Context: An Open-ended, Selfreiiiing Perspective
10
100
90
5
80
Highest Fitness
Minimal Length
70
-5
60
-10
50
40
-15
30
-20
T2P 5 Thes
20
-25
10
-30 0
2k
4k
6k
8k
10k
2k
4k
6k
8k
10k
Iteration
Iteration
Figure The search dynamics various genetic algorithm designs applied to the Yelp-5 dataset using GPT-2, average over three prompts. The left figure illustrates the improvement in the highest fitness score the course of the search (# iterations, i.c., the number of prompts explored), with higher fitness reflecting more effective search performance. The right figure depicts the progress in pruning, as measured the minimum prompt length explored, throughout the search process. At certain generation, if both figures show further improvement, it may indicate premature convergence, suggesting that further optimization has ceased.
1.0
1.0
PromptQuine Original ICL
0.9
0.9
Accuracy
0.8
Accuracy
0.8
0.7
0.7
0.6
PromptQuine
0.6
Original ICL
0.5
0.5
1
2
4
8
Number of Training Examples
Number Training Examples
Figure 8. Task performance when increasing the shots in the ICL prompts. Left figure shows the results on subjectivity classification (Subj) with Mcta-Llama-33888 3-8--Instrucet, Right figure shows the results on multi-choice question answering dataset (PIQA) with Meta- Llama-3-8B 3-8ma------nstrucc
Pruning Effects on ICL: Scaling Shots. First, as shown Figure 8, similar to traditional ICL (Zhao et al., 2021; Agarwal al., 2024), performance can improve as the number of shots increases, though it may plateau after a certain point. This observation applies to most tasks investigated. Interestingly, clear performance gap exists between raw (unpruned) ICL and our pruned ICL, underscoring the value of pruning in enhancing ICL performance, beyond merely scaling up the number of shots.
Pruning Effects on ICL: Impacts of Instructions. Then, we explore the further benefits of appending instructions upon the (pruned) demonstration exemplars. Motivated by our unnatural language findings, it is interesting to investigate whether task-specific natural language instructions can be outperformed by orthogonal task instructions or even random sentences. We investigate this for both original demonstration exemplars and our pruned exemplars by PROMPTQUINE. As a quick study, we directly take the prompts from (Khashabi et al., 2022), consisting of 32 orthogonal task instructions and 30 random sentences. By "orthogonal", we mean that the 32 task instructions are independent from our tasks of interest. For example, for task like sentiment analysis, natural language instruction for machine translation can be viewed as "orthogonal" task instruction. For task-specific instructions, we ask GPT-4o (Hurst et al., 2024) produce 30 diverse natural language instructions for each task (Subj, AG's News, and SNLI) to form comparisons. We then select the prompts based on the validation performance for each dataset, and report their results on official test set with Meta-Llama-33 3--8---nntrtuee We provide some intriguing examples from Table 18 to Table 23. These examples demonstrate that even LLMs with significant alignment, when provided with carefully constructed few-shot natural language context, random sentences or seemingly
39

---

Evolving Prompts In-Context: An Open-ended, Sclf-replicating Perspective
unrelated task instructions conveying different intentions outperform human-intuitive task instructions. This applies pruned unnatural language contexts, too. This challenges the conventional view of LLLM prompting relation alignment, suggesting that practitioners may need to broaden their perspective and place greater emphasis on computational methods for more effective prompting, Perhaps, prompt engineering is still far from dead.
Table 18: An unpruned ICL prompt example for Subjectivity classification -a----lllmm------sttteuee with different
types of instructions appended.
Source
Generated Prompt
Accuracy
Your task is to classify the comment "subjective" or "objective".
Sentence: never engaging utterly predictable and completely void of anything remotely interesting or suspenseful.
Viewpoint: subjective
Task
Sentence: 'the nugget" is a tale about group of three roadworkers who stumble upon the world's biggest nugget, and become instant millionaires or so they think. Viewpoint: objective
69.3
Instruction
Sentence: {Ippu}}
Viewpoint:
Rankings are as of May 14, 2012
Sentence: never engaging, utterly predictable and completely void of anything remotely interesting or suspenseful.
Viewpoint: subjective
Random Sentence
Sentence: 'the nugget' is tale about group of three roadworkers who stumble upon the world's biggest nugget, and become instant millionaires or so they think.
72.9
Viewpoint: objective
Sentence: (Inpur}
Viewpoint:
Write question about the background paragraph and the story. Sentence: never engaging utterly predictable and completely void of anything remotely interesting or suspenseful.
Viewpoint: subjective
Orthogonal Instruction
Sentence: 'the nugget' is tale about group of three roadworkers who stumble upon the world's biggest nugget, and become instant millionaires they think.
82.3
Viewpoint: objective
Sentence: (Ippur}
Viewpoint:
Table 19: A pruned ICL prompt example by PROMPTQUINE for Subjectivity classification on with different types of instructions appended.
Source
Generated Prompt
Accuracy
40

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Continuation of Table 19
Would you classify this sentence as subjective or objective based on its content?
Sentence: never, completely void of anything remotely interesting orful. Viewpoint: subjective
Task Instruction
80.6
Sentence nug about group three the's nug, instant million thinkView
Sentence:
(Ipur)}
Viewpoint:
Rankings are of May 14, 2012
Sentence: never, completely void of anything remotely interesting orful.
Viewpoint: subjective
Random Sentence
85.5
Sentence nug about group three the's nug, instant million think View
Sentence: (Ippu}
Viewpoint:
Craft one incorrect answer to the question given in input.
Sentence: never, completely void of anything remotely interesting orful. Viewpoint: subjective
Orthogonal Instruction
84.5
Sentence nug about group three the's nug, instant million think View
Sentence: (Inpur)
Viewpoint:
Table 20: An unpruned ICI. prompt example for News Topic classification Meta-I -aaa---lam------nnsttue with different types of instructions appended.
Source
Generated Prompt
Accuracy

---

Evolving Prompts In-Context: An Open-ended, Selfrrelliictgng Perspective
Continuation of Table 20
Categorize the given news piece into World, Sports, Tech, or Business based on its central theme or topic.
Article: Israel suspends soldier after girl shot 15 times GAZA CITY The Isracli army yesterday suspended platoon commander on suspicion he emptied an ammunition clip into 13-year-old Palestinian girl from close range after she had already collapsed under fire.
Answer: World
Article: NBA Star Pippen Announces Retirement National Basketball Association star Scottie Pippen has announced his retirement from the game, leaving the Chicago Bulls team he helped lead six NBA titles.
Answer: Sports
Task
86.9
Instruction
After the Bell-Texas instruments up after sets share buyback Shares of Texas Instruments Inc. (TXN.N: Quote, Profile, Research) rose after the market close on Thursday, after the chip maker said it plans to buy back \1$1 billion in stock Answer: Business
Article:
Article: Oracle IQ Earnings Rise 16 Percent (AP) AP- Business software giant Oracle Corp. said Tuesday that first-quarter earnings 16 percent driven by new database license sales that rose 19 percent.
Answer: Tech
Article: {Inpur}
Answer:
271801, at #1 (Tex. App.-Dallas Jan. 3, 2018, pet. ref'd) (mem. op., not designated for
Israel suspends soldier after girl shot 15 times GAZA CITY The Israeli army yesterday suspended a platoon commander on suspicion he emptied an ammunition clip into 13-year-old Palestinian girl from close range after she had already collapsed under fire.
Article:
Answer: World
Article: NBA Star Pippen Announces Retirement National Basketball Association star Scottie Pippen has announced his retirement from the game, leaving the Chicago Bulls team he helped lead to six NBA titles.
Answer: Sports
Random Sentence
87.5
Article: After the Bell-Texas instruments up after sets share buyback Shares of Texas Instruments Inc. (TXN.N: Quote, Profile, Research) rose after the market close on Thursday, after the chip maker said it plans to buy back \)$1 billion in stock
Answer: Business
Article: Oracle IQ Earnings Rise 16 Percent (AP) AP- Business software giant Oracle Corp. said Tuesday that first-quarter earnings rose 16 percent driven by new database license sales that rose 19 percent.
Answer: Tech
Article: {Input} Answer:
42

---

Evolving Prompts In-Context: An Open-ended, Sclf-replicating Perspective
Continuation of Table 20
What the type of the answer corresponding to the given question? Number, Date, or Span?
Article: Isracl suspends soldier after girl shot 15 times GAZA CITY The Isracli army yesterday suspended platoon commander on suspicion he emptied an ammunition clip into 13-year-old Palestinian girl from close range after she had already collapsed under fire.
Answer: World
Article: NBA Star Pippen Announces Retirement National Basketball Association star Scottie Pippen has announced his retirement from the game, leaving the Chicago Bulls team he helped lead six NBA titles.
Answer: Sports
Orthogonal Instruction
85.3
Article: After the Bell-Texas instruments up after sets share buyback Shares of Texas Instruments Inc. (TXN.N: Quote, Profile, Research) rose after the market close on Thursday, after the chip maker said it plans to buy back \1$1 billion in stock Answer: Business
Article: Oracle IQ Earnings Rise 16 Percent (AP) AP- Business software giant Oracle Corp. said Tuesday that first-quarter earnings 16 percent driven by new database license sales that rose 19 percent.
Answer: Tech
Article: {Inpur] Answer:
Table 21: A pruned ICL prompt example by PROMPTQUINE for News Topic classification on Meta-Llama-33 ---mm-----B--- 3-8B--88------ I--ttruttt,,,, with different types of instructions appended.
Source
Generated Prompt
Accuracy
Determine the the topic of the item and then choose from World, Sports, Business and Tech.
Chief Scores Coup in Court CALLAO BASE Peru (Reuters) rebel, Peru's founder Ab scored propaganda his retr be postponed.
: win out on The Dolphins finally gave their reason to celebrate polished performance with their victory season- St. Answer: Sports
Task Instruction
Article:
Echo Posts Addsscribers (Reuters -Star Corp. said third rose on an aggressivecampaign addAnswer: Business
85.1
Yahoo personal search a service designed to let users their sharing with others, the company.
Tech
Article: {Inpur}
Answer:
43

---

Evolving Prompts In-Context: An Open-ended, Sclf-replicating Perspective
Continuation of Table 21
<span |i
A. Maj
Institute Theoretical Physics and Astrophysics) Gda[ñ]{ } University
Wita Stwosza xa057)
80-952 ssk, Poland<</pan>>)
*E-mail address:" 'fizwam univ.gda.pl")
Chief Scores Coup in Court CALLAO BASE Peru (Reuters) rebel, Peru's founder Ab scored propaganda on his retr be postponed.
: win out on The Dolphins finally gave their reason to celebrate polished performance with their victory season- St.
Random Sentence
88.3
Answer: Sports
Article:
Echo Posts Addsscribers (Reuters -Star Corp. said third rose on an aggressivecampaign to addAnswer: Business
Yahoo personal search a service designed to let users their sharing with others, the company.
Tech
Article:
{Input}
Answer:
Write question about the background paragraph and the story. Chief Scores Coup in Court CALLAO BASE Peru (Reuters) rebel, Peru's founder Ab scored propaganda his retr be postponed.
:
:
win out on The Dolphins finally gave their reason to celebrate polished performance with their victory season- St. Answer: Sports
Orthogonal Instruction
Article: Echo Posts Addsscribers (Reuters -Star Corp. said third rose on an aggressivecampaign addAnswer: Business
88.0
Yahoo personal search service designed to let users their sharing with others, the company.
Tech
Article:
(Input}
Answer:
Table 22: An unpruned ICL prompt example for Natural Language Inference (SNLI) on Dotta-Llamaa3-8--Innstte with different types of instructions appended.
Source
Generated Prompt
Accuracy
44

---

Evolving Prompts In-Context: An Open-ended, Sclf-replicating Perspective
Continuation of Table 22
Given the premise and hypothesis, determine if they are related with 'yes' (entailment), 'no' (contradiction), or 'unknown' (neutral).
Hypothesis: The rock band in the dark theatre.
Premise: String orchestra and conductor spotlight surrounded by darkness. Given the premise, is the hypothesis true? Yes, No Unknown? The answer is: No
Hypothesis: The man is Amish he drives a wagon through an intersection and doesn't care.
Premise: A man driving a horse-drawn wagon on busy intersection.
Given the premise, is the hypothesis true? Yes, No or Unknown? The answer is: Unknown
Task
61.5
Instruction
Hypothesis: The old man in the hat was reading. Premise:
The elderly, overweight man is wearing a hat, moccasins, and purple shirt while reading a book on a sidewalk in front tree. Given the premise, is the hypothesis true? Yes, No or Unknown?
The answer is: Yes
Hypothesis: {Hypothesis)
Premise: (Premise)
Given the premise, is the hypothesis true? Yes, No or Unknown? The answer is:
"I know that, but what department placed the order? Was the FSB?" Hypothesis: The rock band in the dark theatre. Premise: String orchestra and conductor in spotlight surrounded by darkness.
Given the premise, is the hypothesis true? Yes, No or Unknown? The answer is: No
Hypothesis: The man is Amish he drives a wagon through an intersection and doesn't care.
Premise: A man is driving horse-drawn wagon busy intersection.
Given the premise, is the hypothesis true? Yes, No or Unknown? The answer is: Unknown
Random Sentence
60.9
The old man in the hat was reading. The elderly, overweight man is wearing a hat, moccasins, and a purple shirt while reading book on sidewalk in front of tree. Given the premise, is the hypothesis true? Yes, No or Unknown?
Hypothesis:
Premise:
The answer is: Yes
Hypothesis: {Hypothesis}
Premise: (Premise)
Given the premise, the hypothesis true? Yes, No or Unknown?
The answer is:
45

---

Evolving Prompts In-Context: An Open-ended, Selfreelliiangg Perspective
Continuation of Table 22
Generate a paraphrase of the given sentence in the input.
Hypothesis: The rock band in the dark theatre.
Premise: String orchestra and conductor spotlight surrounded by darkness. Given the premise, is the hypothesis true? Yes, No or Unknown?
The answer is: No
Hypothesis: The man is Amish he drives a wagon through an intersection and doesn't care.
Premise: A man driving a horse-drawn wagon on busy intersection.
Given the premise, is the hypothesis true? Yes, No or Unknown? The answer is: Unknown
Orthogonal Instruction
62.3
Hypothesis: The old man in the hat reading. Premise:
The elderly, overweight man is wearing hat, moccasins, and purple shirt while reading a book on a sidewalk in front of tree. Given the premise, is the hypothesis true? Yes, No or Unknown?
The answer is: Yes
Hypothesis: {Hypothesis}
Premise: (Premise)
Given the premise, is the hypothesis true? Yes, No or Unknown? The answer is:
Table 23: pruned ICL prompt example by PROMPTQUINE for Natural Language Inference (SNLI) Meta-Llama-3-8B Instruct, with different types of instructions appended.
Source
Generated Prompt
Accuracy
Does the hypothesis align with the premise as yes, oppose it as no, or is it unknown?
rock dark.
Premise: orchestra in by.
the premise the true? Yes, or Unknown?
is
Hypo:
The Amish through an intersection and doesn.
Task
Premise: a horse-drawn.
68.6
Instruction
Given the premise is the true Yes No UnknownThe answer is: Unknown
Hypo The old man in the hat was reading. Premise: The elderly overweight man is ains and shirt while in front of tree.
Given premise is? Yes The answer Yes
Hypo: {Hypothesis}
Premise {P (Premise)
the, the hypothesis true Yes No orThe answer is:
46

---

Evolving Prompts In-Context: An Open-ended, Selfreeliciin Perspective
Continuation of Table 23
"So we might see you around?" she asked. The chill in her voice almost made him shudder.
: rock dark.
Premise: orchestra in by. the premise the true? Yes, Unknown?
is
Hypo: The Amish through an intersection and doesn.
Random Sentence
Premise: a horse-drawn.
72.2
Given the premise is the true Yes No UnknownThe answer is: Unknown
Hypo The old man in the hat was reading. The elderly overweight man is ains and shirt while in front of tree.
Premise:
Given premise is? Yes orThe answer Yes
Hypo: (Hypothesis) Premise (Premise}
the, the hypothesis true Yes No orThe answer
Write question about the background paragraph and the story. rock dark.
Premise: orchestra in by. the premise the true? Yes, or Unknown?
is
Hypo: The Amish through intersection and doesn.
Premise: horse-drawn.
Orthogonal Instruction
Given the premise is the true Yes No UnknownThe answer is: Unknown
74.4
Hypo The old man in the hat was reading.
Premise: The elderly overweight man ains and shirt while on in front of tree.
Given premise is? Yes The answer Yes
Hypo: {Hypothesis}
Premise (Premise}
the, the hypothesis true Yes No The answer is:
47

---

Evolving Prompts In-Context: An Open-ended, Sclf-replicating Perspective
Task Performance (GPT-2)
1.0
ICL (Original) Label)
PromptQuine (Original)
PromptQuine (Original)
0.9
PromptQuine Label)
0.9
PromptQuine Output)
0.8
0.8
0.8
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.5
0.5
0.5
0.3
0.3
0.3
0.2
0.2
0.2
SST-2
SNLI
5ST-2
Subj Dataset
SNLI
SST-2
Subj
SNLI
Dataset
Dataset
Figure Changes in (unpruned & pruned) prompting performance on GPT-2 when labels are removed (left & middle) or even the complete outputs are removed (right).
Table 24: Examples of initializing ICL prompts (specifically for SNLI) with different ICL templates which leads to significant performance variations. Please refer to Section 5.1 for the details.
Template
Accuracy
Premise: (Premise)
Hypothesis: {Hypothesis}
Based on the premise, can we conclude the hypothesis? Answer: Yes, No, Unknown. 60.8
The answer is:
Statement: (Hypothesis)
Evidence: {Premise}
75.1
Can the hypothesis be validated based on the given premise? (Answer with Yes, No, or Unknown)
The answer is:
Table 25, Task performance of 1-shot ICL with random verbalizers. Unpraned ICL denotes the unpruned 1-shot ICL (verbalizers replaced)'s task accuracy. We average the results across four different ICL prompts, varying the verbalizers. These verbalizers are created by GPT-4o (Hurst et al., 2024) for arbitrary words. We present examples in Appendix Table 26.
Unpruned
Task Accuracy (%)
Model
Dataset
ICL
Avg
Min
Max
SST-2
50.2 (0.4)
50.0 (0.1)
49.9
50.2
GPT-2
Subj
50.0 (0.0)
56.4 (744
50.0
64.2
SNLI
32.8 (2.3)
36.2 (4.8
31.9
43.1
SST-2
53.9 (10.2)
58.3 (16.8)
49,9
83.4
Llama3-8B-It
Subj
50.0 (0.0)
68.5 (14.5)
50.0
82.3
SNLI
33.4 (0.9)
41.7 (14.5)
32.9
63.2
SST-2
58.0 (15.6)
69. (22.4)
49.9
90.2
Llama3-70B-It
Subj
50.0 (0.0)
75.4 (12.7)
62.3
91.3
SNLI
31.2 (5.8)
47.6 (15.9)
32.9
48

---

Evolving Prompts In-Context: An Open-ended, Sclf-replicating Perspective
Table 26: Some effective task prompt examples discovered by our PROMPTQUINE for Lat.--laam------nstttteee using random verbalizers.
Dataset
Generated Prompt
Verbalizers
Accuracy
nasty.
Sentiment it bweet andical elements.
Sentiment:
SST-2
Necklace, 2
90.2
Review: {Inpu]}
Sentiment:
never engaging, anything suspenseful.
Viewpoint:
Subj
Butterfly
79.2
Sentence 'theget' tale about'sget become million View: 5Sentence: (Inpur)
Viewpoint
on motorcycle jumping the. ise dirt bike racer jumping racer far. premise, is hypothesis? Cascade or Telescope? is Cascade
Hthesis: A is flying kitePremise: The man slides the sand while holding hang glider.
Given premise the hypothesis true? Cascade, Moon or Telescope? answer is: Moon
SNLI
Cascade, Moon, Telescope
65.1
ypo the concert. Prem many people audience placeels Square premise hypothesis true Cascade, Moon or?
answer Telescope
Hypo {Hypothesis) Prem (Premise) premise is the true? Cascade, Moon or?
answer is:
Table 27: Some effective task prompt examples discovered by PROMPTQUINE for Sita--llaaa-----Insttie
Dataset
Generated Prompt
mostly tiredread of mob talesSentiment: terrible
Review presenting romance in, which us to what's possible and do to make it. Sentiment great
SST-2
Review: (Input) Sentiment
Continuation of Table 27

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Continuation of Table 27
Sentence: for with the for its heartbeatening intensity and body-slam roller delivers it can really get behind. View: subjective
Subj
Sentence long the bay boy is the sedate and past upsideical.
ViewpointSentence: {Impur}
View:
ypothesis cold outside.
Premise withers down snow covered hillGiven the premise, is Yes No UnknownThe is: Yes
Hypo: The person is on the ground.
Premise: sneakers is airboreGivenn, true Yes, or UnknownThe answer: No
SNLI
ball.
Premise child green shirt fish type hat near water blue and.
premise true? Yes No UnknownThe is Unknown
Hypo [Hypothesis] Premise: (Premise)
the? Yes, No or is:
Article Car in Baghdad bombsll yesterday American diplomats desert.
low score Norman will leaders the Championship Co:satt Resort, Brisbane. Answer: Sports
Article Prices; a - Oil:49 in thatsupply northernAnswer: Business
AG's News
Article Hollywood industry cases. Answer: Tech
Article {Input)
Answer:
Sentence: auto race Wilbur the to win three? bad
Sentence: CNN is the abbreviation for? Sentiment: neutral
Sentence Doones character to? Sent
Yelp-5
Sentence Where did get his metaliment: great
Sentence: What causes canker sores? Sentiment terrible
Sentence: (Input)
Sentiment:
Continuation of Table 27
50

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Continuation of Table 27
7... she good at?
Topic music
Sentence Does any think it's to love of the opposite sex at the timeTopic:
Sentence: Does it upset that there people out there that want prove Jesus walk waterTopic: culture
Sentence want to find an e-mail of a a to years hes nameiotoulis.?
Topic: business: Why call hispanic Mexicans?
Topic:Sentence What British the largest ship Ww- Navy? Who sank?
Topic: education picture from space is the look atTopic science
Yahoo
Sentence: can find the information about be jump? Topic: sports
used to beta, recently it when had to reinstall windows xp how do get Topic: computer
am loser?
Topic
Sentence: {Ipput}
Topic
Here is a which is negative: "this place awful!". Here rewrite of text positive: "this place is amazing!".
Yelp Positive Transfer
Here is, which is negative: "initely buy another pair of socks thisthey have worst sock Here is a of the text, which is: "initely will buy another pair socks from store-they have the".
is a text, which negative: " "[Impur}". Here of the, which is positive:"
whichvery.". rewrite the text negative: "very works terribly. isi the, really.
Yelp Negative Transfer
"[Inpur}". the negative:"
:Options) into hold B
A) into- diameter diameter.
Answer
PIQA
Question: (Inpur)
Options: (Option!) B) {Option2)
Answer:
If there3 cars in parking and2 more cars, many cars in the lot?
Let's step by step
There originally cars.
GSMSK
2 cars arrive.
3 +2 =5The answer is 5Question: (Input)
's think step
Continuation of Table 27
51

---

Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Continuation of Table 27
Question: house is % the price.Let think step original up the information. (x,500% of Therefore18511 si0esfrac5500) (0)236 pricel.
MAWPS
Question: (Impur)
Let's step
52


Reviews:
[{"review_id": "f5zQ7DmHUl", "reviewer": "Reviewer_KfU1", "summary": "This paper introduces an evolutionary method called PromptQuine for optimizing few-shot prompts by pruning them. They show that their optimized prompts outperform the original few-shot prompts as well as the RLPrompt baseline on a held-out set of test examples across a wide range of standard language model benchmarks. Interestingly, the resulting pruned prompts are often gibberish text (similar to what was found in other prior work) but retain some key features, including reference to the task at hand.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer KfU1 for **extremely detailed review** on many tech details. We are glad that the reviewer appreciates the *contributions of our insights towards prompting and interpretability*. We address your questions below:\n\nAnonymous link (AL) for tables/data: https://anonymous.4open.science/r/ughj/KfU1/\n>Evaluation Setup\n\nWe agree to restructure paper for clear setups. We'll introduce valid/test separations and what typical objective is when we introduce our algos, and highlight that all follow this. We'll also highlight in section2.2 formalism *Solution Selection*, and make each Exp. clear.  \n>PromptQuine results & Pilot Study\n\n* **Results**: We fully agree for transparency. The current format is constrained by page limits. We'll improve. Re RLPrompt vs ours, Quine is imperfect in its representation power. As Appendix E claims, future work can combine both for improvements. We'll highlight in limitations by KRwQ. Additionally, RLPrompt in paper is built upon different templates and token positions (line1115vs1172). *Prefix-only* appending forms a *fairer* comparison: Acc on template1&2: Quine(83.1/79.5), RL(77.5/76.3). The gap bridged, suggesting nuanced sensitivity towards token compositions.\n\n* **Qualitative analysis**: Inspired by the numbers swapping ranks of methods, we'll add a section for ICL stabilization (ICLS). While pruning can be viewed as alternative for ICLS, *template selection [1] can still largely affect results* (Why/when not work effectively). Altogether, along with ablations, this provides a broad view of the pruning formulation.\n\n>Jailbreaking\n\nWe clarify we didn't manipulate metrics. Varied setups are limitations in many jailbreaking papers, e.g., own held-out sets for valid/test. We follow [2] by prompting Llama-Guard3 (same prompt,line 405) as classifier on AdvBench [2]. The use of these two is indeed popular [3,4,5+], validating our choices. To increase confidence, we use standard Exact Match score [3] to do the eval: 90+ASR. Please check AL (README) for more numbers, which we reply to e5jB. We also provide **vicuna's direct prediction** and mistral-instruct-v0.3 results in the AL for checks. Inspired by e5jB, we'll also expand jailbreaking, and present a clearer picture of success/failures for that task (see our replies to 8j6A, thanks).\n>Label Word Study\n\nWe agree this introduces variances, also mentioned in line 415. We'll expand the texts to make clear. That is also sth always debated by existing work on ICL mechanistics (cf. [6,7] vs [8]). We attempt to present some insights following [8], e.g., intriguing novel finding: pruning has potentials *even with random label words*, which can inspire future work. We'll expand further following ur suggestions on implications.\n>GPQA,MMLU\n\nAs MMLU involves many tasks, we show GPQA-main (llama3-it): 33.7 to 35.3 (split done for search&eval). The results are indeed empirical. More research can be done for filling the theoretical gap.\n>ES intro & ablation\n\nWe'll follow ur textbook tip. For ablation, many designs are indeed helpful, e.g., stagnation in App. D.7. We'll follow section5 in [9], studying reranking-counter deception/overfit (see AL), mutation rate, offspring size-reward lower-quality stepping stones [10], etc. We'll put significant emphasis on the search dynamics discussion in the paper. The studies can help future research.\n>Promptbreeder(PB)&APE\n\nWe excluded PB due to the lack of released code, making faithful reproduction hard. Here, we reimplemented using GPT-4o mutators(1-shot, 3 seeds), each taking 20min for GPT-2 and 30+min for LLaMA3-it(l3it) w/ OpenAI API. Ours outperforms on GPT-2, ~1–7 min and short initializations, but runtime will increase with longer contexts. Given the impact of implementation on search algos [11], we’ll instead claim *decent runtime efficiency* in paper for rigorousness. We chose not to report APE for its constrained exploration vs. PB...\n|PB-Quine|Subj|News|Yahoo|\n|-|-|-|-|\n|l3it|-2.2|-0.6|-1.4|\n|l3it-4shot|-9.5|-1.3|-0.8|\n|GPT2|-5.6|-8.9|-21.5|\n\nPB is strong but limited, e.g., worse in weaker models. UnnaturalLang still good across models-unaddressed.\n>Writing Questions\n\nThanks for the tips. We'll follow and add implications/limitations/future work. Re interpretability, our findings inspire directions for ICL, e.g., why random labels can improve (start from chance, w/o scaling up shots&sizes [12]) and a rethink on label words in-context. We'll make clear. Re hiding intention, extensive RLHF could address, yet imperfect and we think answering why pruning works can contribute.\n\n[1] start worrying about prompt formatting [2] Understanding jailbreak success [3] GCG [4] AdvPrompter [5] ArtPrompt [6] Ground-Truth Labels Matter [7] ICL Learns Label Relationships yet not.. [8] Rethinking the Role of Demonstrations [9] Large-scale evolution of Image Classifiers [10] Why Greatness cannot be planned [11] Larger LMs do ICL differently [12] Implementation Matters in DPG", "claims_and_evidence": "The evaluation setup appears to be correct, although it could be better explained. In particular, it is imperative that the examples on which the prompts are pruned (training / validation splits) are distinct from the examples on which they are evaluated (test split). This is alluded to in Evaluation Settings (line 182) but it is not sufficiently clearly stated. It would be very useful if the authors can separate out an Evaluation section which pertains to the whole paper and which clearly states the nature of the held-out evaluation, include a specific step-by-step example, so that the reader can ensure the rigour of the evaluation procedure. \n\nThe pilot study with hill-climbing search is well-explained and the claims are substantiated by the evidence in this section. \n\nThe results for PromptQuine itself are more questionable. The authors choose to only present the positive results in the main text, with the negative results confined to the Appendix (D5 and D6) without further discussion in the main text. In my opinion the authors should be more transparent about the fact that PromptQuine underperforms RLPrompt in various tasks in the main text, and provide some qualitative insights / analyses as to why that might be case. \n\nSimilarly I find Section 5.2 a little more difficult to justify as \"convincing evidence\". The authors introduce a new metric with which to measure performance in jailbreaking, and it is hard to be certain that this metric has not been chosen so as to demonstrate the strength of their method. I would recommend that they use existing metrics / benchmarks in the literature to the extent possible. \n\nThe \"task label\" analysis appears convincing and gives good insight into which parts of the pruned prompts are important. However it is not sufficiently well explained. How are \"task label\" words identified? Also in Figure 4, the variance appears quite high and this deserves commentary. How much can we conclude from such high-variance experiments? \n\nThe examples in Table 16 are useful, and it would be good to see more of these analysed in the main text, for both situations where PromptQuine works and situations where it does not.", "methods_and_evaluation_criteria": "The choice of benchmark datasets are sensible, although it would be interesting to see results on some more modern tasks too e.g. GPQA and MMLU. \n\nThe methods used are well-motivated. However the explanation of the evolutionary algorithm is long-winded, and too much is made of what is essentially a very standard method. It would suffice to describe the algorithm as a standard evolutionary algorithm (citing a textbook in the literature) and then specify the mutation and selection operators, along with the definition of what constitutes a member of the population. \n\nThe evolutionary algorithms that are used are rather complicated, incorporating regularization, additional re-ranking, hyper-mutation of hyperparameters etc. It may be that each of these components is necessary and motivated, but it would be really useful to have an ablation study of the components back down to a simple genetic algorithm in at least one setting, to understand how much each component adds (and in which combinations the components must be present).", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "I have already commented on this above.", "relation_to_broader_scientific_literature": "The related work should not be relegated completely to the Appendix, in my opinion. There should at least be a paragraph in the main text giving details of the most relevant prior methods and the ways in which this contribution differs.\n\nTwo particular papers that are useful points of comparison are PromptBreeder and Automated Prompt Engineer. Can the authors comment on the reasons why these were not discussed in more detail, and why their results are not used as baselines for PromptQuine? It is hard for me to assess whether their method is stronger / more efficient than APE and PromptBreeder without a numerical comparison. (Note: both papers are cited, but that is not necessarily enough - they seem to be to be directly comparable methods, so it would be useful to compare the performance precisely, in so far as is possible, or to argue why not).", "essential_references_not_discussed": "See response to previous question.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "I suggest that the authors restructure the paper by moving the Pilot Study to the Appendix, opening up more space for talking about both the positive and negative results of PromptQuine in the main text, for providing ablations on the components of the methods, for improving the description of the evaluation, and for providing more specific qualitative examples of the pruned prompts that work and that don't work well. \n\nPlease can the authors comment on the societal implications of their work? In particular, it would be useful to have their thoughts on the implications of this work for interpretability. If LLMs can be prompted for particular behaviors in ways that are not legible by humans, could this be used to hide bad intent? What mitigations might we think about for such problems?", "questions_for_authors": "Questions are present in my comments above. Please address these and give your thoughts / responses to the specific weaknesses and suggestions I have provided above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0LaWRQTOiF", "reviewer": "Reviewer_KRwQ", "summary": "The paper applies evolutionary algorithms to the paradigm of LLM prompt pruning, introducing a new algorithm called PromptQuine. PromptQuine autonomously searches for better pruning strategies through an iterative process of mutation and selection, inspired by biological self-replication and evolutionary dynamics. It demonstrates effectiveness across diverse tasks, consistently outperforming existing prompt optimization methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our sincere thanks to Reviewer KRwQ for their detailed tips to improve our writing. **This is invaluable, and we do learn a lot.** We are also grateful for viewing our work as **significant contribution** by presenting *interesting results* to the community. Here is the Table/Fig link: https://anonymous.4open.science/r/ughj/KRwQ/README.md\n>Vague Terminologies\n\n* **SecretLang & Gibberish**: Apologize for not explicitly defining \"secret language\" and now clarify it in a new paragraph (Section 2.2) following its intro in Para. 1.*Specifically, secret language refers to unnatural language whose syntax and semantics are incoherent and difficult for humans to parse, yet can be surprisingly effective in certain scenarios. In the absence of ..., such prompts are typically regarded as mysterious, hidden, and inherently non-scalable.* Further, the secret language we use in paper refers to prompts generated by prior methods, e.g., RLPrompt, which limits by algorithmic capacity. For the use of gibberish texts, we discuss in intro (line 30) as *syntactically and semantically strange*, which are used less. Hope this won't cause overclaims.\n* **Partial Context Hypothesis**: We agree. So we plan to use formal mathematical languages which avoid use of \"redundant\" and \"well-specified\". Intuition: Given a natural prompt (e.g., ICL) x={...} with performance X and a unnatural prompt z (by existing SOTA, e.g. RLPrompt) with Z, is it possible to prune a few tokens {...} which leads to prompt y with enhanced performance Y, which potentially outperforms Z. y can be purely syntactically and semantically unnatural language.\n* **Conventional Wisdom**: will cite OpenAI guide and [1,2+] for ICL stabilization (i.e., well-tuned). As also inspired by KfU1, we'll add a ICL study highlighting some structured insights. Sorry for the pointer, due to word limits.\n\n>PromptBreeder+\n\nWe cited PB but not explicitly discussed. We'll discuss in related work for any work related to Open-endedness & LLMs. Specifically, we'll explain how ours differs from PB. Re comparisons, we thought before. Unfortunately, they didn't release code for use. As ES involves varied designs, we are unsure if reproduction fully reflects PB's result. In replying to KfU1, we reproduce with numerical results. We may report in paper. For larger space, we generally agree, but want to emphasize: While PB introduces token variations, the search space in our pruning context is also large, typically O(n!). Considering the discrete nature, even a slight token drop can cause large changes, which can be different from ES in continuous space. So I feel that both space by current discrete algo is large, and both **framework** is scalable yet constrained by computations. \n>Search Space\n\nRegarding pruning strategies, we admit that bit-flip can be limited in its representation power. We will introduce an ablation study (by KfU1) for mutation rates, e.g., why up to 4 for **solution sparsity**. That is also why we retain word orders, as we observed that many lead to meaningless mutations under selection pressures (+a study). Regarding open-ended exploration of prompt space, we agree and argue for not fully aligned w/ our target (rethink prompt space). The most relevant term is the *open-ended search* (established OES) we highlight in abstract. That is an insight we aim to deliver, e.g., intriguing yet unnatural prompts remain hidden, lurking just within our reach. We suggest the field to pay more attention to OES (cf. [3,4,5,6]), jumping out of linguistics for diverse, novel **unnatural language**.\n>Optimization challenges\n\nWe argue that Section 4.2 is meant to motivate ES, and the landscape analysis is also new as the insight of pruning towards such large improvement is also new. Also, to answer e5jB, we'll discuss the deception of held-out objective [6,7], which **necessitate reward suboptimal stepping stones** like ES in paper.\n>Other Writing Questions\n\nFig1 is inspired by [8] (Fig4), and we will add relative success rate (RS over/divided by ES) descriptions in caption and polish terminology in section 4.2. Regarding typos/LaTex/future work, we will follow the tips, correct all and state clearly. Regarding low-shot regimes, we borrow this in few-shot learning [9,10], in sense that we search using only low-shot samples, e.g., fitness measure. We can avoid such jargon. Further, we apologize for the ambiguity around noise; We use it to refer to deceptive/imperfect reward (not uncertainty/noise in DG MAP-Elites, thanks!), i.e., stronger prompts are rated as lower ranks, which we address using held-out score (rerank), thought still being limited. We'll make it clear. Fig3:https://anonymous.4open.science/r/ughj/KRwQ/re_Fig3.pdf\n\n[1] Fantastically Ordered Prompts [2] Learning To Retrieve for ICL [3] Minimal criterion coevolution [4] Randomly Wired NN [5] Weight Agnostic NN [6] Abandon Objectives [7] Go-Explore [8] AutoML-Zero [9] Pi-Tuning [10] Loss landscape is all your need", "claims_and_evidence": "Most claims are supported with clear and convincing evidence.\n\nFigure 1 (right), “ES exhibit greater robustness to task difficulty, such as increasing the number of shots, which amplifies solution sparsity.” (grammatical mistake here, should be ES exhibits) and line 237, “the success ratio of RS approaches zero as task difficulty increases”. In Figure 1 (right), I only see one line plotted, and am unsure whether that line is ES or RS. How are the claims derived from Figure 1 (right)?\n\nThe paper claims that PromptQuine automatically optimizes the pruning strategies. However, the mutation that pruning strategies undergo is very limited, i.e., having mutations in the pruning tokens by randomly flipping the bits. This means that the search space may not fully explore more complex pruning strategies. As a result, while PromptQuine demonstrates strong empirical performance, its optimization approach might be constrained to local improvements rather than a truly open-ended exploration of prompt space.\n\nThere is also the claim that incoherent “gibberish” or “secret language” can be more effective prompts than well-crafted ones. However, the usage of these terms is vague and imprecise, lacking a clear definition or formal characterization. Furthermore, since the mutation strategy and search space in PromptQuine is so limited, i.e., word order is still kept, “gibberish” and “secret language” seems like an exaggeration of the prompts produced.", "methods_and_evaluation_criteria": "Yes. The authors compare PromptQuine against competitive baselines across a diverse set of tasks.", "theoretical_claims": "The definition and usage of “Partial Context Hypothesis” are vague. Terms such as “potentially redundant contexts” and “well-specified natural language prompts” are not carefully defined, making it unclear what specific aspects of a prompt contribute to redundancy or well-specification. Despite this lack of clarity, the hypothesis is referenced multiple times throughout the experimental analysis. A more precise definition and formal criteria for evaluating context redundancy and prompt specificity would strengthen the validity of the hypothesis.", "experimental_designs_or_analyses": "All experiments are sound with repeated runs and variance.", "relation_to_broader_scientific_literature": "The paper builds on prior work in prompt optimization, evolutionary search, and in-context learning. The paper proposes PromptQuine, an evolutionary algorithm for prompt pruning. It extends research on automatic prompt optimization (e.g., RLPrompt, LLMLingua) by using mutation and selection rather than RL or token attribution. The study also connects to evolutionary search in NLP and aligns with findings on LLM sensitivity to prompt design.", "essential_references_not_discussed": "Since the key contribution is the application of evolutionary algorithms to prompt design, some recent related works are missing from the citations. Specifically, “Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution” and “Evolution through Large Models”. Promptbreeder could be a good approach to compare PromptQuine against too. Furthermore, since Promptbreeder works in a larger search space than PromptQuine, I think this work should definitely be discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n- I think this paper is a significant contribution and shows interesting results, that just by pruning prompts, PromptQuine is able to achieve better results across a diverse set of benchmarks.\n\nWeaknesses:\n- See above. The overall writing is good, but some of the claims and terminologies used are very vague and ambiguous.", "other_comments_or_suggestions": "For Figure 3, it is difficult to see the differences in performance between the models. Truncating the starting percentage of the plots (e.g., displaying only values from 75% onwards) would improve clarity and make the differences more noticeable.\n\nIn line 202, “we follow (Krishna et al, 2020; …)”. The citations should not be in parentheses.\n\nSimilar to the usage of “gibberish” and “secret language”, the term “conventional wisdom” is also used vaguely. While Wan et al. (2024) is cited as a reference, it is unclear to what extent this truly represents a conventional understanding scientifically. A more precise explanation or broader citation of prior work would help establish its validity.", "questions_for_authors": "1. In the abstract, line 31, what does it mean by “low-shot regimes”?\n2. Optimization challenges in sparse, multimodal landscapes is not a new challenge in evolutionary algorithms. Are there related works that show the same challenges?\n3. To mitigate evaluation noise, re-ranking mechanisms are used. There are related evolutionary works that deal with noise and stochasticity, e.g., Fast and stable MAP-Elites in noisy domains using deep grids.\n4. While the paper briefly touches on its limitations and potential future work throughout, it would be more useful to include a dedicated section in the conclusion to discuss these aspects in greater detail.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VYHPwvTkAq", "reviewer": "Reviewer_8j6A", "summary": "The paper introduces a novel prompt design paradigm that challenges the conventional approach of using well-crafted natural language prompts for large language models. The authors demonstrate that pruning random demonstrations into seemingly incoherent \"gibberish\" can significantly improve performance across a variety of tasks, including classification, multi-choice question answering, generation, and reasoning.\n\nThis effect is shown to generalize across different LLMs, regardless of their alignment. The authors propose a self-discover prompt optimization framework called PromptQuine, which employs evolutionary search to automatically identify effective pruning strategies.\n\nThe paper provides extensive empirical evidence supporting these findings and discusses their implications for understanding in-context learning and prompt design in LLMs.\n\n## update after rebuttal\n\nI thank the authors for the rebuttal. I think this is an interesting paper, but I am keeping my score because the limited mechanistic analysis (Section 6) leaves the \"why\" of pruning’s success underexplored.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are *very grateful* to Reviewer 8j6A for their highly positive comments on our paper. Thanks for acknowledging our efforts on making this paper *empirically rigor* and the **Significance, Creativity and Originality** of this work towards general *prompt tuning and ICL*. We will **make every effort to further improve** the paper, especially the presentations. We address your questions below: \n>Analysis depth\n\nThanks for the interest. \n* **Organize potential research questions into Future work/ Implication**: We also share this with nearly all reviewers. As our work is dense, we plan to leave many of the questions as future work. Indeed, various interpretability questions can arise from our findings, and many of them remain challenging open problems, and we believe each deserves a separate research paper, especially why pruning is effective and why such ICL behavior could work. We hold strong belief addressing them can largely advance the field.\n* **Incorporate additional analysis studies**: Inspired by KfU1, we'll add studies showing the failures of ICL pruning, e.g.. while effective for ICL stabilization, the template selection [1] still matters. Further, as some interests may center around jailbreaking, we'll expand studies on that, e.g., extending studies on varied ICL setups, e.g., in-context attack [2], prompt injection attack [3] which go beyond current priming setup. We plan to analyze this challenging task, where the most common metric provides a sparse reward signal and many other formulation exists (e.g., tools for mechanistic interpretability--steering vectors and output joint probability as gradient methods).\n\n>Computational Resources\n\nAfter further check, we find that we leave many descriptions in text, instead of making explicitly clear in Table 10/12, e.g., LLM with Llama3-it, GPU type, etc. We'll refine the captions. To give a brief overview, for search, in classification/QA tasks, on a single A100 GPU, TAPruning/PromptQuine can run within an hour for low-shot ICL. As parallelization can be an option [4] for PromptQuine, runtime efficiency can improve with more computes. In generation, including reasoning, they take hours as shown in Table 12. TAPruning, which based on first-choice hill climbing, can always be a good start point for particular tasks of interest, as they converge quicker in one GPU. Dealing with long contexts typically take more time, e.g., hours or even days, yet may increase final result. That could be one tradeoff. *One potential follow-up work is to improve the performance/efficiency for long contexts*. Some intriguing starting points could be attempting to exploit model-specific information, e.g., it is also interesting that LLMLingua outperforms LLMLingua2 in this compression as guided search formulation. In summary, we now claim this framework can achieve *decent runtime efficiency*, as what we replied to KfU1.\n>Writing Issues\n\nSorry for the typos. We also find some places that can be improved, including the one suggested by you and KRwQ. We'll ensure that all these will be enhanced/ fixed!\n>Clarity, too dense\n\nApologize for this. As what we replied to KfU1 and KRwQ, we'll **follow all your suggestions** to further structure and highlight the takeaways, and make all settings clear to readers.\n\n[1] Quantifying LMs’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting, ICLR [2] Jailbreak and Guard Aligned LMs with Only Few In-Context Demonstrations, arxiv [3] Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses, NeurIPS [4] Evolution Strategies as a Scalable Alternative to RL, arxiv", "claims_and_evidence": "The claims in the paper are generally well-supported by the empirical results presented in both the main text and the appendix. The authors assert that pruning random demonstrations into \"gibberish\" enhances LLM performance across diverse tasks, and they provide results in tables (e.g., Table 1, Table 2) showing improvements over baselines like RLPrompt across multiple datasets and models.\n\nHowever, the claim that pruned prompts work better lacks a comprehensive mechanistic explanation. While section 6 offers some analysis on the role of label words, the evidence is limited and does not fully elucidate why pruning enhances performance.", "methods_and_evaluation_criteria": "The proposed method, PromptQuine, utilizes an evolutionary search framework based on genetic algorithms to optimize prompt pruning, which is a sensible approach for the discrete optimization problem of finding effective prompt subsequences. This choice aligns with the problem's combinatorial nature, where gradient-based methods are less applicable.\n\nEvaluation criteria include standard metrics such as accuracy for classification and math reasoning.", "theoretical_claims": "The paper does not present theoretical claims or formal proofs. It is primarily an empirical study focused on demonstrating the efficacy of the pruning-based approach.", "experimental_designs_or_analyses": "The experimental designs appear sound and robust. The authors evaluate their method across many tasks and LLMs, using datasets detailed in Appendix C (Table 6) and comparing against baselines like RLPrompt, LLMLingua, and EvoPrompt.", "relation_to_broader_scientific_literature": "The paper situates its contributions within the literature on prompt optimization and in-context learning (ICL).", "essential_references_not_discussed": "The paper cites relevant prior work adequately.", "other_strengths_and_weaknesses": "### Strengths\n\n- Originality: The idea of pruning demonstrations into \"gibberish\" seems novel, creatively challenging the norm of natural language prompts and offering a fresh perspective on ICL optimization.\n- Empirical Rigor: Extensive experiments across tasks and models provide strong evidence of the method’s effectiveness and generalizability.\n- Significance: The findings could inspire new directions in prompt design and ICL stabilization, as noted in the conclusions.\n\n### Weaknesses\n\n- Clarity: The paper is dense, with many details relegated to the appendix.\n- Analysis Depth: The limited mechanistic analysis (Section 6) leaves the \"why\" of pruning’s success underexplored.", "other_comments_or_suggestions": "Typos: In the abstract, \"let alone human intuitions.\" should likely be \"let alone human intuition\"", "questions_for_authors": "Can you provide more details on the computational resources required for PromptQuine (e.g., GPU hours)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EKvSScUDaR", "reviewer": "Reviewer_e5jB", "summary": "The paper proposes PromptQuine, an automated prompt optimization strategy that prunes a given prompt using evolutionary search to improve the performance at a given task. The method outperforms existing methods when validated on classification, multi-choice question answering and reasoning datasets across a wide range of models. Moreover, it is more efficient than previously proposed methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate Reviewer e5jB for suggestions on our experimental setups. We are delighted that you acknowledge the **richness in detail** of our study. We address your concerns below:\n\nAnonymous link (AL) for several tables: https://anonymous.4open.science/r/ughj/e5jB/README.md\n>Recap: Claims & Objective\n\nWe argue we didn't claim *Quine outperforms all others*, in abstract, we use \"Gibberish always matches or surpasses,\" in introduction, we use \"Pruning can near SOTA,\" and in each section, we present results objectively. **Instead**, our goal is to deliver **insight that simple ICL pruning** can be on par with work which use various external information, e.g., +tokens, countering common view and contributing to AI emergence from constraints [1,2]. *Quine/TAP are just what we strive to improve under pruning (pursue knowledge over SOTA)*. Also, comparing search algorithms solely by numbers is limiting, as scalable methods e.g. ES can further improve on the same prune landscape. We'll thus add a figure for sample efficiency, comparing TA/ES/greedy.\n>Models from 350M\n\nWe started from GPT2, 125M (Table 10).\n>Not tuned baselines?\n\nPruning results in Table 2 are reported w/ varied ICL initializations, where ICL baseline are direct comparisons. We can't perform BoN across seeds.\n* BoN for TAPruning/\"Greedy\": We **stated in Algo 2, which used held-out valid for BoN (same samples)**.\n* BoN for ICL: The only setup available is to construct exemplars using samples from held-out valid (e.g.,200). It costs many computes(N=51^4*4!=16568064 for 1-shot agnews, taking **4 yrs for a 1-shot dataset BoN w/ vLLM**). We are unable to support so. That is also why all existing work by validation BoN [3,4+] don't do such enumeration and report ICL as us. We follow such convention. Your request requires other principled algos, beyond our work. We show Bo10 under limited samples in AL.\n\n>gradient baselines?(Gb)\n\nGood questions! Let's clarify.\n* White-box (wb) jailbreaking: Using steering vector (SV) does **NOT** imply wb is required. Sorry for ambiguity. We indeed view it hard for Pruning (line362-sparse, line365-potentials), thus we explore w/ a small-scale priming study for dense signals w/ tools from interpretability(SV). We'll enhance paper: show black-box/bbx results first and explore SV. We provide such results in AL README(+GCG result). Besides, we plan to explore further under varied ICL setups in paper (see reply to 8j6A), for rich picture of pruning.\n\n* Other tasks: bbx is enough, and Gb is empirically weak under FSL[6,7].\n\n>Not substantial Analysis\n\nWe agree the insight is intriguing for analysis. Section 6 presents label words in ICL, where [8,9+] exclusively address it in full papers. Pruned token analysis remains hard, with only viable hypothesis (we) being whether function words are pruned more—an insight we deemed limited, as noted in abstract and intro, let alone by human intuition. Yet, we do plan to add studies for ICL stabilization, inspired by KfU1. Given current density, we leave others for future work. \n>Code Files\n\nApologies for the tight schedule. We clearly state this in README, which we'll release+refactor soon or upon request.\n>TAPruning weaker \"greedy pruning\"?\n\n*We respectfully disagree*. Greedy search may refer to steepest-ascent hill-climbing (SAHC), which has limitations: 1.Search landscape can be multimodal and deceptive (section 4.2), where SAHC can stagnate and TA escapes with speed up [10]. 2.Rigorously, TA has solution regions where SAHC won't touch.\n**Case where greedy stagnates entirely**: See AL traces.csv. SAHC is weaker than TA for no progress on this prompt, limiting its generality. We provide new greedy/TA results in AL, where SAHC takes *days* for search.\n>ES no significance?\n\n*We respectfully disagree*, ES is indeed a pioneer study for handling multimodal landscape. We stated in line250, agreed by other reviewers.\n* Efficiency: TAPruning is indeed our algo. In comparison, Quine is on par with published work in terms of runtime efficiency (Table10). Also it's the first token-level search that can optimize in mins for some tasks(line366). While TAPruning dominate runtime on one GPU, ES has potentials to be parallelized [11] (e.g., reproduction), largely reducing runtime (e.g., line90). We'll make what we claimed clear for \"better runtime\" in Table10.\n* Performance: We argue there're consistent improvements over TA across models in nearly all tables, up to 12%, large when converted to #corrects. Please also refer to TAPruning 4-shot in AL README, where TA/Greedy(e.g., 11% lower) are all weaker than ES. We'll put greedy in paper to enhance why ES for broad open-endedness/deception. [10].\n\n[1] Collective Intelligence for DL [2] Open-Endedness is Essential for ASI [3] AutoPrompt [4] GCG [5] RLPrompt [6] TEMPERA [7] ICL Learns Label Relationships yet not.. [8] Larger LMs Do ICL Differently [9] Abandon Objectives [10] Why Greatness cannot be planned [11] ES as a Scalable Alternative to RL", "claims_and_evidence": "PromptQuine claims to outperform all other prompt optimization methods. The considered models and datasets are extensive enough. However, some important baselines like gradient-based methods (Autoprompt or other variants like GCG) are missing. Moreover, the method is said to be efficient but it takes much more time to run compared to the greedy pruning baseline which has similar performance.", "methods_and_evaluation_criteria": "The method is evaluated on a wide range of datasets using models of varying sizes (from 350M to 70B parameters) and architectures (encoders and decoders).", "theoretical_claims": "None", "experimental_designs_or_analyses": "The baselines are not well-tuned (see questions).\nThe 1-shot ICL setting is misleading. You are still using multiple examples for fitness estimation which effectively extract information form these examples. If multiple examples are used for PromptQuine, Best-of-N selection should be used for manual prompts, TAPruning or greedy pruning for fair comparison. \nThe analysis section is no very substantial. Apart from the task label and signal words, there is no quantitative analysis about the kind of tokens that are pruned by the algorithm. This might help us better understand what kind of tokens models are more sensitive to.", "relation_to_broader_scientific_literature": "The paper proposes a prompt search technique. It is a prompt engineering method that seeks to maximize performance without modifying the model weights. It is also related to feature attribution and how LMs respond to unnatural language.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n- The paper contains a detailed appendix explaining in great detail all the experiments. \n\nWeaknesses:\n- TAPruning seems to be a weaker version of greedy pruning which is not constrained by the order of the tokens to be pruned. The significance of the proposed method is rather limited as its performance is still very close to the baseline while being far less efficient (as show in Table 10).\n Concerning clarity, some parts of the paper like section 5.2 are rather hard to read and seem detached from the rest of the paper.", "other_comments_or_suggestions": "None", "questions_for_authors": "1- What is the 4-shot performance of TAPruning? It is not reported in Table 2. \n\n2- What is the performance of greedily pruning the tokens? (no left-to-right pruning order as in TAPruning but select the best position to prune each time) Please also include the 4-shot performance of this method. \n\n3 - What is the performance of the baselines when using the same number of examples as PromptQuine? For example, best-of-n selection could be used with manual prompting and greedy pruning. \n\n4 - How does a gradient-based method (like Autoprompt or other varians like GCG) compare to PromptQuine?\n\n5 - Why don't you include the latest approaches (like gradient-based ones) for jailbreaking LLMs in your experiments? What is the main advantage of PromptQuine compared to these methods? PromptQuine relies on steering vectors which require access to the language model.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this submission, the authors propose a prompt optimization framework called PromptQuine, an evolutionary search framework that automatically searches for a pruning strategy. This system performs in-context optimization by evolving and refining effective prompts by leveraging only the tokens present within the context. The resulting pruned prompts can often be gibberish but retain some key features, including reference to the task at hand, and demonstrate improvements compared to other baselines.

The reviewers believe that the paper provides an original, significant scientific contribution and shows interesting empirical results. The reviewers appreciated the extensive experiments across various tasks and models which combined provide strong evidence of the method’s effectiveness and generalizability. The reviewers also highlighted several notable weaknesses, such as vague and ambiguous claims and terminology, as well as a lack of clarity throughout the paper. While the authors have addressed these during the rebuttal, I expect these will be incorporated into the final version of the paper. Furthermore, the reviewers also noted limited mechanistic analysis present in the work, which would strengthen the paper.

Overall Assessment: Based on the reviewers' comments and discussions during the rebuttal, I conclude that this paper makes a strong technical contribution and should be accepted. I hope the reviewers’ comments can help the authors prepare an improved version of this submission for camera-ready.
Decision: Accept (poster)