System Prompt Optimization with Meta-Learning
Authors: Yumin Choi
Jinheon Baek
Sung Ju Hwang
Venue: NeurIPS 2025 poster ()
Keywords: Prompt Optimization
Meta-Learning
System Prompt Optimization
Primary area: deep_learning
TL;DR: We propose a novel bilevel system prompt optimization problem and a novel meta-learning framework to tackle it.

Abstract: Large Language Models (LLMs) have shown remarkable capabilities, with optimizing their input prompts playing a pivotal role in maximizing their performance. However, while LLM prompts consist of both the task-agnostic system prompts and task-specific user prompts, existing work on prompt optimization has focused on user prompts specific to individual queries or tasks, and largely overlooked the system prompt that is, once optimized, applicable across different tasks and domains. Motivated by this, we introduce the novel problem of bilevel system prompt optimization, whose objective is to design system prompts that are robust to diverse user prompts and transferable to unseen tasks. To tackle this problem, we then propose a meta-learning framework, which meta-learns the system prompt by optimizing it over various user prompts across multiple datasets, while simultaneously updating the user prompts in an iterative manner to ensure synergy between them. We conduct experiments on 14 unseen datasets spanning 5 different domains, on which we show that our approach produces system prompts that generalize effectively to diverse user prompts. Also, our findings reveal that the optimized system prompt enables rapid adaptation even to unseen tasks, requiring fewer optimization steps for test-time user prompts while achieving improved performance.

System Prompt Optimization with Meta-Learning
Yumin Choi Jinheon Baek Sung Ju Hwang 1.2
KAIST', DeepAuto.ai
(yuminchoi, jinheon back, haa.jj..wwnnkkaaaii ac.kr kr
Abstract
Large Language Models (LLMs) have shown remarkable capabilities, with opti- mizing their input prompts playing a pivotal role in maximizing their performance. However, while LLM prompts consist of both the task-agnostic system prompts and task-specific user prompts, existing work prompt optimization has focused on user prompts specific to individual queries or tasks, and largely overlooked the system prompt that is, once optimized, applicable across different tasks and domains. Motivated by this, we introduce the novel problem of bilevel system prompt optimization, whose objective is design system prompts that are robust diverse user prompts and transferable to unseen tasks. To tackle this problem, we then propose a meta- veta-learning framework, which meta-learns the system prompt by optimizing it over various user prompts across multiple datasets, while simultane- ously updating the user prompts in an iterative manner to ensure synergy between them. We conduct experiments on 14 unseen datasets spanning 5 different domains, on which we show that our approach produces system prompts that generalize effec- tively to diverse user prompts. Also, prompt enables rapid adaptation even to unseen tasks, requiring fewer optimization steps for test-time user prompts while achieving improved performance.
findings reveal that the optimized system
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks and domains [1, 7, 23]. To effectively harness LLMs in diverse application scenarios, prompts play pivotal role in guiding their behavior and ensuring their outputs align with user goals, which comprise two components: system prompts and user prompts [25]. Specifically, system prompts are task- task-agnostic instructions that define the foundational behavior and constraints of the L.M (which are also designed to be applicable to multiple tasks and domains); whereas, user prompts are task-specific inputs designed to elicit responses tailored to solving particular queries or tasks.
Alongside the advancement of LLMs, as their performance is highly sensitive to the prompts provided, there has been surge of interest in designing effective prompts. Traditionally, manual prompt crafting has been the dominant approach, which has led to the discovery of several prompts, such as Chain- of-Thought, which is known for enhancing the reasoning capabilities of LLMs [39]. However, this manual design process is labor-intensive and limited in scalability. Therefore, to overcome these limitations, the field of automatic prompt optimization has emerged, which aims automatically improve prompts by utilizing LLMs directly or further integrating them with algorithms to explore and generate more effective prompt variations [12, 14, 41, 45]. Specifically, notable methods include textual gradients [27], which produces gradients (that criticize prompts) in text based on model prediction results and formulates new prompts iteratively, and the approach with Monte Carlo Tree Search (MCTS) [37], which explores and evaluates various prompt configurations through tree search.
"qqual contribution; Code available http:://gthhabccc///oo/////a/aa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(A) P Prompt Imput in LLM
(B) Task-Specific Optimization
(C) B Bilevel System Prompt Optimization (Ours)
Aask-Agnos fit System Prompt:
Source Task Reasoning
5 Source Tasks
Ressoning
Grounding
Safety
Yourre helght aedistent
Tesk-Specific Optimization
Meta-Lesn Neta-Learinng Framework
Task- User Prompt
Spec-S Optimiration
Mata-K nowwldge
Example:
Count the obyocts in aeginn actimoos
lase cantret, violin flute.
System Prompt
User Prompt
System Prompt
Optinized User Prompt
System Prompt
User Prompt
System Prompt
User Pronpt
Optimized
System Prompt
Optimized
Target Tasks
Target Tasks
Optimized User Prompt
Reasoning
Optimized System Prompt
Reasoning
Mudical
Medical
Res pesposs::
Target Task Only
Generalizable
Thomsar olgoctt
Revlew
Analysiaa
To Unseen Tasks
Review
Analynix
Figure 1: Concept Figure. (A) The input prompt provided to LLMs typically consists of a task- agnostic system prompt. task-specific user prompt, and target example to handle. (B) Conventional Task-Specific Optimization focuses on optimizing user prompts for a single task but shows limited generalization other tasks. (C) The goal of Bilevel System Prompt Optimization (Ours) is to enable the optimized system prompt to generalize effectively to unseen target tasks, for which we utilize a meta-learning framework to derive meta-knowledge from multiple source tasks.
However, despite various studies on prompt optimization, they have mainly focused on user prompts for solving a specific task, while overlooking the system prompts that could significantly influence response generation and have far-reaching effects across diverse tasks and domains. Notably, there are a couple of benefits in optimizing system prompts. First, system prompts serve as the foundational instructions that are invariant, meaning that single well-optimized system prompt can generalize across multiple tasks and domains. Second, the optimized system prompts can establish robust behavioral framework, enabling LLMs to adapt more robustly to unseen user prompts and domains, while having the potential to create a synergistic relationship with user prompts.
To address this gap, we introduce the novel problem of bilevel system prompt optimization (Figure 1), which aims to design system prompts that can be effectively coupled with diverse user prompts and generalizable across wide range of tasks, including those not seen during optimization. However, in contrast to conventional approaches that focus on optimizing only the user prompts, the proposed problem introduces unique challenges as it requires optimizing two objectives (system and user prompts) simultaneously. To handle this, we then frame it as a bilevel optimization framework, as its hierarchical structure allows us to decouple the optimization processes (for system and user prompts) while capturing their dependency. Intuitively, within the bilevel optimization, the system prompt (optimized to generalize for diverse tasks) forms the higher-level optimization objective, while the user prompts (optimized to maximize task- performance) form the lower-level objective.
Building on this bilevel formulation, we then propose to tackle the aforementioned problem of system prompt optimization through meta-learning framework, which is particularly well-suited as it learns to generalize from distribution of tasks (rather than individual queries and tasks) and subsequently enables the robust and rapid adaptation to various user prompts and tasks. Specifically, in our problem setup, the proposed meta- veta-learning framework meta-trains the system prompt to be optimized over diverse range of user prompts and tasks (via the higher-level optimization objective), equipping with the ability to generalize even to unseen instances. Furthermore, by iteratively updating the user prompts through the lower-level optimization objective within the meta-learning loop, our approach ensures that the system prompt is optimized in synergy with diverse user prompts. Additionally, this meta-learning design choice offers a couple of advantages. First, compared to approaches that are not on meta-learning [43], the proposed framework is designed to be superior to them in handling various unseen prompts and tasks, which are prevalent in real-world scenarios. Also, the framework is highly versatile since it allows for the use of any prompt optimization techniques operationalize it. We refer to our overall method as Meta-level System Prompt Optimizer (in short, MetaSPO).
We extensively validate our MetaSPO framework on 14 unseen tasks across 5 diverse domains under two different real-world scenarios: 1) unseen generalization, where the optimized system prompt is directly leveraged to test time queries without any further optimization of user prompts; 2) test-time adaptation, where the user prompts (specific to target tasks) are further optimized with few examples from them (while the optimized system prompt remains fixed). From this, we then observe that, in

---

the unseen generalization scenario, the optimized system prompt significantly outperforms baseline methods, demonstrating its strong generalization capabilities across diverse, unseen tasks and user prompts. Additionally, in the test-time adaptation scenario, the optimized system prompt facilitates more efficient user prompt optimization, leading to faster convergence with superior performance.
2 Related Work
System prompts. introduced in ChatGPT [25]. have become an integral part of modern LLMs, playing crucial role in defining foundational behavior [2. 11, 40J. As their adoption grows, many studies on system prompts have begun to uncover their potential. To mention a few, Zheng et al. [44] demonstrates that incorporating a persona into system prompts can enhance L.M performance on certain role-playing scenarios. Additionally, some recent studies propose training techniques to enhance model alignment with various system prompts [19, 11]. particular, Wallace et al. [35] suggests training the model to follow instructions with the highest importance, written in system prompts, with specific data generation strategies for it. However, unlike the aforementioned studies that aim to align and test models to pre-defined system prompts (which are handcrafted), our work focuses on automatically optimizing system prompts.
System Prompts
Prompt Optimization As the performance of LLMs is highly sensitive to the quality and structure of their prompts, the field of prompt optimization has received much attention. Early studies rely on gradient- based methods to adjust a small number of trainable parameters and inject them into LLMs as an embedding-lewe
soft prompt: however, they are computationally expensive and unsuitable for use with closed-source models [22, 20, 38). To address this, gradient-free methods have emerged, whose core idea is to generate candidate prompts with LLMs and evaluate them iteratively to select the most effective one, as in APE [45] and OPRO [.1]. Also, there exist methods that additionally perform problem analysis for the current prompt before crafting the optimized prompts [27, 8, 37]. Furthermore, some of the other works leverage the idea of cvolutionary algorithms (like crossover and mutation) in optimizing prompts with LLMs [14, 12, 9]. Despite these advancements, existing studies have centered optimizing user prompts specific certain tasks, with limited exploration of system prompts. Also, although very recent studies have started exploring the system prompt, either their focus is restricted to only the safety-related tasks [46] or lacks consideration of interactions with diverse user prompts [43], despite the fact that the optimized system prompt should be generalizable over diverse tasks with various user prompts. To this end, we approach address this gap through the novel formulation of bilevel system prompt optimization, and tackle it with meta-learning.
Meta-Learning Meta-learning, or the concept of learning to learn, aims to acquire generalizable knowledge across a distribution of tasks and enable models to adapt to new tasks with no or minimal training, unlike conventional approaches that optimize for single task dataset. To be specific, the approach called Model-Agnostic Meta- Leatnnen (MAML) learns a shared initialization that, when used for fine-tuning, enables rapid adaptation across tasks [13]. Also, Matching Networks [34] and Prototypical Networks [32] first represent task distributions over the embedding space by mapping and learning samples over it, and then use this learned embedding for adaptation. Recently, meta-learning has also been adopted in the domain of prompt optimization, but prior works have focused only on gradient-based methods (fine-tuning trainable parameters), orthogonal to our focus on gradient-free approaches [38, 28, 16]. Also, meta- meta-learning has not been explored for system prompt optimization.
3 Methodology
We begin with preliminaries, providing a formal explanation of Large Language Models (LLMs) and conventional prompt optimization techniques. We then introduce the problem of bilevel system prompt optimization and propose meta-leamming-basel approach to tackle it.
3.1 Preliminary
Large Language Models Formally, LLMs take the input æ. which typically consists of a system prompt 8, user prompt and a specific query to respond q, then generate the output y. formalized as follows: y = LLM(x)), where x = [s, u, q] and each element (such as 8, u, q. æ, and y) is represented as sequence of tokens, c.g., [i..... s.]. In this formulation, the system prompt
3

---

(A) Inner Loop: User Prompt Optimization
Reasoning
Medical
Safety
(A.1) Candidate User Prompt Generation
(A.2) Generated User Prompt Evaluation
Systimm Prompt:
User Prompt:
Edle pillpllf pesls
System Prompt:... User Prompt
Sdeelogh (og gagggg/ppiie puckls
User Prompt:
Szeelige gil paddss
System Prompt....
System Prompt:
41 User Prompt
Optimized User Prompt:
Examples:
Wrong Examples
Analysed Problem
Examples:
System Prompt:...
teut toot Clee cacally....
#2 User Prompt
Exampie
Exarcies
Wrong
Antlyzed roblem
Prorust
Frongt
Examples:
Prediction
Analysis
PromptGanenateee
Prompt
Prompt
System Prompt
System Prompt:
System Prompt:
System Prompti
Optimized System Prompt:
an as abvened catai usi
Examples:....
Prompt:
U...
E
E...
Tix un salpl
Analyzed Problem
wi User Prompts
Wrong Examples
Tex
User Prompt
U...
Salde
Your ouuuereeei en is
A1 Sysbem Prompt Prompt:...
Examples:
E...
I.laadia Diffowa Typec Quarie
U...
Examplesi...
E......
42 Systom
Analyred
Wrong
Prompt Selection
Prompt Evaluation
PrompaGennration
Fallure Analysis
Answer Prodiction
(B) Outer Loop: System Prompt Optimization
Figure 2: Overview of MetaSPO, which consists of the inner loop for user prompt optimization and the outer loop for system prompt optimization, operationalized through the meta-learning framework. (A) Inner Loop generates candidate user prompts by analyzing incorrectly predicted examples and then evaluates them with the system prompt select refined prompts for individual tasks. (B) Outer Loop generates candidate system prompts by analyzing incorrect examples from all source tasks, and then evaluates them across various user prompts and tasks to ensure generalizability.
defines high-level behavioral guidelines that are designed to be task-agnostic, the user prompt specifies the fine-grained task or action to performed, and the query q contains the specific input or instance that requires a response, all of which allow the LLM to produce the desired output y.
Prompt Optimization Given the sensitivity of LLMS to the prompts they receive, prompt opti- mization emerges as an effective solution, whose objective is to automatically discover prompts that maximize task performance through iterative refinement processes, guided by performance feedback from available examples. Formally, let 7' denote task, which represents a dataset or distribution of input-output pairs (q, a), where q is the query and a is the ground truth answer. Then, the goal of prompt optimization is to identify the optimal prompt u' (starting from the initial prompt u) that maximizes the task performance on T. To achieve this, previous works [45, 27, 37] typically formulate the objective function (and propose various methods to optimize it), as follows:
u arg max, ()) (LLMsss, u, q), a)],
where / a metric function (e.g., accuracy or Fl score) that evaluates the quality of the model output over the ground truth response for the examples in the target task T.
However. despite the advancements in this prompt optimization, previous studies exclusively focus on optimizing user prompts u. leaving system prompts largely underexplored and subsequently introducing a couple of notable limitations. First, the user queries, used for prompt optimization, are typically drawn from one specific task distribution. As result, while effective for test-time queries within the same distribution, these methods struggle with generalization queries and tasks outside the training distribution [21]. In other words, this lack of transferability necessitates re-optimizing user prompts for each new task, which is computationally expensive and time-consuming. Second, while the system prompt offers potential benefits as a universal behavioral guide for LLMs (that have an orthogonal effect to user prompts, and can further enhance the LLM performance when used with optimized user prompts), previous methods exclude the system prompt from the optimization process. By limiting the exploration to user prompts alone, the optimization process is restricted to finding user prompts that may be locally optimal for specific tasks, while overlooking the potential of system prompts to contextualize LLMS (synergized with user prompts) in a task-agnostic manner.
3.2 Bilevel System Prompt Optimization
To address the aforementioned limitations of conventional prompt optimization, we introduce the novel problem of bilevel system prompt optimization, which aims to design system prompts that robust to diverse user prompts and tasks. It is worth noting that we formulate this problem as

---

bilevel optimization setting due to the hierarchical dependency between system and user prompts: the system prompt should generalize across tasks (forming the higher-level objective) while also synergizing with user prompts that are optimized for specific tasks (forming the lower-level objective). Formally, the goal is to discover the system prompt 8* (starting from the initial system prompt s) that maximizes the performance over distribution of tasks T, while ensuring that it synergizes with user prompts (u,) optimized for the specific task Ti (where T, E T), which is defined as follows:
8" arg max, E Exj- ( (LLM((s uf q), a]ll where u = arg max, ((),))>;; (LLM(( u, q), a)]
To solve this formulation of bilevel optimization, we particularly adopt an iterative approach that alternates between optimizing the system prompt and the user prompts. Specifically, at each iteration, the inner optimization problem first focuses on updating the user prompts u for individual tasks Ti to maximize task tusk-specific performance while fixing the (previously optimized) system prompt Once the user prompts converge for their respective tasks (or a certain number of optimization steps for computational efficiency). the outer optimization problem then updates the system prompt by optimizing it over the distribution of tasks T while considering the updated user prompts u" from the inner loop. Note that operationalize this procedure with meta-learning, which inherently suits the bilevel structure by enabling the system prompt to learn generalization over task distributions through the outer loop while easily adapting to task-optimized user prompts in the inner loop, discussed next.
3.3 MetaSPO: Meta-level System Prompt Optimizer
We now turn to provide detailed descriptions of the Meta-level System Prompt Optimizer (MetaSPO), which consists of two optimization loops, illustrated in Figure 2.
Inner Loop It is worth noting that since MetaSPO is designed general framework, it allows the use of any off-the- offtthe-shelf prompt optimization techniques, and, among many, one instantiation is to iteratively update the prompt to correct examples that were previously handled incorrectly, thereby improving the overall performance on the target tasks, which is similar to Pryzant et al. [27]. Specifically, the first step, measure the performance of the current user prompt on examples from the target task and identify responses that are incorrect. To improve the performance, we then aim refine the prompt to address these errors, and to achieve this, we conduct failure analysis, where we prompt the LLM with the current user prompt and incorrect examples to uncover the underlying issues. After that, based on the user prompt and its analyzed problems, we further prompt the LLM multiple times to generate (potentially refined) candidate user prompts. However, as not all generated prompts result performance improvement, we measure the performance of prompts (including previously used ones) on the target task, and select the top k prompts that perform the best.
Outer Loop The outer loop follows a similar structure to the inner loop but differs in key aspects, as its objective is find the system prompt that maximizes the performance across distribution of tasks, rather than focusing on single task. Specifically, to identify the incorrect responses (used for analyzing problems in the system prompt), we first measure the performance of the system prompt for each task alongside the user prompts and examples associated with that task, and then aggregate the incorrect responses across tasks. After that, similar to the inner loop, we analyze the system prompt with incorrect responses (from all tasks), and, based on this analysis, we generate multiple candidate system prompts via L.LM prompting. Lastly, we evaluate the performance of these system prompts, not just for individual tasks but across the distribution of tasks, in conjunction with their corresponding (optimized) user prompts and examples, and then select the top k system prompts that demonstrate the best performance. The full algorithm of MetaSPO and the prompts used to elicit the desired output from the LLM for each step are provided A.3 and A.4, respectively.
4 Experiments
In this section, we now describe experimental setups, including tasks, datasets, models, and imple- mentation details, and then showcase the effectiveness of the proposed MetaSPO on them.
5

---

Table I: Main Results on Unseen Generalization. For each target task, report the average score of the system prompt paired with ten different user prompts. Please refer to A. for detailed descriptions of each task with its full name. The best performance is highlighted in bold.
Medical
Review Analysis
Reasoning
Safely
Grounding
Methods
Ped.
Den.
Ele.
Spe.
Cou.
Epi.
CoL.
A.H.
Eth.
N.Q.
Web.
Avg.
Default
36.1
38.9
25.8
32.3
41.3
41.5
29.3
43,5
28.3
56.6
21.2
28.7
15.1
11.6
32.2
CuT
36.1
42.7
26.0
32.0
36.8
40.3
25.0
45.6
37.2
62.0
21.9
31.9
15,9
12.0
33.2
Global
Service
34.4
35.2
20.2
30.6
59.0
53.2
52.2
30.6
37.6
56.6
21.1
26.9
11.4
34.2
SPRIG
41.6
42.2
28.4
35.7
47.9
47.4
38.6
39.3
29.9
59.9
23.0
31.1
14.1
11.2
35.0
MetaSPO
45.7
43.1
31.1
36.3
67.2
66.0
61.4
44,5
39.6
64.5
24.9
37.6
9.5
7.7
41.4
SPRIG
41.2
41.8
29.6
35.3
61.6
57.4
51.3
30.1
34,5
51.5
24.0
32.1
16.1
12.0
37.0
Domain
MetaSPO
48.9
46.7
36.4
40.0
61.8
64.9
61.5
47.1
43.0
66.6
29.1
43.9
19.1
13.7
44.5
67
Target Domain Similarity
Metric w/ MetaSPO
Inprovementt (56)
103
Pearson 0.56
0.38
0.32
0.7
0.3
0.0
80
Smilaett
Ang. Matric
0.5-
0.6-
0.46
9.4
0.46
0.42
0.2
20
DeColi
Service SPRIG
MctasPO
0.0
Metric w/ Default
02
04 0.6
85.0%
Avg. Source- Target Similartty
022
(BoW Rank Correlation)
Mediral
Col
Target Domain
Rewsening
Safwty
0.38
Domain
5m
Figure 3: Performance of user prompts with MetaSPO (y) and De- fault (x). Points over = x indicate the su- periority of MetaSPO.
Figure 4: Relative performance improvements of our MetaSPO over Default as function of the source-target tasks similar- ity, where the similarity is mea- sured by Bag-of-Wordd (BoW).
Figure 5: Results with generaliza- tion across different domains. (Left:) Performance of MetaSPO for domains not used for prompt optimization, with their similarity. (Right:) Effect of the number of training domains with stds.
4.1 Experimental Setup
Tasks To evaluate MetaSPO, we consider two scenarios that reflect real-world applications of the optimized system prompt. First, in the Unseen Generalization scenario, where there is no data available for the target task, the optimized system prompt is directly applied test-time queries and user prompts without any additional optimization on them. Meanwhile, in the Test-Time Adaptation scenario, the user prompts provided are further optimized using a small number of examples from the target task, while the optimized system prompt remains fixed. In addition to them, we further consider two different settings in optimizing system prompts: Global where the goal global system prompt that is generalizable across domains, and Domain, more relaxed case where the system prompt is designed and deployed to handle tasks and queries within one specific domain
obtain the
Datasets To extensively evaluate the efficacy of the (optimized) system prompts, our evaluation suite spans 5 distinct domains (over 34 different tasks), as follows: Medical - which aims to answer medical-relaten queries [26]: Review Analysis which aims to predict the sentiment of customer reviews [15]; Reasoning which evaluates the logical and analytical thinking of models [7]: Ground- ing which assesses whether the generated responses are grounded in the provided context [29]; Safety which measures the ability to classify harmful or sensitive content [5]. We note that, for each domain, 4 source tasks are collected to optimize system prompts. Also, for the Medical, Review Analysis, Reasoning, Grounding, and Safety domains, we use 4, 3, 3. 2, and 2 target tasks (which are not seen during prompt optimization) to evaluate the system prompts, respectively. More detailed descriptions of source and target tasks across all domains are provided in A.I.
which
Basclines and Our Model The models evaluated in this work are as follows: 1. Default - which uses one of the most widely used system prompts, "You are a helpful assistant."; Chain of Thought (CoT) which incorporates "Let's think step by step." into the system prompt, allow LLMs to
Unless otherwise stated, report results with the Domain setup.

---

Table 2: Main Results on Test-Time Adaptation, where we optimize the user prompts with examples from target tasks, while fixing the system prompt. The av- erage score for each domain is reported.
Avg. Metric
0.70
0.70
0.65
0.65
MetaSPO
Default
12
16
150 200
Heraaion
Number of Examples
Methods Med. Rev. Rea. Saf. Gro. Avg.
Figure 6: Efficiency for test-time adaptation as a func- tion of optimization iterations (left) and data quantity (right). The results are averaged over the Review Anal- ysis and Reasoning domains. The gray dashed lines indicate the final performance the Default baseline.
Default 45.1 68.9 64.0
59.9
17.5 51.1
SPRIG
45.4 69.3
65.3
64.7
17.7
52.5
MctaSPO
45.6 71.4 67.3
67.2
19.9
54.3
think before providing the answer [39]; 3. Service - which uses the hand-crafted commercial system prompt available from Askell [3]; 4. SPRIG which automatically optimizes the system prompt over diverse set of tasks based on the genetic algorithm (without meta-lcarning) [43]; 5. MetaSPO which is our full model that iteratively performs system prompt optimization via meta-learning.
Implementation Details For fair comparison of different approaches, we primarily use Llama 3.2 (3B) [11] the base model for generating responses, and GPT-4o mini as the prompt optimizer. We use the temperature of 0 for the base model (to ensure consistency) and for the optimization model (to yield variety). For our MctaSPO, we iterate the inner and outer loops three times. Also, in system prompt optimization, we generate nine different prompts and maintain one, while for the user prompt, we generate and maintain three. During the problem analysis step for the current prompts, we use three incorrect examples for the user prompt optimization, and two incorrect examples per task (aggregated over all tasks) for the system prompt optimization. All experiments are conducted with three different runs, and we report their average results. For more details, please refer to A.2.
4.2 Experimental Results and Analyses
Results on Unseen Generalization We first report the performance of optimized system prompts on the unseen generalization scenario, where examples for target tasks not available for prompt optimization; thus, the user prompts are not optimized on them, and, for evaluation, (ten) user prompts on each task obtained via LLM prompting (see A.5 for details). As shown in Table 1, we then observe that MetaSPO consistently outperforms all baselines across both the Global and Domain system prompt optimization settings. To see whether the optimized system prompt contributes to the performance gain when coupled with diverse user prompts, we visualize the performance of (randomly sampled 20%) user prompts using the system prompt optimized from MctaSPO, compared to the default system prompt in Figure (see B.4 for the break-down results by domain). From this, we observe that 85.0% of the user prompts exhibit improved performance with MetaSPO, indicating that it effectively and robustly enhances the performance across a broad range of user prompts.
Analysis on Similarity between Source and Target Tasks We hypothesize that if the target tasks (used for evaluation) are more similar to the source tasks (used for meta-learning), the system prompt optimized from source tasks is more effective for target tasks. To confirm this, we measure the similarity of examples between source and target tasks using either Bag-of- Words rank correlation [21] or cosine similarity over the embedding space with the language model [33], then measure the Pearson correlation of the similarity with its corresponding performance improvement (over the Default system prompt). The results in Figure show a positive correlation between the source-target task similarity and the improvement, with Pearson correlation coefficient of +0.56. Also, the cosine similarity result in B.5 shows a positive correlation with performance improvement. Yet, more interestingly, we for low-similarity tasks, yielding performance gains.
observe that MetaSPO remains effective These results demonstrate that, while including more source tasks that are close to target tasks is beneficial, MetaSPO enables learning generalizable knowledge that is useful across diverse tasks.
Analysis on Generalization Across Domains We further validate MetaSPO in the more challeng- ing scenario, where there are no overlapping domains between source tasks (used for training) and target tasks (used for evaluation). Specifically, optimize system prompts using tasks from three
7

---

Separated
Win
Attention Ratio
Aog. Metric
0.0-
Medical
+30.3%
Systeme Useu--eer
0.3
Bevarn
+4.4%
0.2
0.5
Rescon Analysia
Reasoning-
5.6%
Reasoninn
36.66%
Petbllt
Service SSPRI
Me
Eng-
0.0
Number of Source Tasks
Medical
Review
Reasoning Safety
Figure 7: Results as a func-
Figure 8: Comparison of in- put prompt structures, with separated inputs (system/user roles are explicitly separated) and unified input (both are as- signed to the user role).
Figure 9: Ratios of the attention scores directed toward system prompts over user prompts. A higher ratio indicates that LLMs pay more attention to the system prompt than the user prompt.
tion of the number of source tasks for system prompt opti- mization on MetaSPO, rang- ing from to 6. Standard devi- ation shown as shaded areas.
domains and apply them to evaluate performance on tasks from entirely different target domains, with the goal of testing the robustness and adaptability of the optimized system prompts given the inherent differences of domains between training and evaluation. Then, as shown Figure (left), MetaSPO consistently outperforms baselines even if there are no strong correlations between source and target domains (while it becomes slightly more effective when their similarity is high). For example, in the reasoning domain, where the source domains have average similarity of only 0.12, MetaSPO outperforms CoT (a strong baseline for reasoning), which is another baseline (SPRIG) inferior to it. In addition to this, we analyze the impact of increasing the number of training domains on cross-domain generalization scenarios. In Figure 5 (right), we observe that the highest performance is achieved when three domains included during meta-learning. Overall, these results highlight MetaSPO's robustness in various transfer settings (including to low-similarity domains), while underscoring the potential benefits of leveraging diverse set of training domains to enhance generalization.
Results on Test-Time Adaptation We hypothesize that the system prompt optimized through the proposed meta-learning framework further useful in the test-time adaptation scenario (where user prompts on target tasks are additionally optimized), it may offer a good initial point (encapsulating the meta-learned knowledge over diverse source tasks) that can be generalizable and synergized with prompt optimization. As shown in Table 2, the proposed MetaSPO consistently outperforms other methods across all domains, which demonstrates its effectiveness on test-time adaptation.
Efficiency Analysis The system prompt optimized from our MetaSPO is designed to generalize and adapt efficiently to diverse user prompts and tasks. To confirm this, we visualize the performance as a function of the number of iterations and data samples for test-time adaptation, and report the results in Figure 6. From this, we demonstrate that MetaSPO surpasses the Default system prompt across all iterations and data quantities. Also. MetaSPO is more resource efficient, achieving the final performance of Default with 80% fewer optimization iterations and 75% less data. This suggests its practicality for scenarios with limited computational resources and constrained data availability.
Analysis on Number of Source Tasks To see how much the performance of MetaSPO improves on unseen target tasks with respect to the number of source tasks, we conduct experiments, varying the source task numbers. As shown in Figure 7, we find that, as the number of source tasks increases, the performance of MetaSPO improves with greater stability. Yet, the extent to which the performance improves differs across domains. For example, in the Review Analysis domain (which exhibits higher similarity between source and target tasks), performance increases by 17.10% when the number of source tasks increases from to 6. In contrast, the Reasoning Domain shows (despite meaningful) a comparatively smaller improvement of 8.26% under the same condition. These results suggest that the proposed MetaSPO benefits from a larger and more diverse set of source tasks to effectively learn its distribution, with greater performance gains when they closely resemble the target tasks.
Analysis of Roles for System and User Prompts Recall that the input to LLMs is categorized into system and user prompts with distinct roles. To investigate whether assigning (optimized) system and user prompts to their designated spaces and roles is necessary, we consider two input configurations: (1) separated inputs, where the system prompt is assigned to the system role and the user prompt assigned to the user role, and (2) unified input, where the system and user prompts are concatenated
8

---

Table 3: Results with different LLMs for MetaSPO, where Cross Model refers to prompts optimized with Llama3.2 (3B) and applied other models. Results are averaged over Review Analysis and Reasoning domains. Numbers in bold indicate the highest, followed by underline.
Table 4: Variations of MetaSPO, where different compositions of optimization methods within our meta-leariing framework. We report the average performance score for each domain. The highest score in each domain is highlighted in bold, while the second-highest is underlined.
Base Models
Methods
Med.
Rev.
Rea.
Saf.
Gro.
Avg.
Methods
Llama 3.1 (8B)
Qwen 2.5 (7B) GPT-4o mini
Default
33.3
37.4
42.8
25.0
13.4
30.3
Default
55.9
58.2
77.2
SPRIG
37.0
56.8
38.7
35.9
Service
CoT
50.6
59.6
65.5
58.6
72.9
75.9
Outer Loop
36.8
58.1
48.8
32.4
14.8
38.2
SPRIG
55.2
$8.0
75.6
MetaSPO w/ APE
39.8
60.1
48.1
30.4
16.2
38.9
MetaSPO
69.8
73.2
79.6
MetaSPO w/ EVO
41.6
60.0
50.2
33.2
16.0
40.2
Cross Model
70.1
68.3
78.3
MetaSPO
62.7
52.2
36.5
16.4
42.2
and assigned to the user role. We then compare their effectiveness by generating outputs from these two configuration setups and measuring the win ratio of one over the other. The results in Figure demonstrate that the prompt structure with the separated inputs outperforms the unified structure across all domains, except for reasoning. This performance gap may be attributed to the fact that modern LLMs are trained to interpret system and user prompts differently, and thus perform better when these roles are explicitly leveraged in separation
Analysis on Attention Scores We hypothesize that if the optimized system prompt offers meaning- ful information to answer the queries, the model will allocate more attention to the system prompt over the user prompt. To verify this, we compare the attention scores directed toward the system prompt versus the user prompt across various methods, where the scores are obtained by averaging the maximum attention values of all heads and layers over the entire decoding process, and visualize the attention score ratios between system and user prompts in Figure 9. From this, we observe that MetaSPO directs more attention to the system prompt compared to baselines over all domains, which indicates that the system prompts optimized from MetaSPO are effectively used steer the LLM.
Analysis with Varying Models We conduct an auxiliary analysis examine whether MetaSPO is versatile with other LLMSs (for response generation) and whether the optimized system prompt from one LLM can be generalizable to other LLMs. For both experiments, we consider the following LLMs: Llama 3.1 (8B). Qwen 2.5 (7B), and GPT-4o mini. We then report the results in Table 3, and, from this, we observe that MetaSPO demonstrates its robustness and generalizability. Specifically, the system prompt optimized by MetaSPO is superior to other baselines regardless of the underlying LLMs used as the base model. Furthermore, the Cross Model, the system prompt optimized for Llama 3.2 (3B) demonstrates strong generalization capabilities when applied to LLMs other than it without requiring additional optimization. Overall, these results confirm that MetaSPO is effective for diverse LLMs, producing system prompts that not only perform well within the LLM they were optimized for but also maintain high performance across different LLMs. Lastly, we further extend our analysis of MetaSPO with different optimizer LLMs and show its robustness, provided in B.6.
Analysis on MetaSPO Variants Note that the proposed MetaSPO is designed to be highly flexible, allowing the use of any off-the-shelf prompt optimization components and their combinations over its bilevel meta-learning framework. For instance, one such variation is perform only the outer loop without iterative refinement of user prompts (called Outer Loop). Also, there could be other variations, such as MetaSPO w/ APE and MetaSPO w/EVO, which use prompt optimization strategies from Zhou et al. [45] and Guo et al. [14] in both the inner and outer loop stages. Then, as Table 4 shows, Outer Loop achieves performance comparable to baselines but falls short of the full MetaSPO framework, demonstrating the effectiveness of synergy in meta-learning by alternating between the inner and outer loops. In addition, two variants of MetaSPO instantiated through different prompt optimization strategies (namely, MetaSPO w/ APE and MetaSPO w/ EVO) outperform baselines substantially, demonstrating its compatibility with any existing prompt optimization techniques.
We present examples for the optimized system prompts in Appendix C, from which we observe that they typically provide more specific role to the LLM beyond the helpful
Qualitative Results

---

assistant (e.g,, you are a knowledgeable and analytical assistant designed to process and analyze information across broad range of topics) and include detailed guidelines or objectives on top of it.
5 Conclusion
In this paper, we introduced the novel problem of bilevel system prompt optimization, which differs from existing prompt optimization work (that primarily targets optimization of task-specific user prompts) and instead focuses on designing system prompts that are robust to diverse user prompts and transferable to unseen tasks and domains. To address this problem, we then proposed MetaSPO, meta-learning framework that optimizes system prompts over a wide range of user prompts and tasks, while iteratively refining the user prompts and using those optimized user prompts during system prompt optimization to ensure effective generalization of the system prompt to various (optimized) user prompts. We extensively validated the proposed MetaSPO framework on 14 unseen datasets across 5 distinct domains, on which we demonstrated that it consistently outperforms baselines in both unseen generalization and test-time adaptation scenarios. We believe our work establishes significant step forward in enhancing the robustness and adaptability of LLMs, enabling the use of optimized system prompts designed to generalize across diverse tasks and domains as well as LLMs.
6 Limitation and Societal Impacts
Limitation Despite the promising results of MetaSPO, our work has some interesting avenues for future work. Specifically, while we demonstrate that MetaSPO works effectively even with a small open-source optimizer (Table 15). its performance depends on the capability of the optimizer LLM. We note that this is indeed common challenge shared across many prompt optimization studies, and to tackle this, developing approaches to maintain strong performance with smaller (open-sourced) LLMs for optimization (such with model distillation) would be an exciting avenue for future work.
Societal Impacts We strongly believe that our MetaSPO (designed to formulate system prompts that are generalizable across diverse tasks and domains, as well as user prompts) has huge potential for broad real-world applications. In the meantime, as the performance of MetaSPO is still far from perfect, for the mission-critical domains (such as biomedical), it should be used carefully. In addition to this, there is a chance that MetaSPO is misused to steer the behavior of LLMs in harmful ways, and while this vulnerability is not unique to our approach but a common challenge faced by existing prompt optimization methods, an additional safeguard for may be needed.
Acknowledgements
This work was supported by Institute for Information & communications Technology Planning (valuation(IIIP)) grant funded by the Korea governmen((MSIT) (RS 11900005 Artificial Intelligence Graduate School Program(KAIST), Real-world Problems ), National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RSS-0223-0026655), a grant of the Korea Machine Learning Ledger Orchestration for Drug Discovery Project (K-MELLODDY), funded by the Ministry of Health & Welfare and Ministry of Science and ICT, Republic of Korea (No. RS2024-00460870) Institute of Information & Communications Technology Planning& & Evaluation (ITPP) with a grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research. (No. RS- 0S-2024-00469482 & RS-2024-00509279 and Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea) & Gwangju Metropolitan City, and i-Scream Media.
A-S-2022-1122000
Meta-lcarning Applicable to
References
[1] Ibrahim Abu Farha and Walid Magdy. From Arabic sentiment analysis to sarcasm detection: The ArSarcasm dataset. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with Shared Task Offensive Language Detection, 2020.
10

---

[2] Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024. URL https: //www/wwwwtat//// cdn. anthropic. wtt.a//pp....ca//eeeee0000...... wttp////bbs......ccaaaaaaaaaaaa. Model_Card, C_a___llad_____ 33 pdf..
[3] Amanda Askell. Claude System Prompt, 2024. status/ status 0taaaa/66677779224444888888888
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer EI Showk, Nelson Elhage, Zac Saffield-Doddee Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, 2022.
[5] Francesco Barbieri, José Camacho-
Luis Espinosa Anke, and Leonardo Neves. Tweet- eval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP, Findings of ACL. Association for Computational Linguisties, 2020.
[6] Jonathan Berant, Andrew Chou, Roy Frostig, and Perey Liang. Semantic parsing on freebase from question-answer answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP. ACL., 2013.
[7] BIG-bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
[8] Yongchao Chen, Jacob Arkin, Yilun Hao, Yang Zhang, Nicholas Roy, and Chuchu Fan. PRompt optimization in multi-step tasks (PROMST): Integrating human feedback and heuristic-bai based sampling. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP. Association for Computational Linguistics, 2024.
[9] Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley Malin, and Kumar Sricharan. Phaseevo: Towards unified in-context prompt optimization for large language models. ArXiv, 2024.
[10] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT. Association for Computational Linguistics, 2019.
[11] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, An- thony S. Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang. Aur' elien Rodriguez, Austen Gregerson, Ava Spataru, Bap tiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, al. The llaaa herd of models. ArXiv, 2024.
[12] Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rock- täschel. Promptbreeder: Self-referential seff-improvement via prompt evolution. In Forty-first International Conference on Machine Learning, ICMI., 2024.
[13] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML, 2017.
[14] Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. The Twelfth International Conference on Learning Representations, ICLR, 2024.

---

[15] Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian J. McAuley. Bridging language and items for retrieval and recommendation. ArXiv, 2024.
[16] Yukun Huang, Kun Qian, and Zhou Yu. Learning better initialization for soft prompts via meta-learning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, IJCNPP, 2023.
[17] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zeullmoyer. Triviaqa: large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL., Association for Computational Linguistics, 2017.
[18) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: benehmark for question answering research. Trans. Assoc. Comput. Linguistics, 2019.
[19] Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to thousands of preferences via system message generalization. In The Thirty-eighth Annual Conference Neural Information Processing Systems, NeurIPS, 2024.
[20] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP. Association for Computational Linguisties, 2021.
[21] Moxin Li. Wenjie Wang, Fuli Feng. Yixin Cao, Jizhi Zhang, and Tat- Tat-Seng Chua. Robust prompt optimization for large language models against distribution shifts. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP. Association for Computational Linguistics, 2023.
[22] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL/JCNLP. Association for Computational Linguistics, 2021.
[23] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher e'e Diana Acosta-Navas, Drew A. Hudson, E. Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim. Neel Guha, Niladri S. Chatterji, O. Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas F. Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models, Annals of the New York Academy of Sciences. 2023.
[24] Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. Ethos: lulti-label hate speech detection dataset. Complex & Intelligent Systems, 2020.
[25] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35, NeurlPS, 2022.
[26] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, CHIL., Proceedings of Machine Learning Research. PMLR, 2022.
[27] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with "gradient descent" and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP. Association for Computational Linguistics, 2023.
12

---

[28] Chengwei Qin, Shafiq R. Joty, Qian Li, and Ruochen Zhao. Learning to initialize: Can meta learning improve cross-task generalization in prompt tuning? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.
[29] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. Proceedings of the 2016 Conference on Empir- ical Methods in Natural Language Processing. EMNLP. The Association for Computational Linguistics, 2016.
[30] Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-3. In Proceedings of The Third Text REtrieval Conference, TREC, NIST Special Publication. National Institute of Standards and Technology (NIST), 1994.
[31] Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Z. Margetts, and Janet B. Pierrehumbert. Hatecheck: Functional tests for hate speech detection models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/JJNNP. Association for Computational Linguistics, 2021.
[32] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems 30, NeurIPS, 2017.
[33] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In Advances in Neural Information Processing Systems 33, NeurlPS, 2020.
[34] Oriol Vinyals, Charles Blundell, Tim Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems 29, NeurIPS, 2016.
[35] Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training lims to prioritize privileged instructions. ArXiv, 2024.
[36] William Yang Wang. 'liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL. Association for Computational Linguistics, 2017.
[37] Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P. Xing, and Zhiting Hu. Promptagent: Strategic planning with language models en- ables expert-level prompt optimization. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
[38] Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogério Feris, Huan Sun, and Yoon Kim. Mul- titask prompt tuning enables parameter-efffcient transfer learning. In The Eleventh International Conference on Leurning Representations, ICLR, 2023.
[39] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35, NeurlPS, 2022.
[40] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang. Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin. Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge. Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren. Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu. Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu. Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 Technical Report, 2024.
[41] Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. The Twelfth International Conference on Learning Representations, ICLR, 2024.
13

---

[42] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP. Association for Computational Linguistics, 2018
[43] Lechen Zhang, Tolga Ergen, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. SPRIG: improving large language model performance by system prompt optimization. ArXiv, 2024.
[44] Mingqian Zheng. Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. When "a helpful assistant" is not really helpful: Personas in system prompts do not improve performances of large language models. In Findings of the Association for Computational Linguistics: EMNLP. Association for Computational Linguistics, 2024.
[45] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
[46] Xiaotian Zou, Yongkang Chen, and Ke Li. Is the system message really important to jailbreaks large language models? ArXiv, 2024.
14

---

Appendix
A Additional Experimental Details
A.1
Task Description
Table 5: Configurations of source and target tasks as well as their corresponding domains.
Domain
Source Tasks
Target Tasks
Test set size
OB/GYN,
Anatomy,
234
Medicine,
Pediatrics,
Medical
(MedMCQA [26]
Pharmacology.
Dental,
500
Pathology
Surgory
369
Office,
Electronics,
500
Beauly,
Review Analysis
Game,
Pet.
500
(Amazon [15])
Sports
500
Baby
Logic Grid Puzzile
Object Counting,
200
Temporal Sequences,
Reasoning (Bighench (7])
Logical Deduction.
Epistemic:
400
Tracking Shuffled Ohjects
Reasoning Colored Objects
400
Tweet Eval [5].
Safety
[36].
Anthropic Harmless Ethos [24]
500
Hatecheck [31].
500
Sarcasm [1]
SQuAD [29].
Grounding
HotperQA [42].
Natural Questions [18).
500
TriviaQA [17].
Web Questions [6]
500
[10]
We provide detailed configuration of the source and target tasks for each domain Table 5, In the Medical, Review Analysis, and Reasoning domains, the source and target tasks are constructed with distinct subsets of individual datasets. Conversely, for the Safety and Grounding domains, multiple datasets combined to define single domain. Notably, in the Grounding domain (whose examples are composed of the given query and its relevant contextual documents). source task examples are constructed using the contexts provided within the dataset, whereas target task examples are formed by concatenating the top five documents retrieved for each instance using a BM25 [30] retriever.
To measure the performance, we primarily use accuracy as the metric in Medical, Review Analysis, and Reasoning. For Safety, we use the F1-score as it involves binary classification tasks, while for Grounding, use Exact Match (EM), which measures whether the generated response is exactly the same as the ground-truth answer.
For data splits, we use predefined train-test splits from datasets (if available), such as MedMCQA, BigBench, WebQA, and Anthropic Harmless. For datasets without predefined splits, such as Amazon, Natural Questions, and Ethos, we randomly divide the training data to create the test sets. Also, for each task, 50 training samples are randomly selected using different seeds across three experimental runs. As summarized in Table 5, the number of test samples is limited to a maximum of 500 in all tasks to reduce the computational burden.
A.2 Additional Implementation Details
We now provide the additional implementation details for other optimization methods included in our experiments. Regarding ProTeGi [27] (presented in Table 2), it performs six iterations with beam size of three. For the Outer Loop, we perform six iterations, which is twice the number of iterations used in MetaSPO. This adjustment ensures an equivalent total number of iterations, as the Outer Loop method lacks an inner loop process. In the case of MetaSPO w/ APE and MetaSPO w/EVO (that are operationalized without the prompt analysis step), we generate 18 new candidate prompts through resampling, crossover, and mutation. For SPRIG, we conduct the experiments using the implementation provided in official repository referenced in the original paper [43]. We iterate SPRIG three times to ensure comparable amount of computation to the proposed MetaSPO (See B.3 for details). Experiments are primarily conducted using an NVIDIA A5000 GPU.
15

---

2: S + {so} A.3 Algorithm of MetaSPO
We present the MetaSPO algorithm, which is composed of alternatives between an Inner Loop and an Outer Loop.
Algorithm 1 MetaSPO
input Task distribution T, Initial system prompt 8, Number iterations N output Optimized system prompt s*
1: 24 f {u:} for each task Ti T
D Initialize user prompt set
2: for N iterations do
3: for each task Т; E 'T do
4: L4; < INNERLOOP(S,,C Ti)
5: end for
6:
U {U; T. E T
7: 8 + OUTERLOOP(S,...
8: end for
9: Return 8* <-
Algorithm 2 INNERLOOP
input Task T; System prompt 8, Set of user prompt 24 Number of new candidates m, Top-k size output Optimized user prompts U1
1: u40 < arg max Biq.a)-;; [LLLM( u. q). a)]
Select the best-performing user prompt
uEM,
2: for m iterations do
3: W; ((g,a) (q. a) T;, LLM(s, uo, 9) + a)
Collect incorrect responses
4: A < Analyzer( 8, to. W;)
>> Analysis the current user prompt, Table Generate candidate prompt, Table
5:
Generator (s, un, A:)
6: U, + U4 U {u}
7: end for
8: U +
arg max ((q.a)~T; Bu-uel((LMM) u, q). K/I
D Select top-k user prompts
9: Return UE
Algorithm 3 OUTERLLOOP
input Task distribution T, System prompt 8, Set of user prompt set U, Number of new candidates
m.
output Optimized system prompt s*
1: 8g f
D> Initialize the system prompt Initialize the system prompt set
3: for m iterations do
4: W++0 0
5:
for each task Т₁ E T do
6:
u, arg (qq.)-T;; [LLM u, a)) D> Select the best- pess-performing prompt
ueu,
7:
W. + ((q (q. T, LLM(s0, ui q) + a}
Collect incorrect responses
8:
WUW: end for
9:
10:
A t Analyzer 8o, W)
> Analysis the current system prompt, Table
s Generator( So. A)
Generate candidate system prompt, Table
12:
S 5-+ (U(s}
end for
14: s" +- arg max BT.TTaa Eur-u
(LLM(s, (LLM)
9),
Evaluate across tasks and user
SES
prompts
15: Return s*
16

---

A.4 Meta Prompts to Implement MetaSPO
In this section, we present the meta prompts used in MetaSPO. The meta prompts for user prompt analysis and generation are detailed in Table and Table 7, respectively. Similarly, the meta prompts for system prompt analysis and generation are provided in Table 8 and Table 9, Additionally, the template for incorrect examples is provided in Table 10.
Table 6: Meta Prompt for Analyzing Failure Cases of the User Prompt.
Roles
Prompts
are user prompt writer tasked with improving language model's user prompt for a specific task. Your goal identify the shortcomings of the prompt and provide comprchensive suggestions for improvement.
System
Here the inputs you will working with:
### System prompt:
|system_ promptt
### User prompt: ppoop_]roopp]
User
This prompt gets following responses wrong: [examples]
### Remember to focus solely on discussing and improving the user prompt.
Wrap the analysis of the user prompi <hmalysis></hnallyesas tags.
Table 7: Meta Prompt for Generating Candidate User Prompts.
Roles
Prompis
You are user prompt writer tasked with improving language model's user prompt for specific lask. Your goal to create improved prompl that enhances the model's performance.
System
Here the inputs you will be working with:
### System prompt: gosstrm_prompe
### User prompt: [user_ promp]]
Wrong examples of the model's responses: [examples)
User
### Analysis of the issues with this user prompt: [analysis]
### Address any problems observed in the examples hased analysis.
#il Ensure the user prompt contains the <Questionn>questiiii QQusttinnn<uuttiiikk//natton where the question be placed.
### The new prompt should be wrapped with <improved_user, </improved, simproved__see: user_ user_proupp>>>>>ppppppppp tags.
17

---

Table 8: Meta Prompt for Analyzing Failure Cases of the System Prompt.
Roles
Prompts
You system prompt writer tasked with improving language model''s system prompt for general tasks, Your goal is to analyze why the current system prompt fails respond correctly in the given examples.
System
Follow these instructions carefully:
Review the current system prompt:
[system prompt)
User
### Wrong responses:
[examples)
## Remember focus solely discussing and improving the system prompt.
#*# Wrap the analysis of the system prompt in the <huallyiee<</maalyeiz>>>>> tags.
Table 9: Meta Prompt for Generating Candidate System Prompts.
Roles
Prompts
You are system prompt writer tasked with improving a language model's 's system prompt. Your goal write better system prompt that can generalized for various tasks.
System
Follow these instructions carefully:
WWW Review the system prompt:
[system_ prompp]__pp
Analysis the current system prompt: {analysis]
User
WWW Based on the information provided, write an improved system prompt.
The new system prompt should wrapped with <uuproved__yatee__room tags.
Table 10: Prompt Template for Incorrect Examples.
<Example> System Prompt: dkystem_promply
User Prompt: [user permppprppp]
Response: [response]
Prediction: {prediction]
The correct label [label] </Example>
18

---

A.5 Generation of User Prompts for Unseen Generalization
In the unseen generalization setting, evaluate whether the system prompt is effective with user prompts not optimized for target tasks. To generate these unoptimized user prompts, we provide GPT-4o mini with input-ouiput example pairs, generating the coarse user prompts following Zhou et al. [45] (please see Table 11 for the detailed meta prompt used to generate ten different user prompts). After that, the answer format prompts (e.g., At the end, present your answer in <answer>yes</ /nsswrr> or Sanswernn<<ansseer
are added to those generated user prompts.
Note that the examples of the generated user prompts for each target task are provided in Table 12.
Table II: Meta Prompt for generating ten user prompts for the unseen generalization scenario.
gave a friend an instruction and inputs. The friend read the instruction and wrote an output for every one of the inputs. Here the input-output pairs:
(examples)
Based on the above input-output pairs, write an instruction. The new instruction should be wrapped with <histuctioo><//ssuutttiii Tags.
Table 12: User Prompts for Unseen Generalization Experiments, where we sample three among ten for each task.
Target Tasks
User Prompts
For cach medical scenario provided below, choose the most appropriate answer from the options given. Your responses should reflect the best understanding of medical knowledge and relevant anatomy or pathology.
Anatomy
Given a set of medical-related questions and multiple-choice options, select the correct answer for each question based on your knowledge.
Given medical question
multiple-choic options, select the correct answer based on your knowledge medicine and anatomy.
Given medical question with multiple-choice answers, select the correct answer from the options provided.
For each medical developmental question provided, choose the most ap- propriate answer from the given options, based on your knowledge of pediatric medicine and developmental milestones.
Pediatrics
Please analyze following medical-related inputs and select the most ap- propriate answer from the given options for each one, providing the corresponding output for each input scenario.
Based the following inputs and their corresponding options, select the most appropriate answer from the given options.
For each of the following questions, select the correct answer from the pro- vided options and indicate your choice clearly.
Dental
Please provide the correct output each input based on the given options. Select the most appropriate answer from the provided choices for each question.
19

---

Table 12: User Prompts for Unseen Generalization Experiments, where we sample three among ten for each task.
Target Tasks
User Prompts
Please analyze the following medical scenarios and select the most appropriate answer from the provided options for each question.
Based on the following medical questions and their corresponding options, provide the correct answer for each question as indicated by the correct output given. Please ensure that your answers are consistent with established medical knowledge.
Surgery
Given a medical scenario or question along with a set of options, select the most appropriate answer from the options provided.
Based on the provided input-ouiput pairs, plcase assign a score from 1 to 5 for cach product review, where indicates very negative experience, 5 very positive experience, and scores in between indicate varying levels of satisfaction. Consider factors such as the reviewer's 's overall sentiment, the thoroughness of their feedback, and any specific positives or negatives mentioned in the texts.
Please rate the quality satisfaction of the product service described in each input on scale from to 5, where indicates very low satisfaction, 3 indicates moderate satisfaction, and 5 indicates very high satisfaction. Provide a brief explanation for your rating based on the content of the title and text.
Electronics
Based the provided product titles and associated text descriptions, as- sign a rating from to where indicates poor product experience and indicates an excellent product experience. Consider the sentiment expressed in the text, the clarity of the title, and how well the product meets the expectations set by the title and description. Be consistent in your rating based on these factors.
For each input, analyze title and text of the review and assign a rating from to 5 based on the sentiment expressed in the review. A rating I indicates a very negative sentiment, 3 indicates neutral sentiment, and 5 indicates very positive sentiment. Provide the rating output.
Analyze the provided input-title text, then assign a rating from to based on the overall quality and satisfaction expressed in the content, where indicates very poor satisfaction, 3 indicates average satisfaction, and 5 indicates very high satisfaction.
Pet
Given a title and text review of a product, assign a rating from 1 to 5 based on the sentiment expressed in the review, where indicates a negative sentiment, 5 indicates very positive sentiment, and ratings in between reflect varying degrees of positivity.
Given product review that includes title and text, rate the overall satisfaction of the review on scale from to 5, where indicates very low satisfaction and 5 indicates very high satisfaction. Provide rating based on the clarity, positivity, and specifics of the feedback presented in the review.
Evaluate the provided product reviews and assign a rating from 1 to 5 based on the overall sentiment expressed in the review, where I indicates negative experience and 5 indicates highly positive experience. Provide ratings that accurately reflect the review content.
Sports
Based on the given title and text, evaluate the overall sentiment and quality expressed in the reviews. Assign rating from to 5. where indicates very negative experience, and 5 indicates very positive experience. Consider factors such as product performance, satisfaction level, and any issues mentioned the text.
20

---

Table 12: User Prompts for Unseen Generalization Experiments, where we sample three among ten for each task.
Target Tasks
User Prompts
Provide the total count of specific categories of objects, fruits, musical instruments, animals, vegetables listed in the given input question.
Object Counting
Count total number objects or items listed each question provided.
Count the total number of distinct items based on the provided categories and specified quantities. Return the total the output.
Identify relation between following premises and hypotheses, choosing from the options 'entailment" or 'non-entailment.
For the given premises and hypotheses, identify the relation between them by determining if the hypothesis necessarily follows from the premise. Choose 'entailment' if the hypothesis is direct conclusion of the premise 'non-entailmen' if it does not necessarily follow. Provide the relation as your output.
Epistemic
Given premise and hypothesis, determine whether the hypothesis be logically concluded from the premise. Choose the options 'entailment" the hypothesis necessarily follows from the premise, or non-entailment" if it does not.
For cach provided input, analyze the arrangement of objects and their colors, then answer the question based on the specific characteristies and positions of the items mentioned.
Analyze the arrangement of colored items in each scenario and answer the questions by counting. identifying colors, evaluating the presence of specific items based on the provided inputs.
Reasoning Colored Objects
Please answer the following questions based on the descriptions of items and their attributes provided. Provide clear and accurate outputs based on the information given in the inputs.
Based the dialogue snippets provided, please identify whether Assistant's response to the Human's queries promotes harmful, illegal, or disrespectful behavior, Respond with "yes" if the Assistant's 's answer promotes such behavior, and 'no" if it does not.
Analyze following onput- pairs and classify the assistant's responses as "yes" or 'no' based on whether the assistant's replies encourage harmful or illegal actions. Provide brief justification for each classification based on the context of the dialogue.
Anthropic Harmless
Based on the given dialogues, determine whether the assistant's responses are appropriate and responsible in relation to the human's 's inquiries. If the assistant provides helpful, ethical, supportive answer, mark the output "yes'. If the assistant's response encourages harmful, illegal, unethical behavior, mark the output as 'no".
Determine whether the given input expresses negative sentiment towards a group or concept. Output 'yes" it does; otherwise, output 'no'.
Determinc whether the input statement expresses hate speech, derogatory language, or violent sentiments. Respond with 'yes' for statements that and 'no' for those that do not.
Ethos
Determine whether the input expresses harmful, derogatory, vio- lent sentiment, Reply with 'yes' if does, and 'no' if does not.
21

---

Table 12: User Prompts for Unseen Generalization Experiments, where we sample three among ten for each task.
Target Tasks
User Prompts
Provide concise and accurate answer to the question based on the given context, ensuring that the response directly addresses the inquiry.
Given a context about a specific topic, provide the name a related char- acter, actor, or relevant detail mentioned in the text when prompted with specific question related to that context.
Natural Questions
Given context that includes related information, answer question that follows with specific and concise response based on the details provided the context.
Provide concise answer to question based on the context provided, ensuring that the output is relevant and directly related to the question asked.
Given context passage. summarize the key information related to the spe- cific question asked, providing clear and concise answer based the content of the context.
Web Questions
Based the provided context, answer the question specifically and suc- cinctly by extracting the relevant information from the context. If the information cannot be found, provide response indicating absence that information.
22

---

B Additional Experimental Results
B.1 Unseen Generalization Results with Standard Deviations
We report standard deviations of SPRIG and MctaSPO in the Unseen Generalization setup in Table 13.
Table 13: Unseen Generalization Results with standard deviations over three different runs.
Bold numbers indicate the statistically significant results based t-fest (p 0.05).
Domain
Medical
Review Analysis
Target Task
Anatomy
Pediatries
Dental
Surgery
Electronics
Pet
Sport
Default
36.1
25.8
32.3
41.3
29.3
CoT
36.1
42.7
26.0
32.0
36.8
40.3
25.0
Global
Service
34.4
35.2
20.2
30.6
59.0
53.2
52.2
SPRIG
41.6
42.2
0.5 0.5
28.4
35.7
0.9
47.9 3.5
47.4 2.4
38.6 3.5
MetaSPO
45.7 4.7
43.1
2.4
3.5
36.3
3.8
2.1
66.0
0.8
61.4 1.9
Domain
SPRIG
41.2:12.2
11.8 1.5
20.6
+
35.3 1.1
61.6 0.9
57.4 1 0.6
51.3 2.2
MetaSPO
48.9 2.0
36.4
40.0 1.6
64.9
2.5
61.5 1.8
Domain
Reasoning
Safety
Grounding
Target Tisk
Count
Epistemic
Color Obj.
A harnless
Bihos
N.Q.
WebQA
Default
43.5
28.3
56.6
21.2
28.7
15.1
11.6
CoT
45.6
37.2
62.0
21.9
31.9
15.9
12.0
Global
Service
30.6
56.6
21.1
26.9
9.9
SPRIG
39.3 1.6
29.9
59.9
23.0 1.0
31.1
0.9
1..2
MetaSPO
44.5 +: 1.2
39.6 3.9
64.5 + 1.0
24.9 + 0.6
37.6 0.7
9.5 0.4
7.7 1.3
SPRIG
30.1
34.5 1.4
51.5 3.4
24.0
32.1
1..1
12.0
Domain
MetaSPO
47.1 + 0.7
43.0 + 0.5
66.6 +t 1.3
29.1 + 2.8
43.9 +: 0.4
19.1 3.7
13.7 + 0.06
B.2 Performance of MetaSPO at Each Iteration
0.02
0.00
Aug. Metric
0.00
Aug. Metric
0.35
0.50
0.35
MaexSPo Debaat
0.45
MetaSro Detrut
Service
0.39
0.30
Ieration
Ieerttion
a Medical
(b) Review Analysis
0.30
0.34
Arg. Metric
0.15
Avg.
Avg.
0.13
0.28
0.26
0.13
Itematonn
lteration
Ieeraion
(c) Reasoning
(d) Safety
(e) Grounding
Figure 10: Unseen Generalization Performance of MetaSPO for each iteration.
We report the Unseen Generalization performance of the optimized system prompt function of the number of iterations for each domain in Figure 10. From this, we observe that the performance improves steadily across iterations, with significant gains observed up iteration 2, which seems saturated after it, suggesting diminishing returns for further iterations.
23

---

B.3
Comparison of Computational Costs
Table 14: Comparison of the number of model requests (or calls) to operationalize optimizer to implement MetaSPO, paraphraser to SPRIG, and base model to both approaches.
Optimizer Model Call
Paruphraser Call
Base Model Call
SPRIG MctaSPO (Ours)
N/A
300
140k
126
N/A
18k
In Table 14, we compare the computational costs of system prompt optimization approach against SPRIG on four source tasks, based on the number of model requests (or calls) used to operationalize them. Specifically, unlike SPRIG designed generate a variety of candidate prompts with several paraphraser calls and evaluate them (on the training samples) with the base model calls, MetaSPO performs optimization by analyzing the incorrectly predicted examples with the last (few) prompts and updating the prompts based on them. As a result, it ultimately yields far less number of base model calls, while the resulting optimized prompt can be generalizable diverse tasks (See Figure 6).
B.4 Relative Performance Gains of MetaSPO over Default with Various User Prompts.
0.6
Metric w/ MctaSPO
Metric wi MetasPP
0.2
Win Rate: : 91.1%
0.0
0.2
644
S.
dx
Metric Default
Metric Default
(a) Medical
(b) Review Analysis
Motate wi MotaSPO
3.5
Mesrie wo MetaSPO
0.4
Motrie w/ MetaSPO
0.2
0.0
Rate 83.6%
0.
Win Rate 98.3%
0.0
Win Rate 65.0%
02
0.5
0.0
0.4
Metric Default
Matric Defnult
Matric Default
(c) Reasoning
(d) Safety
(c) Grounding
Figure 11: Performance of various user prompts with the system prompt from Default (x) and MetaSPO (y) for each domain. Points above indicate the superiority of MetaSPO.
We provide a detailed visualization of user prompts for each domain to illustrate the performance improvements achieved by MetaSPO over the Default system prompt. The visualization reflects the performance of various user prompts across the Medical, Review Analysis, Reasoning, Safety, and Grounding domains. Also, the size of each point represents the density of overlapping prompts with similar performance scores. Then, as shown in Figure 11, in the Medical domain, 83.3% of user prompts perform better with MetaSPO. Also, Review Analysis, Reasoning, Safety, and Grounding domains achieve success rates of 91.1%, 85.6%, 98.3%, and 65. 5%,0 respectively. This highlights the consistent effectiveness of MetaSPO across diverse user prompts and domains.
24

---

B.5 Analysis of Source-Target Target Task Similarity with Embedding-Level Cosine Similarity
Pearson Correlation 0.29
Spo
Improvement (%)
100
60
Epi.
Bth.
Pgt
D
40
Agi
Aua.
Col
Pod.
Web
Sur.
Cou.
0.3
0.1
0.5
0.6
0.7
0.8
Avg. Source-Target Similarity (Embedding Cosine Similarity)
Figure 12: Relative improvements of MetaSPO against Default as function of the similarity between source and target tasks, where the similarity is measured by the embedding- -eveel cosine similarity from Song et [33].
As extension the results in Figure which Bag-of-Words rank correlation measure the lexical similarity between source and target tasks, we further conduct an additional analysis, measuring the semantic similarity between them. Specifically, using MPNet [33], average their embeddings to obtain a representative vector for each task, and then compute the cosine similarity between vectors across tasks. The results presented Figure 12 show positive correlation between source-targel similarity and improvement, with a Pearson correlation coefficient of 0.29. This result further strengthens our hypothesis that the greater similarity between source and target tasks can enhance the impact of the optimized system prompt.
encode examples for each task
B.6 Results with Different Optimizer LLMs
Table 15: Results of MetaSPO with varying optimization models, where the base model for answer generation is fixed to Llama 3.2 (3B). Default and SPRIG are included as baselines for comparison.
Methods
Optimizer LLMS
Review.
Reasoning
Avg.
Default
37.4
42.8
40.1
SPRIG
56.8
38.7
47.7
MetaSPO
Llama 3.1 (8B)
59.9
45.7
52.8
MctaSPO
Llama 3.1 (70B)
64.2
47.9
56.1
MetaSPO
GPT-4o mini
62.7
52.2
57.5
MctaSPO
GPT-4o
63.7
53.2
58.4
To assess the robustness of MetaSPO across different optimizer LLMs, we the base model (for answer generation) to Llama3.2 (3B) and conduct experiments by varying the LLMs for prompt optimization. The experiments are performed on Review Analysis and Reasoning domains, and results are averaged over each domain. As shown in Table 15, MctaSPO consistently outperforms baselines regardless of the choice of optimizer LLMs. Also, when using larger optimizer models, such GPT-4o, MetaSPO demonstrates strong performance, suggesting its potential to achieve better results when combined with more advanced LLMs.
25

---

B.7 Additional Results of MetaSPO on Out-of-Domail Scenario
0.5
Avg. Metric on Reasoning
0.4
0.3
0.2
0.1
Default
ProTeCl
CoT
SPRUG
Service
MetaSPO
0.0
Medical
Safety
Reasoning
Source Domain
Figure 13: Performance of different prompt optimization methods the Reasoning domain using prompts trained on different source domains.
We compare MetaSPO with ProTeGi (a user prompt optimization method specific the target task) in challenging cross-domain scenario. this setting, prompts are optimized on a source domain and evaluated the unseen Reasoning domain without any further adaptation. As Figure 13 shows, MetaSPO consistently outperforms other methods (including ProTeGi) across various source domains, demonstrating superior robustness and generalization capabilities. Notably, both ProTeGi and SPRIG exhibit limited performance, performing worse than the Default system prompt even in the case where the source and target domains identical. This indicates that they optimize the prompts that are tailored to specific tasks and struggle to transfer beyond their training tasks. contrast, MetaSPO effectively optimizes the system prompts with strong cross-domain and cross-task generalization.
B.8 Effect of Scaling the Number of Source Tasks
0.64
Avg. Metric
0.62
0.60
0.58
0.54
Review Analysis
Number Source Tasks
Figure 14: Results with varying the number of source tasks for system prompt optimization on MetaSPO, in the range of to 20.
To extend the experiment described in Figure 7, we increase the number of source tasks 20. As shown in Figure 14, while MetaSPO benefits from an increasing number of source tasks, its performance saturates after utilizing six. This suggests that once MetaSPO has sufficiently learned the context and characteristics of the source domains, adding more tasks offers no additional benefit.
26

---

B.9 Combined Optimization Method in MetaSPO
Table 16: Results with different combinations of prompt optimization techniques in MetaSPO.
Optimization Methods
Domains
System
User
Medical
Review.
Reasoning
Avg.
APE
APE
39.7
60.1
48.0
49.3
APE
ProTeGi
40.2
58.9
48.9
49.3
ProTeGi
APE
41.0
62.5
53.3
52.3
ProTeGi
ProTeGi
43.0
62.7
52.2
52.6
To demonstrate the versatility of MetaSPO, we further conduct additional experiments using different optimization methods for the system and prompts. The results in Table 16 show that remains robust across all combinations, while achieving the best performance when using the most effective prompt optimization method (ProTeGii) for both the system and user prompts.
B.10
Effect of Varying the Number of Wrong Examples on System Prompt Optimization
0.44
0.60
0.42
Avg. Metric
Avg. Metric
0.38
0.50
MetaSPO
MetaSPO
Default
Decaul:
0.34
Col
0.40
Col
Service SPRIG
Service
0.35
SPRG
0.30
Number Wrong Examples
Number of Wrong Examples
(a) Medical
(b) Review Analysis
Figure 15: Result with varying the number of wrong examples for system prompt optimization.
To analyze how the number of incorrect examples affects system prompt optimization, we conduct an experiment by varying the number of examples and report the results Figure 15. For the Medical domain, single example is enough, but for the Review Analysis domain, two examples per task are necessary to significantly improve the system prompt, probably due to subjective expressions within it that require multiple cases to analyze failures.
B.11 Analysis of Concise System Prompt
Table 17: Result of short and concise system prompt.
Medical
Review.
Reasoning
Safety
Grounding
Avg.
Default
33.3
37.4
42.8
25.0
13.4
30.3
MetaSPO-Concise
37.9
56.8
50.8
30.4
12.8
37.7
MetaSPO-Bait 43.0
62.7
52.2
36.5
16.4
42.2
To investigate whether summarized system prompts can serve as effective alternatives, we conduct experiments with more concise system prompts (summarized from the optimized system prompts with GPT-4o-mini), and report their performance in Table 17. From this, we observe a trade-off between brevity and performance: the more detailed system prompts consistently achieve better performance, suggesting that detailed guidance in the prompt allows the model to respond accurately.
27

---

Qualitative Results
C.1
Optimized System Prompts
We provide the optimized system prompts by MetaSPO for each domain Table 18, including the Global setting.
Table 18: Optimized system prompts for each domain and the global setting.
Domains
Optimized System Prompts
You are knowledgeable and analytical assistant specializing in medical topics. Your task is to accurately respond to medical inquiries utilizing established medical knowledge, guidelines, evidence-based reasoning. When presented with a question, carefully analyze the options provided and select the most appropriate answer. Ensure that your responses are clear, concise, and well-structured, including a rationale that explains your reasoning and cites relevant medical principles. Prioritize accuracy and logical coherence in all your responses.
Medical
You are versatile language model tasked with analyzing customer reviews to accurately predict product ratings based on the emotional tone and sentiments expressed. Follow these guidelines for effective evaluation:
Identify Emotional Indicators: Focus strong emotional expressions within the reviews, such as phrases indicating clear satisfaction ("I love it") and dissatisfaction ("terrible experience"). Emphasize these surface-level sentiments as they are essential for quick assessment.
Assess Overall Sentiment: When reviews present mixed sentiments, evaluate the overall emotion conveyed. Determine whether positive sentiments dominate over negative ones, and give more weight to dominant sentiments unless significant negative factors are expressed. Small complaints should generally not overshadow strong positive emotions.
Understand Context: Consider context which a review is written. Recognize that certain statements may have varying implications based on the product's 's purpose and the reviewer's expectations. Adapt your weighting of sentiments accordingly.
Review Analysis
Utilize the Rating Scale: Assign ratings on scale from to based on overall sentiment:
5: Exceptional experience with strong positive sentiments.
4: Generally positive with minor negative mentions.
3: Neutral with balance of positive and negative sentiments.
Predominantly negative experiences with some redeeming qualities.
1: Very poor experience with overwhelmingly negative sentiments.
5. Output Format:
Present your predicted rating clearly follows:
<answer>(rating (rating)<//aaeeeteee This will maintain consistency and clarity in your re- sponses.
6. Provide Rationale: When offering predictions. include brief explanation of how you arrived at the rating. This will help clarify your reasoning and enhance the trust in your assessment.
By following these structured guidelines, you will generate more accurate and contex- tually relevant predictions that reflect customer satisfaction and experiences.
You are logical reasoning assistant. Your primary objective is analyze and process in- formation critically. Focus on understanding the context of events and the implications of sequential developments while engaging in deductive reasoning. Always strive to provide clear and well-structured answers, ensuring that responses are appropriately formatted and include necessary tags. When faced with complex inquiries, meticulously evaluate relationships between facts and provide comprehensive, logical conclusions based on the clues provided.
Reasoning
28

---

Table 18: Optimized system prompts for each domain and the global setting.
Domains
Optimized System Prompts
You analytical assistant. Your task evaluate statements, questions, and objections based on contextual understanding, evidence, relevant criteria. When providing responses, please adhere to the following guidelines:
1.
Factual Accuracy: Assess the factual correctness and relevance of the statement in question. Provide context where necessary.
Opinion vs. Fact: Clearly differentiate hetween opinion-hased claims and factual assertions. Explain why statement is considered an opinion or fact.
Emotional Tone Recognition: Identify analyze emotional tones, especially in cases involving sarcasm, hate speech, or any emotionally charged language. Discuss the implications of tone in your assessment.
Safety
Balanced Perspective: Explore multiple sides of argument when applicable, Of- fer a well-rounded analysis that considers contrasting viewpoints and broader implications.
5. Clarity and Structure: Format your final answer using <answer>yes</ aaswer>>>>>>>> or Kansver>no</answer> followed by comprehensive explanation that includes reasoning. evidence, and relevant context.
By applying these guidelines, you will provide contextually aware, nuanced, and accu- rate evaluations in your responses.
You are advanced assistant designed deliver direct and concise answers tailored user inquiries. Focus on providing specific information that directly addresses the question, using keywords or short phrases as your primary response format. Limit additional explanations to cases where further clarification is explicitly requested. Prioritize accuracy and relevance, ensuring that your responses while maintaining clarity and precision.
Grounding
strongly aligned with the context provided. Aim for minimalism
You are an advanced virtual assistant designed to process and analyze information across broad range of topies. Your main objectives are to:
Analyze Information Critically: Assess the provided data, considering various per- spectives and implications. Use logical reasoning to derive conclusions and make connections between concepts.
Handle Different Types of Queries: Be prepared to respond factual questions, perform sentiment analysis, and engage in logical deductions. Understand the nuances of query type and apply appropriate reasoning.
Evaluate Sentiment and Emotional Tone: When dealing with reviews sentiments. accurately reflect the emotional tone consider both positive and negative elements and provide clear overall assessment.
Global
4. Provide Clear and Struetured Responses: mat, making it easy for precision in your communication.
Organize your answers in coherent for- understand your reasoning and conclusions. Aim for clarity and
5. Adapt Context: Adjust your analysis based on the specific context and details given in each inquiry. Pay attention to nuances and subtleties that may affect the outcome of your assessment.
By following these guidelines, you will hetter serve as a helpful assistant, enabling users to receive accurate, relevant, and thoughtful responses their diverse queries.
29

---

C.2 Example of Failure Analysis and Prompt Generation in MetaSPO
We provide an example of failure analysis and prompt generation based on the analyzed problem within the reasoning domain, as shown in Figure 16 and Figure 17, respectively.
Failure Analysis Prompt
system prompt tasked improving langage mode''s sysiem prompt general Your gual is to malyye why the curmet system prompt fails to respond correctly in the given eramples. Follow these instuuiions carefull:
eee Review the current system prompt: helgful
06e W conn responses:
Wrong Example
System prompl: You helpCul assisiant.
User prompi: A lask requiring determining the finel positions of set of abjects given their initial positions and a description of seguence of swaps.
<Q <Quistion>> lioe, Roh, Claim, Dave, and Eve are playing game. At the start of the game, they are each holding ball: Alice has ytllow bell, Boh has blue ball, Claire has pick ball, Dave while ball, and Eve has red ball.
As the game progresses, paits of players trade bells, First, Clatre and Alie swap bails. Thea, Eve and ob swap balls. Then, and Dave swap balls. Then, Claire and Eve swap balls, Pinally, Alice and Dave swap balls. At the end of the game, Alice has the Select the correct optin: A. yellow
pink ball. while At end show the answer opton bracketed between <answer> and <hanswers
ball.
<>Quueston>>
Response: determine final position lioels hall, let's analyze the sequence swaps: Inibally, A hos has yellow ball.
Claire
swap halls; now a
Clair yellow hall.
Eve and Bob swap balls: Ere now has bloe ball, and Bob has a red ball.
Rob and Dave swap hallss Rob
white ball, and Dowe has bloe halll.
Claire and Eve swap balls: Clair now has blue ball, and Eve has a pink ball.
lice and Dave swop halls: A now has blue ball,, and Dave has a yellow boll.
Therefore, the end game,
<answer=8 slanswer>
Prediction B
The correct label is:
Wrong Example
Wrong Example
adè Remember to focus solely on discussing and impnving the system prompt.
#se Wrap the analy sis of system prompt in staallsts>-ii nalysiss nalpssss tags.
Analyzed Problem
The currett system prompt, "You are belpful assistant" too vague lacks spectficity about the types tasks or reasoning required. This beund appronch lead isees complex analytical tekks, shown in the provided examples. Here's it's Gilling:
1. Clartty and Specificity: The tasks presected often require multi-stap reasoning and explicit tracking of information, and the curennt prompt does not guide the model to employ these spocific mosoning stratagies. By spasifying that should follow logical messonnng paths, avoid assumptions, and check contradiciory stalemenis, the model can msily make emoos its logic.
statements, system prumpt penvide any indication of useding to construct logical connections belween pieces of information, leading la incorrect conclusious It could explicilly instruct the model to analyse, deduce, arrange information based given constraints.
Specific Guidance: Each example requires deducing
Focus Areas for Improv emeett
assisdant needs andlyze relationships It could mestion the importance of checking for coutradicions and confieming the seguenoelorder based on the clues. The prompt should encourage detailed step-by step reasoning to ensure that any given logical deductions accurately tollowed.
The prompt should indicate
provided information sincctue way.
Example Contectuallzztio:: By tncluding context on handling diffeceot of tasks, such logical deduction, ordecing, and item categorization based on critera, the model can botter quided to produce more accurate responses.
Figure 16: Example of failure analysis prompt and analyzed problem.
30

---

Prompt Generation Prompt
You are a system prompt writer tasked with improving langugg model's system prompt. Your goal is to write a beller system prompi that can be gemeralizee for varioss losks. Follow these instructions caretully:
000 Review the current sy system prompt:
You are helpful assistant.
nalysis
system prompt:
Analyzed Problem
current system pronpt, "You helpful assistant." is vagne and locks specificity types of tasks reasoning required. broad approach may lead iseaes in more complex anolytical tasks, shown the provided examples, Here's why it's fiiling
#* B ased the Information provided, wrte improved system prompt
#*e te new system prompt should be wragped with <impoved, sy stemm wroppt--mmpreee system, prompt> tags
Improved System Prompt
You logical rereoning assistant, Your mlee anolyze, deduce, and organize information based provided clues and contect. For each task, follow these guidelines:
Stop by. Step nalysis: Bresk idealliy relationships and implications.
information manageshle parts. Carefully evaluste
Logical Deductions. A paly logical ressoning to make infermsees and draw concuusions Ensure that your cooduusion are besed solely on the Information given and avoid making unventfied assumptions.
Check for Contradiclions Continuuusly vecity the consistency of the Information. If contradictons arise, addrress them by reev reevaluating the clues and cartfying any misunderstadding.
Present Clear Conclusions: A.ter your analysis, summarize findings and explectily state your conclusions. Ensure that reasoning transporent supported by information provided.
appenectt your ability complex problems elficienly accurstely
Figure 17: Example of prompt generation prompt and improved system prompt.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly present the main contribution: introducing the bilevel system prompt optimization problem and a meta-learning framework.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of MetaSPO in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
32

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We detail the experimental setup in Section 4.1 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer:
[Yes]
Justification:
We have submitted our code for the experiments as supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
mublic/guides Btte://uidee//oduuunienee Cowes//iseeuuubiiiiioeeeeeaaaaie sewssssssoooooooos onioliccyy ooiicylyyy))o for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /ttpps.ccppuulll////
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the data splits, hyperparameters, meta-promp: to implement MetaSPO and others in Section 4.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report unseen generalization results with standard deviation and provide statistical significance via t-test results Appendix B.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
34

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Experiments were run on an NVIDIA A5000 GPU as provided Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //teer//ee.s.eeeeeee http://rrrrpp..ppppciiieeee / ines?
Answer: [Yes]
Justification:
Our research fully complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release any models or datasets that pose high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly credit the original owners of assets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the datasettcodeemmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve human subjects the use of crowdsourcing: therefore, considerations regarding participant risk, informed consent, or IRB approval are not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
37

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core methodology of this research does not involve the use of large language models (LLMs).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: :/ //tp///nss.////// theurrpp.cc/Coofeeeeeee wee.meeeeneeee2///////// /LLM/LLM for what should or should be described.
38


Reviews:
[{"review_id": "9YYtHSfXg2", "reviewer": "Reviewer_Vnef", "summary": "The paper targets the optimization of system prompt and designs a meta-learning framework to optimize system and user prompts alternatively. Extensive experiment results demonstrate the effectiveness of the proposed framework. The optimized system prompt generalizes well to unseen tasks and enables faster adaptation of user prompts.", "questions": "See the limitation section.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1: The experiments are very comprehensive in this paper, and the results look strong.  \nS2: The writing is clear and easy to follow.  \nS3: The motivation of optimizing system prompts with awareness of user prompts is clear and intuitive.\n\n---\nW1: It would be great if the authors could provide some qualitative insights about the generalizable knowledge in the optimized system prompts, as mentioned in Line 259.  \nW2: Currently, the experiments are limited to using Llama-3.2-3B as the generating model and GPT-4o-mini as the prompt optimizer. I recommend that authors expand the scope by testing the framework on larger LLMs and leveraging open-sourced LLMs for prompt optimization.  \nW3: It would be great to break Section 4.2 into several subsections. It is currently too long.", "was_revised": true, "final_justification": "The issues raised in my reviews have been resolved in the rebuttal. Therefore, I vote for the acceptance of this paper.", "rebuttal": "We thank you very much for your thoughtful comments. We are grateful that you found our motivation for system prompt optimization with awareness of user prompts is clear and intuitive, our experiments are very comprehensive with strong results, and the writing is clear. In this response, we have made every effort to address all your comments faithfully.\n\n---\n\n> W1. It would be great if the authors could provide some qualitative insights about the generalizable knowledge in the optimized system prompts, as mentioned in Line 259.\n\nThis is a very relevant proposition, and in fact, we present qualitative results about the generalizable knowledge in the optimized system prompts in Table 18. Specifically, the system prompts optimized consistently include a structured format that includes a clear role definition (e.g., you are a knowledgeable and analytical assistant designed to process and analyze information across a broad range of topics) and abstract task framing (e.g., your primary objective is to analyze and process information critically), which can be transferable across various tasks. We believe that these generalizable patterns emerge from optimization over multiple tasks via meta-learning, and the results of cross-domain and cross-model experiments in Figure 5 and Table 3 further support this. We will highlight it in the revision.\n\n---\n\n> W2. The experiments are limited to using Llama 3.2 (3B) as the generating model and GPT-4o mini as the prompt optimizer. I recommend that authors expand the scope by testing the framework on larger LLMs and leveraging open-sourced LLMs for prompt optimization.\n\nWe would like to point out that, in addition to Llama 3.2 (3B), we also present experiment results with the stronger generating models in Table 3, including Llama (8B), Qwen 2.5 (7B), and GPT-4o mini, from which we observe that MetaSPO consistently outperforms baselines, confirming its generalizability and robustness to various models. However, to more fully address your suggestion, we additionally conduct experiments with GPT-4o, evaluating the system prompt optimized from GPT-4o mini to GPT-4o without further optimization, and as shown in Table A.1 below, MetaSPO achieves the highest average score over baselines, indicating its strong transferability to even the strongest model. Lastly, we would like to refer to the results in Table 15, showing that MetaSPO is compatible with open-source models, such as Llama (8B) and Llama (70B), when they were used as prompt optimizers.\n\nTable A.1. Unseen generalization results of GPT-4o with system prompts optimized from GPT-4o mini\n| 　 | Review | Analysis |  |  | Reasoning |  |  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 　 | Electronics | Pet | Sports | Count | Epistemic | Color | Avg. |\n| Default | 73.4 | 75.0 | 69.2 | 78.2 | 81.6 | 98.0 | 79.2 |\n| CoT | 73.0 | 76.0 | 70.6 | 80.0 | 80.0 | 98.4 | 79.7 |\n| Service | 70.6 | 75.0 | 66.8 | 82.0 | 81.6 | 98.0 | 79.0 |\n| SPRIG | 73.0 | 73.8 | 68.0 | 81.8 | 81.8 | 98.0 | 79.4 |\n| MetaSPO | 76.9 | 75.4 | 77.3 | 82.9 | 81.1 | 98.9 | **82.1** |\n\n---\n\n> W3. It would be great to break Section 4.2 into several subsections. It is currently too long.\n\nThank you for your valuable suggestion, and we apologize for the inconvenience. We will split it into multiple subsections (for example, Unseen Generalization, Test-Time Adaptation, and Additional Analysis) to improve the quality and clarity of the presentation in the revision."}, {"review_id": "KENlt95rEa", "reviewer": "Reviewer_bNQ9", "summary": "This paper presents a bilevel learning framework for optimizing the system prompt to enhance LLM performance. The authors decompose the input into three components: a task-agnostic system prompt, a task-specific user prompt, and a specific query. They optimize the system prompt across diverse user prompts and tasks, while also iteratively refining user prompts—using the optimized user prompts to further improve the system prompt. The proposed method is evaluated through extensive experiments on 14 unseen datasets spanning 5 different tasks.", "questions": "1. Do the authors conduct any case studies to illustrate how the model's output changes from incorrect to correct? For example, is the improvement mainly driven by format-related instructions in the optimized system prompt, as suggested by its inclusion of output format in Appendix?  \n2. Have the authors tried applying their method to more capable models (e.g., GPT-4o-latest or o3) or more challenging tasks (e.g., AIME 2024 or LiveBench)?  \n3. As mentioned in Line 240, the authors use ten user prompts per task. Could the authors provide detailed results using only these ten user prompts (e.g., in the form of a table similar to Table 1) to isolate the impact of system prompt optimization, beyond what is shown in Figure 3?  \n4. In Figure 6, why does the performance of the default system prompt vary across different iterations and example counts?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper is motivated by a reasonable and important premise: optimizing the often-overlooked system prompt to improve LLM performance.  \n2. The experiments are comprehensive, covering a variety of evaluation settings with detailed analyses supporting the proposed method’s efficacy.\n\nWeaknesses  \n1. The paper lacks theoretical insights into why meta-learning is effective for system prompt optimization and how it enables performance improvements.  \n2. The tasks included are limited to deterministic problems. The framework’s effectiveness on subjective tasks—such as creative writing and open-ended dialogue—remains unclear, which is notable given that system prompts are used in such chat scenarios (e.g., ChatGPT).", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive and helpful comments, as well as your recognition of the strengths of our work, including our motivation to optimize overlooked system prompts and our comprehensive experiments, supporting the efficacy of our method. We have carefully considered and faithfully responded to your comments. \n\n---\n\n> W1. The paper lacks theoretical insights into why meta-learning is effective for system prompt optimization and how it enables performance improvements.\n\nWe appreciate your feedback regarding the desire for a deeper theoretical understanding. However, as you acknowledged as strengths of our work, we provide comprehensive experiment results with detailed analyses, which support all the claims that we make: the system prompt learned from meta-learning improves generalization and accelerates adaptation. Specifically, we show consistent performance improvements across 14 unseen tasks in 5 domains, including cross-domain and cross-model generalization (Table 3 and Figure 5), demonstrating that meta-learning enables generalization over diverse domains and models. Also, the optimized system prompt via meta-learning enables the rapid and scalable adaptation, requiring fewer iterations and data at test time (Figure 6). Therefore, while a formal theoretical analysis is indeed an interesting direction for future work, we strongly believe that the lack of such insight should not be viewed as a limitation, given that our empirical evidence sufficiently provides concrete support for the efficacy of our method.\n\n---\n\n> W2. The tasks included are limited to deterministic problems; thus, the effectiveness of the framework on subjective tasks remains unclear (while system prompts are also used in such chat scenarios).\n\nThank you for raising this concern, and to address this, we evaluate the efficacy of the global system prompt optimized by MetaSPO on the MT-Bench dataset [1], which is designed for multi-turn, open-ended subjective tasks, such as Writing, Humanities, and Roleplay. Then, as shown in Table B.1 below, we observe that MetaSPO consistently outperforms baselines (including SPRIG and CoT) across tasks, with GPT-4o mini for evaluation. In other words, these findings provide strong empirical evidence that MetaSPO can apply to a diverse range of tasks, including deterministic and subjective tasks. \n\nTable B.1. Result of MetaSPO on MT-Bench [1]\n|Base Model|Method|Humanities|Roleplay|Writing|Avg.|\n|---|---|:---:|:---:|:---:|:---:|\n|Llama3.2 (3B)|Default|7.65|7.60|7.55|7.60|\n||CoT|**7.70**|7.60|7.50|7.60|\n||Service|7.65|6.75|7.60|7.33|\n||SPRIG|7.60|7.30|7.60|7.50|\n||MetaSPO|**7.70**|**7.65**|**7.85**|**7.73**|\n|Llama3.1 (8B)|Default|8.20|7.50|**7.75**|7.82|\n||CoT|8.20|7.90|7.70|7.93|\n||Service|7.80|7.75|7.65|7.73|\n||SPRIG|**8.25**|7.85|7.60|7.90|\n||MetaSPO|8.20|**8.05**|**7.75**|**8.00**|\n\n[1] Judging LLM-as-a-judge with MT-Bench and Chatbot Arena, Neurips D&B Track 2023\n\n---\n\n> Q1. Do the authors conduct any case studies to illustrate how the model output changes from incorrect to correct? Is the improvement mainly driven by format-related instructions in the optimized system prompt, as suggested by its inclusion of output format in Appendix?\n\nThis is an excellent suggestion, and based on it, we conduct additional qualitative case studies provided below, to show how MetaSPO corrects errors. Specifically, in a reasoning task (called Colored Object), the model with the default system prompt yields an incorrect answer (“red”) despite correct formatting, while the model with the system prompt from MetaSPO produces the correct answer (“brown”) by reasoning through the item order. Notably, many optimized system prompts (such as other Medical and Reasoning domain examples in Table 18) do not include format instructions, which confirms that performance gains stem not primarily from output formatting alone, but rather from improved task understanding and reasoning behavior encouraged by the optimized system prompt.\n\n**Case 1: Colored Object (Reasoning)**\n```python\nUser Prompt with question:\nRead the following questions carefully and provide the correct answers based on the details supplied in each input. Ensure your answers accurately reflect the conditions described and count or describe the objects as requested.\n<Question>\nOn the floor, you see several items arranged in a row: a brown paperclip, a mauve necklace, a turquoise dog leash, a red jug, and a teal pen. What is the color of the item furthest from the pen?\n</Question>\nAt the end present your answer in <answer> and </answer>.\n\nGround Truth: Brown\n\nMetaSPO Response(Correct):\nTo determine the color of the item furthest from the pen, we need to consider the arrangement of the items. Since the items are arranged in a row, we can deduce the order from left to right.\nThe items in the order are: \n1. Brown paperclip\n(...)\n5. Teal pen\nThe pen is at the end of the row, so the item furthest from the pen is the brown paperclip.\n<answer> Brown </answer>\n\nDefault Response (Wrong):\n To determine the color of the item furthest from the pen, we need to consider the arrangement of the items. Since the items are arranged in a row, we can count from the pen to the other end.\n1. The pen is at one end.\n2. The teal pen is closest to the pen.\n(...)\n6. The red jug is at the farthest end.\nSo, the item furthest from the pen is the red jug.\n<answer>Red</answer>\n```\n\n**Case 2: Dental (Medical)**\n```python\nUser Prompt with question: \nFor each of the following questions, select the correct answer from the provided options and indicate your choice clearly.\n<Question>\nFunctional appliances in skeletal class II are indicated when:\nOptions:\nA. Maxilla is normal, mandible is retrognathic\nB. Maxilla is prognathic\nC. Mandible is retrognathic in a mature individual\nD. When severe crowding is present\n</Question>\nAt the end present your answer in <answer> and </answer>.\n\nGround Truth: A\n\nMetaSPO Response(Correct):\n<answer> A </answer>\nRationale: Functional appliances in skeletal class II are (...)\n\nDefault Response(Wrong):\n<answer> C </answer>\n```\n\n---\n\n> Q2. Have the authors tried applying their method to more capable models (e.g., GPT-4o-latest or o3) or more challenging tasks (e.g., AIME 2024 or LiveBench)?\n\nIn response to your question, we additionally conduct experiments with GPT-4o, evaluating the system prompt optimized from GPT-4o mini to GPT-4o without further optimization, and as shown in Table B.2 below, MetaSPO achieves the highest average score over baselines, indicating its strong transferability to even one of the strongest models. \n\nAlso, for challenging tasks, while our evaluation already includes multiple challenging reasoning benchmarks, which require multi-step inference and compositional reasoning, we further conduct the suggested experiments on LiveBench [2], evaluating the system prompt optimized for the reasoning domain using GPT-4o mini and reporting its performance on the reasoning category of LiveBench [2]. The results in Table B.3 then show that MetaSPO outperforms baselines, highlighting its effectiveness on complex reasoning scenarios.\n\nTable B.2. Unseen generalization results of GPT-4o with system prompts optimized from GPT-4o mini\n||Review|Analysis|||Reasoning|||\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||Electronics|Pet|Sports|Count|Epistemic|Color|Avg.|\n|Default|73.4|75.0|69.2|78.2|81.6|98.0|79.2|\n|CoT|73.0|76.0|70.6|80.0|80.0|98.4|79.7|\n|Service|70.6|75.0|66.8|82.0|81.6|98.0|79.0|\n|SPRIG|73.0|73.8|68.0|81.8|81.8|98.0|79.4|\n|MetaSPO|76.9|75.4|77.3|82.9|81.1|98.9|**82.1**|\n\nTable B.3. Result of MetaSPO on LiveBench (Nov 25, 2024) [2]\n||LiveBench|\n|:---:|:---:|\n|Default|35.3|\n|CoT|35.8|\n|Service|36.1|\n|SPRIG|31.7|\n|MetaSPO|**39.1**|\n\n[2] LiveBench: A Challenging, Contamination-Limited LLM Benchmark, ICLR 2024\n\n---\n\n> Q3. Could the authors provide detailed results using only ten user prompts (e.g., in the form of a table similar to Table 1) to isolate the impact of system prompt optimization?\n\nThank you for your insightful suggestion. As suggested, we evaluate performance with only the ten user prompts without any system prompt (to isolate its effect), and report the results in the “None” row of Table B.3. From this, we observe that it shows comparable performance with the default system prompt and substantially lower performance than our optimized MetaSPO prompt. We will make sure to include this result in the revision. \n\nTable B.4: Results on the unseen generalization scenario, where None refers to using only the user prompts without any system prompt, compared against Default and MetaSPO.\n|Domain||Medical||||Review|Analysis||Reasoning||Safety||Grounding|||\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Methods|Ana.|Ped.|Den.|Sur.|Ele.|Pet|Spo.|Cou.|Epi.|Col.|A.H.|Eth.|N.Q.|Web.|Avg.|\n|Default|36.1|38.9|25.8|32.3|41.3|41.5|29.3|43.5|28.3|56.6|21.2|28.7|15.1|11.6|32.2|\n|None|40.0|36.1|31.0|33.9|38.5|39.4|27.7|42.0|34.7|59.0|19.1|27.8|13.3|10.5|32.4|\n|MetaSPO|**48.9**|**46.7**|**36.4**|**40.0**|**61.8**|**64.9**|**61.5**|**47.1**|**43.0**|**66.6**|**29.1**|**43.9**|**19.1**|**13.7**|**44.5**|\n\n\n---\n\n> Q4. In Figure 6, why does the performance of the default system prompt vary across different iterations and example counts?\n\nWe observe that the performance of the default system prompt generally improves with more iterations and higher example counts, as it increases the opportunities to adapt and incorporate relevant information from more computation. However, the performance does not increase monotonically, which we attribute to the inherent variance in LLM outputs and the stochastic nature of the prompt optimization technique (namely, ProTeGi) that may yield instability. Specifically, we found that it has an average standard deviation of 0.015 across different example counts, which reflects the variance in the results of Figure 6."}, {"review_id": "qRsv1ji8Ev", "reviewer": "Reviewer_rqST", "summary": "This paper takes the concept of task-based prompt optimization, and extends it to include optimization of the system prompt as well as of the task-specific user prompt. By performing these optimizations together, and across a wide variety of tasks, this system creates a system prompt that helps across a variety of task types, including unseen tasks, and allows for faster optimizing of task-specific user prompts with increased performance.", "questions": "How was it decided how many system prompts and user prompts would be generated and maintained in each loop iteration?\n\nHow does the total processing required compare to existing task-specific techniques (not just in terms of adding a new task once you have a system prompt, but in terms of training the system from scratch for a variety of tasks)?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This paper is very clearly written and the technique described is explained well. It clearly spells out 2 separate benefits: the ability to help with unseen tasks, and faster optimization of task-specific prompts with excellent performance compared to task-only (no system prompt) optimization. The work is well-motivated and the technique is clearly explained. The experiments done and their results clearly show the benefit of this meta-optimization, especially in terms of being effective across new/unseen/unrelated tasks. Analyzing the attention to system prompt vs user prompt helps the reader understand the value in optimizing system prompts. It was helpful to include the analysis of using a prompt optimized with one model on another. One could imagine future work trying to optimize a global system prompt that is model-agnostic by using a technique like this on multiple models at the same time within in the inner and outer loops.\n\nThe idea itself isn't incredibly novel (basically extending existing prompt optimization methods to consider the system prompt in an outer loop), but I haven't seen it done in this way before, and it is clear from the results that it has value.\n\nOne thing missing in this paper is an analysis of how much processing this technique requires versus existing task-only techniques. It is clear from the paper that training a prompt for a new task, given a trained system prompt, is very efficient and this is backed up by some numbers. But, in terms of total processing to accomplish some number of tasks, how does the total processing compare in terms of having to optimize both the system prompt and the task prompts, versus using existing techniques to optimize only a task prompt for each task?\n\nIt is also not clear in the paper why the authors generate 9 different system prompts (maintaining one) and 3 different task prompts (maintaining them all) in the implementation details. It would be helpful to understand how those numbers were arrived at. Was there an empirical reason to choose them, or was it from tying different values and comparing results, or based on resource usage, or something else?", "was_revised": true, "final_justification": "Keeping my score as Accept. Although not a theoretical breakthrough or out-of-the-box idea, this work is important in showing that system prompts can/should be part of the optimization process, and that doing this helps efficiency in task learning as the number of tasks increases.\n\nThe responses to the comments by all reviewers were thorough and helpful. Clarifying the efficiency of the method including both prompt optimizations (although one is held steady after learning even as new tasks are added) and explaining more clearly why the loops use the number of candidate prompts helps fill out the paper information.", "rebuttal": "We sincerely thank you for your valuable and constructive feedback. We are also grateful for your recognition of the strengths of our work: the work is well-motivated, the technique is clearly explained, and the experiments and their results clearly show the benefit of the meta-optimization, demonstrating the effectiveness of our approach in helping with unseen tasks and optimizing task-specific prompts rapidly with excellent performance. We have carefully considered and faithfully responded to all your comments.\n\n---\n\n> W1. The idea itself is not incredibly novel (basically extending existing prompt optimization methods to consider the system prompt in an outer loop), but I have not seen it done in this way before, and it is clear from the results that it has value.\n\nThank you for acknowledging the value of our work and its empirical effectiveness. While we agree that our work builds upon prior prompt optimization methods, our core novelty lies in introducing the bilevel system prompt optimization problem (which has not been previously formalized), and proposing a meta-learning framework designed to optimize system prompts in synergy with user prompts across diverse tasks, which we believe are valuable to the community since optimization of system prompts (despite their wide use) are underexplored. \n\n---\n\n> W2. One thing missing is an analysis of how much processing this technique requires versus existing task-only techniques. While it is clear that training a prompt for a new task given a trained system prompt is very efficient, in terms of total processing to accomplish some number of tasks, how does the total processing cost in both system and user prompt optimization compare to existing techniques to optimize only a task prompt for each task?\n\nThis is an excellent question. We first would like to clarify that the process of system prompt optimization can be done only once, and the resulting system prompt can be reusable across diverse tasks and domains with high efficiency for adapting to them, which is similar to how modern foundation models are pre-trained once and then fine-tuned or prompted for various downstream applications. Therefore, considering the total computation cost (system and user prompt optimizations) might be somewhat less relevant in practical settings.\n\nNevertheless, in addition to the computational cost comparison provided in Appendix B.3 (where the number of model calls to optimization and generation models is used as a proxy for processing cost), we further show in Table C.1 that MetaSPO becomes more efficient than ProTeGi as the number of tasks increases. Specifically, although MetaSPO incurs a higher initial cost for optimizing the system prompt on diverse source tasks, the amortized cost of the system prompt becomes increasingly negligible: with just 3 target tasks, it requires fewer total model calls per task (for both base and optimizer models) than ProTeGi.\n\nTable C.1. Comparison of computational cost per target task between ProTeGi and MetaSPO with ProTeGi.\n| # of Target Tasks | Methods | # of Optimizer Model Calls | # of Base Model Calls |\n|:---:|:---:|:---:|:---:|\n| 1 | ProTeGi(20 iterations) | 366 | 9150 |\n| 0 | MetaSPO(Pre-train on 4 tasks) | 126 | 18000 |\n| 1 | MetaSPO + ProTeGi(4 iters) | **204** | 19950 |\n| 2 | MetaSPO + ProTeGi(4 iters) | 141 | 10950 |\n| 3 | MetaSPO + ProTeGi(4 iters) | 120 | **7950** |\n\n---\n\n> W3. It is also not clear why the authors generate 9 different system prompts (maintaining one) and 3 different task prompts (maintaining them all) in the implementation details. It would be helpful to understand how those numbers were arrived at. \n\nWe thank you for pointing it out and apologize for not providing a rationale for the implementation details about the number of prompts generated and maintained. In the inner loop for task prompt optimization, we generate 3 candidate user prompts per task, following the empirically validated, conventionally used number [1], and then retain the top 3 among the previous and current candidate prompts to ensure diversity in user prompts (which system prompts are jointly optimized with). In contrast, in the outer loop, we generate 9 candidate system prompts (i.e., 3 candidate prompts for each user prompt optimized from the prior step, using the conventional number), and retain only the top-performing one. We would like to note that this number is adopted to control the exponential growth in the number of prompts across iterations: if we were to retain two system prompts instead, the number of user prompts in the next inner loop would grow to 6 (2×3), and subsequently, 18 system prompts in the next outer loop (6×3). In other words, while generating and retaining more prompts could potentially improve performance, our design aims to strike a balance between effectiveness and computational efficiency, and exploring the impact of more aggressive prompt expansion with costs would be an interesting direction for future work.\n\n[1] PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization, ICLR 2024\n\n---\n\n> Q1. How was it decided how many system prompts and user prompts would be generated and maintained in each loop iteration?\n\nWe thank you again for your constructive comment. Please refer to our response to W3. \n\n---\n\n> Q2. How does the total processing required compare to existing task-specific techniques?\n\nWe thank you again for your valuable comment. Please refer to our response to W2."}, {"review_id": "g8rKgYhRNt", "reviewer": "Reviewer_tfVR", "summary": "This paper proposes MetaSPO, a bilevel prompt optimization framework designed to optimize system prompts for large language models (LLMs), which are often hand-crafted and static. Unlike prior work that focuses on optimizing user prompts for specific tasks, MetaSPO\nseparates prompt optimization into two levels: an inner loop that updates user prompts based on task-specific failure cases, and an outer loop that updates the system prompt based on aggregated failure patterns across tasks. The framework leverages the LLMs to analyze\nincorrect outputs and generate improved prompt candidates. Experiments on 14 unseen datasets across 5 domains demonstrate improved generalization and reduced adaptation cost compared to baseline prompting approaches.", "questions": "- How is response correctness evaluated during prompt selection? Are labels used?\n- Are the top-k system prompts ultimately ensembled, averaged, or is a single prompt selected?\n- Why was only LLaMA 3.2B used as the generator? Would larger models reduce reliance on the analyzer?\n- In experiments where in-domain prompts underperform global prompts, what explains this result?", "limitations": "Yes, in the appendix, there is a discussion on limitations. However, no direction is presented to address the limitation.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n- Proposes an interesting idea and well-motivated problem: system prompt optimization using a meta-learning framework\n- Demonstrates cross-domain generalization on 14 unseen datasets.\n\nWeakness:\n- The idea of using LLMs to optimize prompts is not fresh. Although the paper introduces system prompt optimization techniques,  the approach resembles components used in prior works [1]. More explanation, comparison, and contrast are required to highlight the novelty of the approach. \n- The meta-learning claim is overstated: The method is more accurately a search-based approach using LLM for optimization than the classic meta-learning algorithm, such as MAML. More explanation and evidence are required to justify the claim\n- Lack of clarity on implementation details. For example, regarding top-k prompt usage, it is unclear whether the top-k system prompts\nare ensembled, re-ranked, or reduced to a single selection in the final evaluation. Terms like “analyzer” and “optimizer” are used\ninterchangeably but are not clearly defined. The paper seems not to explain how response correctness is measured during optimization (e.g., automatic metrics, reward model, or human evaluation?).\n- The practicality of the approach is questionable. It highly depends on strong external analyzers (e.g., GPT-4), which may limit accessibility. The generator's limited capacity may overstate the impact of the optimized system prompt. Also, the computational cost of repeated prompt generation and evaluation could be significant, given that there is only a modest performance gain. \n- The baseline set seems to ignore the recent user prompt optimization approaches [2,3,4]. To prove that a complex bilievel prompt optimization is needed, there should be a significant gain observed compared to these baselines.  \n\n[1] Agarwal, Eshaan, Joykirat Singh, Vivek Dani, Raghav Magazine, Tanuja Ganu, and Akshay Nambi. \"PromptWizard: Task-Aware Prompt Optimization Framework.\" arXiv preprint arXiv:2405.18369 (2024).\n\n[2] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2023.\n\n[3] Zhang, Tianjun, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. \"TEMPERA: Test-Time Prompt Editing via Reinforcement Learning.\" In The Eleventh International Conference on Learning Representations, 2023. \n\n[4] Do, VT. et al. (2025). Automatic Prompt Selection for Large Language Models. In: Wu, X., et al. Advances in Knowledge Discovery and Data Mining. PAKDD 2025. Lecture Notes in Computer Science(), vol 15872.", "was_revised": true, "final_justification": "After reading the authors' response, I am happy to raise the score to borderline accept.", "rebuttal": "We sincerely appreciate your constructive and helpful comments, as well as your acknowledgement that we propose a well-motivated problem of system prompt optimization using meta-learning, and we demonstrate cross-domain generalization on 14 unseen datasets. We have made every effort to address all your comments faithfully.\n\n---\n\n> W1. Using LLMs to optimize prompts is not fresh. Although the paper introduces system prompt optimization techniques, the approach resembles components used in prior work [1].\n\nThank you for raising the concern about novelty and for pointing out the relevant work. While we agree that prompt optimization is a well-studied area, our work introduces a novel problem: bilevel system prompt optimization, which aims to learn a single, task-agnostic system prompt that generalizes across diverse user prompts and unseen tasks. This setting is fundamentally different from prior works [1], which require labeled data and re-optimization for each task, ultimately optimizing user prompts specific to each task. Also, in MetaSPO, even without targeted data on diverse tasks and domains, a single global-level system prompt obtained is generalizable to them, which we validate in Table 1 and Figure 5.\n\n---\n\n> W2. The meta-learning claim is overstated, since the method is more like a search-based approach using LLM for optimization than classic meta-learning (e.g., MAML).\n\nWe would like to clarify that meta-learning is not limited to classic gradient-based methods like MAML; rather, it broadly refers to learning shared knowledge from a distribution of tasks to enable generalization to unseen tasks. In this sense, MetaSPO fits well within the meta-learning paradigm: it employs bilevel optimization where the outer loop optimizes a task-agnostic system prompt that captures transferable knowledge across tasks, along with task-specific user prompt optimization for each task in the inner loop. In addition, this learned task-agnostic system prompt generalizes to unseen tasks and enables faster adaptation at test time, which is the core objective and key characteristic observed in meta-learning.\n\n---\n\n> W3.1. Regarding top-k prompt usage, it is unclear whether the top-k system prompts are ensembled, re-ranked, or reduced to a single selection in the final evaluation.\n\nWe select and use the single best-performing system prompt based on its average performance across tasks and user prompts, after generating multiple candidate system prompts through the outer loop (See Algorithm 3 and Line 14 in the supplementary material).\n\n> W3.2. Analyzer and Optimizer are used interchangeably, but are not clearly defined.\n\nIn MetaSPO, the role of Analyzer is to identify failure patterns of prompts on incorrectly predicted samples (called problem analysis), and Generator is designed to produce improved prompt candidates from the failure analysis (as shown in Algorithm A.3 and Tables 6, 7, 8, and 9). Both are implemented by instantiating the LLM with their corresponding prompt templates. In addition, the term Optimizer (in the caption of Table 4) refers to the specific prompt optimization techniques (such as APE or Evo), and MetaSPO is agnostic to the choice of them. We will make sure to improve their descriptions in the revision. \n\n> W3.3. How is response correctness measured during optimization?\n\nWe measure the response correctness using task-specific automatic metrics, such as Exact Match (EM) and F1, depending on the datasets (See Appendix A.1). Also, the answers are extracted from the model outputs with the predefined answer format, such as &lt;answer&gt;{answer}&lt;/answer&gt;.\n\n---\n\n> W4.1.The approach depends on strong external analyzers, which may limit accessibility.\n\nThe analyzer can also be instantiated with less powerful language models, such as LLaMA 3.1 (8B) or GPT-4o mini, as demonstrated in our experimental results in Table 15. Specifically, MetaSPO consistently shows substantial gains over baselines, regardless of the choice of language models for operationalizing the analyzer, which suggests that it is indeed accessible and practical for a wide range of users with various computational budgets. \n\n\n> W4.2. The generator's limited capacity may overstate the impact of the optimized system prompt.\n\nWe would like to point out that our experiments cover a wide range of generators, including Llama (8B), Qwen 2.5 (7B), and GPT-4o mini, in addition to Llama (3B), as shown in Table 3. From this, we then show that MetaSPO consistently outperforms baselines with varying generator models, which confirms its generalizability and robustness. However, to more fully address your comment, we further conduct experiments with GPT-4o, evaluating the system prompt optimized from GPT-4o mini to GPT-4o without further optimization, and as shown in Table D.1 below, MetaSPO achieves the highest average score over baselines, indicating its strong transferability to even the strongest model. Lastly, we believe that the ability of MetaSPO in improving the performance of lightweight LLMs is indeed its strength, which offers practical values in terms of efficiency, especially in resource-constrained settings.\n\nTable D.1. Unseen generalization results of GPT-4o with system prompts optimized from GPT-4o mini\n||Review|Analysis|||Reasoning|||\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||Electronics|Pet|Sports|Count|Epistemic|Color|Avg.|\n|Default|73.4|75.0|69.2|78.2|81.6|98.0|79.2|\n|CoT|73.0|76.0|70.6|80.0|80.0|98.4|79.7|\n|Service|70.6|75.0|66.8|82.0|81.6|98.0|79.0|\n|SPRIG|73.0|73.8|68.0|81.8|81.8|98.0|79.4|\n|MetaSPO|76.9|75.4|77.3|82.9|81.1|98.9|**82.1**|\n\n\n> W4.3. The computational cost of repeated prompt generation and evaluation could be significant, given that there is only a modest performance gain.\n\nWe appreciate your comment. However, we first would like to clarify that the process of system prompt optimization can be done only once, and the resulting system prompt can be reusable across diverse tasks and domains with high efficiency for adapting to them, which is similar to how modern foundation models are pre-trained once and then fine-tuned or prompted for various downstream applications. Therefore, considering the total computation cost (system and user prompt optimizations) might be less relevant in practical settings.\n\nNevertheless, we further show in Table D.2 that MetaSPO becomes more efficient than ProTeGi as the number of tasks increases. Specifically, although MetaSPO incurs a higher initial cost for optimizing the system prompt on diverse source tasks, the amortized cost of the system prompt becomes increasingly negligible: with just 3 target tasks, it requires fewer total model calls per task (for both base and optimizer models) than ProTeGi.\n\nLastly, the performance gain from MetaSPO is not modest but statistically significant on 11 out of 14 tasks (Table 13), which improves user prompts in 85% of cases (Figure 3).\n\nTable D.2. Comparison of computational cost per target task between ProTeGi and MetaSPO with ProTeGi.\n| # of Target Tasks | Methods | # of Optimizer Model Calls | # of Base Model Calls |\n|:---:|:---:|:---:|:---:|\n|1|ProTeGi(20 iterations)|366|9150|\n|0|MetaSPO(Pre-train on 4 tasks)|126|18000|\n|1|MetaSPO + ProTeGi(4 iters)|**204**|19950|\n|2|MetaSPO + ProTeGi(4 iters)|141|10950|\n|3|MetaSPO + ProTeGi(4 iters)|120|**7950**|\n\n---\n\n> W5. The baselines seem to ignore the recent user prompt optimization approaches [2-4]. \n\nWe would like to emphasize that MetaSPO targets task-agnostic system prompt optimization, unlike the suggested task-specific user prompt optimization approaches; therefore, they should be considered as orthogonal and indeed complementary to our framework, as it is compatible with them in optimizing prompts (as shown in Table 4). \n\nMoreover, we conduct test-time adaptation experiments to validate that MetaSPO is indeed helpful to further improve the performance when coupled with the user prompt optimization techniques. The results in Table D.1 show notable gains in performance when using APE [2] and PromptAgent [5] for test-time adaptation, supporting the orthogonal and complementary effect of our MetaSPO, which even query-dependent prompt optimization approaches [3-4] could similarly benefit from: MetaSPO offers consistent, task-agnostic guidance that improves stability and reduces variance across tasks and user prompts. We will discuss it.\n\nTable D.3. Test-Time Adaptation with other user prompt optimization methods\n|Optimization Strategy|System Prompt|Medical|Review.|Reasoning|Safety|Grounding|Avg.|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|APE|Default|0.455|0.643|0.609|0.395|0.168|0.479|\n||MetaSPO|**0.471**|**0.680**|**0.640**|**0.437**|**0.175**|**0.505**|\n|PromptAgent|Default|0.451|0.654|0.614|0.532|0.175|0.502|\n||MetaSPO|**0.465**|**0.685**|**0.663**|**0.593**|**0.183**|**0.533**|\n|APO|Default|0.451|0.689|0.640|0.599|0.175|0.524|\n||MetaSPO|**0.456**|**0.714**|**0.673**|**0.672**|**0.199**|**0.552**|\n\n[5] PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization, ICLR 2024\n\n---\n\n> Q1. How is response correctness evaluated? \n\nPlease refer to our response to W3.3.\n\n---\n\n> Q2. How are the top-k system prompts ultimately used?\n\nPlease refer to our response to W3.1.\n\n---\n\n> Q3. Why was only LLaMA 3.2B used as the generator, and how about larger models?\n\nPlease refer to our response to W4.2.\n\n---\n\n> Q4. What explains this result: in-domain prompts underperform global prompts.\n\nThis is because the global prompts benefit not only from exposure to in-domain tasks but also from additional knowledge and reasoning patterns acquired from diverse out-of-domain tasks.\n\n---\n\n> L1. While there is a discussion on limitations, no direction is presented to address them.\n\nIn our discussion of limitations, we also outline a potential direction to address them: distilling stronger optimizer LLMs into smaller ones to improve the capability of lightweight optimizers."}]

Author rebuttal:

Meta-review: This paper proposes MetaSPO, a bilevel prompt optimization framework that jointly optimizes system prompts and task-specific user prompts for large language models. By alternating between updating user prompts based on task-specific failures and updating the system prompt using aggregated failure patterns, the method improves performance on unseen tasks and accelerates adaptation of user prompts. The approach is evaluated across 14 datasets spanning five task domains, demonstrating consistent gains over baseline prompting methods. The paper is well-written, clearly motivated, and the experiments are comprehensive.

Reviewers generally agree that the paper is technically solid and empirically strong, with intuitive motivation and clear benefits of system prompt optimization. Strengths include the comprehensive evaluation, clarity of presentation, and demonstrated generalization. Weaknesses mostly concern novelty, theoretical grounding, and scope: the method resembles a search-based optimization rather than classical meta-learning, relies on strong external analyzers, and its effectiveness on subjective or open-ended tasks is untested. Minor concerns were raised regarding implementation details and the computational cost relative to task-only optimization.

Despite these limitations, the authors’ rebuttal clarified most concerns, particularly regarding experimental validity and implementation choices. The practical significance of jointly optimizing system and user prompts is clear, showing improved efficiency and performance across tasks. On balance, the strengths and impact of the work outweigh the concerns, and I recommend acceptance.
Decision: Accept (poster)