Pareto Prompt Optimization
Authors: Guang Zhao
Byung-Jun Yoon
Gilchan Park
Shantenu Jha
Shinjae Yoo
Xiaoning Qian
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Prompt Optimization
Multiobjective Optimization
Reinforcement Learning
DPO
Primary area: foundation or frontier models, including LLMs

Abstract: Natural language prompt optimization, or prompt engineering, has emerged as a powerful technique to unlock the potential of Large Language Models (LLMs) for various tasks. While existing methods primarily focus on maximizing a single task-specific performance metric for LLM outputs, real-world applications often require considering trade-offs between multiple objectives. In this work, we address this limitation by proposing an effective technique for multi-objective prompt optimization for LLMs. Specifically, we propose **ParetoPrompt**, a reinforcement learning~(RL) method that leverages dominance relationships between prompts to derive a policy model for prompts optimization using preference-based loss functions. By leveraging multi-objective dominance relationships, ParetoPrompt enables efficient exploration of the entire Pareto front without the need for a predefined scalarization of multiple objectives. Our experimental results show that ParetoPrompt consistently outperforms existing algorithms that use specific objective values. ParetoPrompt also yields robust performances when the objective metrics differ between training and testing.

Published as a conference paper at ICLR 2025
PARETO PROMPT OPTIMIZATION
Guang Zhao', Byung-Jun Yoon 1.2, Gilchan Park Shantenu Jha Shinjac Yoo', Xiaoning Qian 1.2
1 Brookhaven National Laboratory. Texas A&M University, Princeton Plasma Physics Laboratory 'Rutgers University- New Brunswick, Prrnneeton University gzhao@bnl. gov, biyoonStamu.co edu, gparkEbnl.gov, shantenu@pppl. gov, s sjyoo8bnl. .gov, xqianaece.tamie tamu.
ABSTRACT
Natural language prompt optimization, or prompt engineering, has emerged as a powerful technique unlock the potential of Large Language Models (LLMs) for various tasks. While existing methods primarily focus maximizing sin- gle task-specific performance metric for L.M outputs, real-world applications often require considering trade-off's between multiple objectives. In this work. we address this limitation by proposing an effective technique for multi-objective prompt optimization for LLMs. Specifically, we propose ParetoPrompll Prompt, a rein- forcement learning (RL) method that leverages dominance relationships between prompts derive policy model for prompts optimization using preference-base loss functions. By leveraging multi-objective dominance relationships, Pareto- Prompt enables efficient exploration of the entire Pareto front without the need for a predefined scalarization of multiple objectives. Our experimental results show that ParetoPrompt consistently outperforms existing algorithms that use specific objective values. ParetoPrompt also yields robust performances when the objec- tive metrics differ between training and testing.
INTRODUCTION
Advancements in Large Language Models (LLMs) have attracted significant interest due to their re- markable capabilities across various natural language processing (NI.) tasks. Prompting, a method that utilizes natural language prefix context to guide the L.MSs to complete desired task, allow-
ing us to utilize LLMs' capabilities without re-training LLMs (Wei et al., 2022; Wen et al. 2024 Reynolds & McDonell] 202T). However, crafting effective prompts often necessitates significant manual efforts, requiring expertise in both LLMs and the specific task domains (Wang et al. 2024
Prompt optimization emerged powerful solution, leveraging algorithms automate the search for optimal prompts. These algorithms encompass diverse techniques such as gradient-based opti- mization (Wen et al. 2024), reinforcement learning (RL) (Deng et al. 2022 Zhhag et al.) 2022). evolutionary algorithms (Zhou et al.) 2022), beam search (Pryzant et al. 2023), and inverse RL. (Sun ct al. 2023). These methods typically formulate prompt optimization focusing on single objectives, aiming to optimize single chosen performance metric, such as accuracy or fluency. In real-world applications, however, prompt effectiveness often involves trade-offss between multiple objectives. For example, a prompt designed for text style transform may need to balance style consistency with content accuracy, while prompt for summarizing factual topics may need consider both infor- mativeness and conciseness.
Unlike single-objective formulations (Wen et 2024) where we may obtain a clear "best" prompt, in multi-objective prompt optimization problems with multiple conflicting objectives, there is not single prompt that excels all objectives simultaneously. Instead, we aim for set of prompts from the so-called Pareto front, which represents the best possible trade-off's between these objectives. To search for and maximally cover the Pareto front, optimization algorithms- srnithhmsinncludnng RL or evolu- tionary algorithms often utilize indicator function reward function to evaluate the quality of prompts. These solution strategies translate the multi- objeitvbeeeetve performance into a single value that reflects how good a of prompts for example, via the weighted sum of objectives, S metric or

---

Published as a conference paper at ICLR 2025
Hypervolume of the dominated region (Baumann & Kramer 2024). and product of multiple objec- tives (Jafari et al. 2024). The introduction of these indicator/reward functions in ad-hoc ways helps guide prompting towards the Pareto front. But it also imposes rigid assumptions about the trade-offs between objectives, which oversimplifies the nuanced preferences involved in text generation. For example, the weighted sum of objectives assumes the linear trade-offs between objectives and the weight is predefined. While S-metric assumes uniform preference across all regions of the objec- tive space. Some objective of text generation can be measured by various metries, such as fluency by perplexity grammaticality, each of which may have non-linear relationships. The previously men- tioned weighted sum and S- S-metric fail to capture these complexities, as they overlook the possibility of different preferences arising from evaluating the same objective by varying metrics.
In this study, rather than using a sealar metric to describe the multi-objective performance of prompts, we propose to guide the prompt search by comparing pairs of prompts based on funda- mental principles of multi-ohjective problems. Specifically, if one prompt dominates another, the dominating prompt is considered more preferable. Conversely, if a pair of prompts do not dominate each other, then we do not prioritize one over the other. This approach ignores the specific objec- tive values of the prompts. While discarding specific values might seem to reduce the algorithm's effectiveness, two key factors motivate our approach. First, in language generation tasks, accurate and reliable evaluation of absolute objective values is often unavailable or unreliable. Relative pref- erences, on the other hand, are more accessible and robust in handling the inherent vagueness in evaluation objectives. Second, using dominance relationships avoids imposing assumptions on the underlying structure of the objectives. This also climinates the need to assume the additive contri- butions of objectives, uniform preferences across regions, or biases introduced by reference points.
Based on these motivations, we propose ParetoPrompt, a novel multi-objective prompt optimiza- tion method driven by preference based RL. In our formulation, prompts are generated by a policy model. During each iteration, the algorithm samples pairs of prompts for the same input instance and compares their dominance relationship, then update the policy model accordingly. The algorithm can be combined with various types of RI.-hased prompt generation methods, such as direct text to- ken generation from language models (Deng et al., 2022 Wu et al., 2022). or using RL-trained edit agents (Zhang et al.) 2022). We have conducted experiments comparing ParetoPrompt with com- peting baselines. Our results clearly show that, despite solely based on dominance relationships, ParetoPrompt achieves better comparable performance than the algorithms relying on specific objective values. Additionally, our method demonstrates robust performance even when the training metric differs from the evaluation metrics used during testing. The code for our implementation is made available at
2
RELATED WORK
We first review existing research on Prompt Optimization and Multi-objective Optimization for LLMs, including the relevant recent work on Direct Preference Optimization (DPO).
2.1 PROMPT OPTIMIZATION
Prompting has become a prevalent approach for guiding LLMs towards specific tasks within the NLP domain. Soft prompting techniques require access to the latent embeddings, limiting their applica- bility in closed-source LLMs (Li & Liang 2021). Natural language prompt optimization leverages optimization algorithms to generate effective text prompts without modifying LLM parameters (Wen)
al. 2024 Deng al. [0222] Zhang al. 2022 Zhou et al. 2022 Pryzant al. 2023 Sun al.
2023 For instance, Lin et al. (2024) learn reward model trained on human preference data and
then optimize the reward model to find the optimal prompt. Fernando et al. (2023); Guo et al. (2023)
employ evolutionary algorithms for single-objective prompt optimization. However, a majority of these works focus on single-objective optimization formulations.
2.2 MULTI-OBIECTIVE OPTIMIZATION FOR LANGUAGE MODELS
The field of multi-objective optimization has been explored for both prompt optimization and LLM fine-tuning. [aaumann & Kramer (2024) proposed an Evolutionary Algorithm (EA) where LLMs perform "crossover" and "mutation" operations on prompts during the optimization process. Jafari
2

---

Published as a conference paper at ICLR 2025
et al. 2024) adapted various scalar reward functions (c.g., HyperVolume Indicator, Expected prod- uct objectives) for multi-objective prompt optimization using RL. Jang et al. (2023) introduced a method for fine-tuning LLMs for multiple objectives by training separate policy models and merging their parameters for personalized preferences. /hhou et al. (2023) proposed Multi-Objective Direct Preference Optimization (MODPO), RI.ffree algorithm extending Direct Preference Optimiza- tion (DPO) for datasets with multiple dimensional preference. They prioritised preferences with weight vectors and derived multi-objective preference reward function. Our work distinguishes itself from these previous efforts by utilizzig dominance preference signals between prompt pairs, avoiding the need for predefined scalar metrics or reward functions, which often introduce ad-hoc assumptions of the objectives.
2.3 DIRECT PREFERENCE OPTIMIZATION
DPO Rafailov et al. 2024) offers an alternative to RL for fine-tuning pre-trained language models. It utilizes human preference data directly for updates instead of training reward model like in Rein- forcement Learning from Human Feedback (RLHF). Azar et al. (2024) proposed Identity Preference Optimization (IPO), a generalization of DPO, that replaces non-decreasing function related the Bradley-Terry model (Bradley & Terry 1952) with the identity function, which mitigates overfitting issues DPO. The prompt optimization problem different from the DPO fine-tuning problem. Prompt optimization depends on the specific task and language model in use, leading to the lack of a widely accepted general prompt dataset for prompt optimization. Therefore, we adopt RL approach to interact with the task-specific language model, enabling us to learn the most effective prompts for particular problems. Our work utilizes the DPO/IPO reward function for dominance preference data in a multi-objective optimization framework.
PARETO PROMPT OPTIMIZATION
We now present ParetoPrompt (Fig. [. for prompt optimization that aims to cover the corresponding Pareto front of multi-objective NLP tasks with pre-trained LLMs.
Task Model
2g
dominance?
non-dominant pair
dominant pair (x,y))
dominant
(x(o)))
og(x,y)
Policy
Model
update
(x.
(x
(x(3y)
lele,, 21:2)
Figure ParetoPrompt Iteration: ParetoPrompt trains policy model generate diverse and "Pareto optimal" prompts for an instance æ. In each iteration, the policy model samples pair of prompts. Based the dominance relationships of their outputs, it calculates either the dominant or non-dominant loss, which is used to update policy model.
3.1 MULTI-OBJECTIVE PROMPT OPTIMIZATION
We consider the problem of generating Pareto optimal prompts to improve desired responses from task-specific LLM, denoted as T. Given an input instance x, which can be reviews, queries or code, our objective is design the prompts guide the task-specific LLM towards generating high-quality responses y. Achieving high quality response y can involve multiple, sometimes conflicting, objec- tives. Denote these objectives O y)
y), - On (I. v)), where each element, 0; y)

---

Published as a conference paper at ICLR 2025
represents specific quality metric, such relevance, fluency and/or creativity. These metrics can be automatically estimated using various techniques, for example based on the BLEU Score, Gram- matical Error Rate, and Semantic Distance, evaluated by another LLM Zhang et al. 2023
We aim to train a policy model To. using RL to generate prompts that approach the Pareto front. The policy model takes input x: and generates corresponding prompts with the probability mo(2|x)). Here the prompt 2 refers to the text provided to the task-specific LLM to generate the response y = T=)). This includes both the instruction or query and any context provided before the the model's ss generated output. The simplest form of a prompt involves adding prefix text tokens in front of the imput æ (Deng al., 2022). prompt is considered "Pareto optimal" if there exists no other prompt that can improve one objective in Ozz, y) without degrading at least one other objective. Mathematically, for two prompts 21 and 22, if (xr T(2)) 3 O(z, T(22)), and Ozx; T(=1)) o Oor, (22)), then 21 is "Pareto dominated" by #2. Therefore, a prompt is "Pareto optimal" if it is not dominated by any other prompt. Finding such prompts allows us obtain the best trade-off prompts considering multiple conflicting objectives.
The Pareto front represents the set of all Pareto optimal prompts, i.e. the set of prompts that "Pareto dominate" all the other prompts but mutually incomparable. In guiding the search for the Pareto front by RL training of the policy model, a common approach is to use a scalarization function that transforms the multi-objective problem into standard single-objective problem. While convenient for training, these functions can impose oversimplified assumptions about the objective structure and limit the exploration of the entire Pareto front. The most common function is the weighted sum function, which prioritizes the solutions that maximize a pre-defined linear combination of objectives, neglecting potentially valuable parts of the Pareto front. Similarly, the hypervolume indicator (Zitzler et al. 200TD. while rewards improvement in all objectives simultaneously, it may prioritize points that contribute most the overall hypervolume increase, potentially neglecting areas with smaller hypervolume contributions. Furthermore, the preference of hypervolume may vary based on the choice of the reference point, introducing bias in the search process (Ishibuchi et al.] (017). Consequently, although using sclarization function provides guidance in approaching the Pareto front in RL, it can also introduce bias in the search, limiting the algorithm's 's capability to cover the whole Pareto front.
In the following subsections, we introduce an innovative reward function for multi-objective prompt optimization. This approach leverages dominance relationships between prompts to guide the pol- icy model towards the Pareto front. By doing so, we eliminate the need for predefined preferences imposed by traditional scalarization functions, allowing for more flexible and comprehensive ploration of the solution space.
3.2 DOMINANCE PREFERENCE BASED LOSS FUNCTION
Given input instance æ. we consider the dominance preference between two prompts zu and z based on their corresponding outputs you and yn- Specifically, if O Vm)) > O(z, yr), we define Zu be dominance preferable prompt over 21, we denote as Zw > 21 for brevity. We define data pair (z, Zw. =) as dominance preference data, where Zu dominates 21 in terms of the resulting outputs. This data serves as the foundation for directly learning the policy model that generates Pareto optimal prompts.
Rafailov al. (2024) introduced Direct Preference Optimization (DPO), a method that updates the policy model based preference data without training a separate reward model. Since policy model To is guided by the reward function, they showed that the reward function is implicitly connected to the policy model. This connection can be expressed as The (2, 2) x log [=][=|])* where Teer denotes a reference model. This reference model serves as baseline or starting point for learning preferences, and it typically chosen as an initialized model or a pre-trained model.
(a(alz)
DPO utilizes loss function derived from the Bradley-Tere model for preference modeling:
(DPO z1; æ) = - log [3h(2w, 21; x1]]]]
(1)
where 3 is a scaling hyperparameter, a is the logistic sigmoid function and h is the reward difference between 2n and 2 defined:
To (z x)
log To (2(2)
h(zw, 2) = log
(2)
Tief Zz [z)
Teer (z) (r)
4

---

Published as a conference paper at ICLR 2025
This loss function IPo increases the reward for the dominating prompt 2w and decreases the reward for the dominated prompt 21, thereby promoting the generation 2w.
Extended from DPO, Identity Preference Optimization (IPO) (Azar al. 2024) replaces the non- decreasing function related to the Bradley-Tene -erry model by the identity function, resulting in a simpler loss function:
( x;) h(Z2w =1; 7
where 7 is regularization hyperparameter, controlling h(zw. 21; 2)) the reward difference between Zw and 2t. DPO aims to maximize the difference to infinity, leading to overfitting to the preference
- dataset. In contrast, IPO aims make the difference close to - therefore mitigating the risk of
overfitting. The comparison of /pro and (po as functions of h(z 21; x) is shown in Fig. 2
Both DPO and IPO losses utilize dominance preference data to update the policy model, aligning prompt generation with the dominance rela- tionship between outputs. Note that dominance preference data is inde- pendent of specific objective values, so it remains robust against scaling or monotonic transformation of ob- jectives.
Juro
10
Ano
0.6
0.4
0.
-2
3.3 NON-DOMINATED
22:22
LOss FUNCTION
Figure 2: Comparison of loss functions, where turo and 4no increase h(2w. 21; 2), the reward gap between dominant and dominated prompt pairs; while Ind reduces the reward gap between non-dominant pairs to near zero
Based on preference update with dominance preference data, we are able to generate Pareto optimal prompts. However, there is no mechanism guaranteeing the coverage of the entire Pareto front. We therefore propose non-dominated loss function based on the non-dominated data to encour- age the policy model to generate diverse prompts to explore different trade-offs on the entire Pareto front.
Given a data pair (z, 31, 22), it is possible that the corresponding outputs y/1 and 3/2 are mutually non-dominated (incomparable), which we denote 22 and call (x, 21, non-dominated data. For these cases we do not expect a strong preference for either prompt. We want the policy model assign similar likelihood in generating them. However, there should still be some tolerance for small likelihood differences between non-dominated prompts; otherwise, the loss function would force the model to assign identical likelihoods all prompts. Therefore, we define a loss function that penalizes large differences in the rewards of 21 and z2, but tolerates small difference for non- dominated prompt pairs:
(1, x) = À max( [h22,, x0] - e, 0),
(4)
where À is scaling hyperparameter, and is the tolerance of the difference between the reward function values. Note that as [h(21, =2; = log =0(=u|2)
am)(;; the reward function =Ą(22|z) log mur(22
difference actually reflects the deviation between the policy model To and the reference model wee in generating non-dominated prompts. Therefore, this loss function only becomes effective after the policy updates lead the difference exceeding the threshold This ensures that the policy prioritizes learning dominant prompts first before focusing on diversifying non-dominated prompts.
The curve of Ina is shown in Fig. From the comparison with 4ro, we can conclude that when using lPo and Ind as loss function terms, it's important to set e < - for hyperparameter selection to let Lnd take effect.
3.4 PARETOPROMPT ALGORITHM
We now describe ParetoPrompt, our proposed training algorithm, for policy model generating Pareto-optimal prompts in detail.
5

---

Published as a conference paper at ICLR 2025
The policy model leverages pre-trained generative LLM with its latent embedding layers kept frozen. To fine-tune the model for prompt generation, a residual adapter module, implemented as Multi-Layer Perceptron (MLP), is inserted between the latent layers and the model head. The reference model is set to be identical to the initial state of the policy model.
Briefly, denote either the DPO IPO loss function la, then the overall loss function for ParetoPrompt is:
C(TTi Par) [1((1, 2x11 =2) (1222 x2)
thna(21, 22) (2) ~ 22)]
(5)
The training process is as follows (Fig. D
1. Randomly sample training instance r.
2. Use the reference model to generate pair of prompts, 21 and 22 for
3. Estimate the objectives of the corresponding outputs and y2 and determine their dominance relationship, then select the either la or lnd loss function based on the dominance.
4. Use gradient-based optimization algorithm (e.g., Adam) to update the policy model parame- ters based on the calculated loss.
5. Periodically update the reference model to match the current state of the policy model. This ensures that the reference model can leverage improved prompts for training as the policy model evolves.
6. Repeat above steps for a specified number of training iterations.
Advantage of our algorithm is that by combining the dominance preference loss function with non-dominated loss function, we encourage the policy model to generate Pareto optimal prompts while diversify the generation explore the entire Pareto front. While ParetoPrompt avoids mak- ing assumptions about the multi-objective structure by not using scalarization function, the way that ParetoPrompt currently treats non-dominant pairs cannot provide guide for generating better prompts as no preferences needs to be learned from them. However, in multi-objective problems, dominant pairs become rarer and non-dominant pairs become more frequent, especially with the increasing number of objectives, in "many-objective" problems. As result, ParetoPrompt can be inefficient these scenarios.
EXPERIMENTS
To validate performance of proposed PranetoPrompt, apply it both classification and text generation tasks, with multiple objectives. We also provide analyses of the ParetoPrompt algorithms.
Baselines.
We compare proposed ParetoPrompt with the following baselines:
1. Summation (Deng et al. 2022): This RL-based algorithm uses the reward function defined by scalarization, simply the summation of different objectives: r(z, y) (z, y).
2. Product: This RI.-based algorithm defines the reward as the product of different objectives::
r(a, y) = (o,
3. HVI: The algorithm uses the HyperVolume Increment (HVT) as the reward function within
RL framework. tracks the Pareto front during training and use the hypervolume increment bring by each prompt the reward.
4. Reward-Guided IPO (R-IPO): This preference- based RL algorithm calculates the sum of objectives, and determines the preference of prompt pairs based on the summation, then updates the policy model using IPO loss in equation 3
5. InstOptima (Yang& & Li 2023): This evolutionary algorithm utilizes the NSGA-II framework for multi-objective optimization. The mutation and crossover operators for prompts are exe- cuted using an LLM with corresponding operation prompts. In our experiments, we employ LLaMa 2 (7B) for prompt operations.
6. ParetoPrompt DPO/PO (PP-DPO/IPO): Our proposed ParetoPrompt algorithms.

---

Published as a conference paper at ICLR 2025
4.1 FEW-SHOT TEXT CLASSIHICATION (TWO-OBJECTIVE TASK)
We conduct experiments on single-sentence classification across various datasets using token in- filling with a BERT model (Brown 2020). Classification is based on the probability of to- kens corresponding to a set of verbalizers as class labels. We follow the prompt template [Input] [Prompt] [C] ass] as in Deng al. (2022) and select the verbalizer token with the highest predictive probability at the [Class] position. For few-shot classification, We only take small number of training samples and search for better prompts. We conduct experiments on a di- verse set of popular few-shot classification tasks, including MR (Pang & Lee 2005), SST-5 (Socher
et al. 2013). Yelp-5 and Yahoo (Zhang et al. 2015).
Objectives and Misaligned Metrics We define a two-objective prompt optimization problem: besides optimizing the accuracy of the classification task, we also aim optimize the fluency of the prompts. Various metrics can be used to evaluate the fluency. In this set of experiments, we use the grammatical acceptability score of Corpus of Linguistic Acceptability (CoLA) task (Morris et al. 2020). We denote the score as the CoLA score. This model is trained to classify whether sentence is grammatically correct or not, and we aim to maximize the CoLA score to generate fluent prompts.
RoBER model fine-tuned for the
Since fluency can also be quantified by the perplexity of text as calculated by a language model, to demonstrate the robustness of ParetoPrompt against potentiallly misaligned metries, we employ two distinct training signals: the CoLA score and, in a separate series of experiments, the perplexity calculated using GPT-2, while consistently take CoLA scores as the true objective in the testing stage. We have analyzed the CoLA and Perplexity scores of the prompts used our experiments, and the results show non-linear relationship between the CoLAA scores and the perplexity scores. This non-linear relationship leads to poor performance when RL algorithms,, trained using specific perplexity scores reward signal, while evaluated using CoLA scores. Details of the correlation analysis can be found in Appendix A.2
Experimental details We use RoBERTa- large (Liu et al., 2021) as the LM for classification, while our policy model is based on DistilGPT2 (Sanh et al. 2019) with two-layer MLP adaptor added before the head layer. The prompt search space consists of discrete tokens. For all datasets, we randomly sample 16 samples per class for both the training and validation sets. The final per- formance is evaluated using a sufficiently large test set. For all RL-based algorithms (excluding InstOptima), 16 prompts are sampled for each iteration to calculate reward functions. Algorithms using dominance relationships (R-IPO and PP-DPO/IPO) employ 8 prompt comparison pairs for reward function calculation. Therefore, during the training, the total number of task language model queries is: 16 x class num x 16 x 6.000. Hyperparameters for the loss functions in equations (1), (3) and (4) are set as = 0.5,* r 0.5, À and 0.1. Fach RL algorithm runs for 6K iterations for training. For InstOptima, we intialized with a population of 16 manually designed prompts and exe- cute 60 generations of NSGA-II, matching the computational time of the RL algorithms. During the test stage, we generate 64 prompts for each dataset to perform comprehensive comparison in terms of multi-objective performances. We conduct five independent runs to obtain average performance measures.
We use Hypervolume (HV) to evaluate the multi-objective performance of classification Accuracy and the prompt CoLA score, with the reference point set at (0, O). HV for experiments using the CoLA score the training signal is denoted as C-HV, while HV for experiments using Perplexity as training signal is denoted as P-HV. The difference between the two metrics, DVff-HV, measures the robustness of algorithm performance across these two settings. Based on the experimental results shown in Table I ParetoPrompt algorithms (IPO and DPO) con- sistently demonstrate superior performance across both C-HV and P-HV and have a small Diff-HV. In contrast, the Diff-HV for algorithms using scalarization rewards (Summation, Product and HVT) is significantly larger than other algorithms. That is due to the non-linear relationship between CoLA score and perplexity not only changes the absolute reward value, but also the relative rank- ings among prompts, leading to degraded performance. While the dominance relationships utilized by the ParetoPrompt algorithms remain unaffected by such transformations. Notably, InstOptima shows a small Diff-HV, partially due to its usage of NSGA-II, which relies on ranking based on dominance relationship.
Experimental Results
7

---

Published as a conference paper at ICLR 2025
Table Comparison of dominated hypervolume (HV) by different methods in the bi-objective space defined by classification accuracy and CoLA score. "C-HV" refers to using the CoLA score as the training signal, and "P-HV" indicates using perplexity scores training signal. The term "Diff- HV" represents the hypervolume difference between these two metrics. Higher values of "C-HV" and "P-HV" are preferable, while "Diff-HV" close to indicates robustness against metric change.
Dataset
Metric
Summation
Product
HVI
InstOpt
R-IPO
PP-DPO
PP-IPO
С-HV
76.5(8.4)
63.3(5.2)
77.2(6.9)
70.6(9.1)
75.5(8.5)
80,9(4.3)
83,0(4,6)
MR
P-HV
54.4(11.1)
46.5(11.2)
55.3(12.0)
67.2(0.1)
61.0(5.1)
66.8(11.5)
68.3(7.1)
Dif--HV
22.1
16.8
21.9
3,4
14.5
14.1
14.7
C-HV
32.0(2.1)
27.4((1.)
36.2(008)
3I(3.7)
29.5(7.0)
34.1(2.5)
35.3(0.5)
SST-5
P-HV
23.6(3.1)
20.4(1.2)
24.4(2.5)
32.6(1.9)
29.9(1.3)
34.3(0.4)
34,0(0.5)
Din'HV
8.4
7.0
11.8
-1.6
-0.4
-0.2
1.3
C-HV
39.8(1.9)
27.5(1.2)
42.2(2.6)
40.7(3.9)
24.9(1..)
47.1(1.7)
48.2(0.5)
Yahoo
P-HV
21.2(4.8)
17.0(7.4)
2..3
31.5(4.2)
20.6(4.3)
31.3(5.2)
35.5(2.7)
Dift-HV
18.6
10.5
13.9
9.2
4.3
15.8
12.7
C-HV
35.5(2.8)
27.066.7)
40.4(2.6)
24.1(1.4)
40.0(377)
41.3(311)
40.6(2.1)
Yelp-5
P-HV
21.9(2.1)
19.7(4.8)
20.5(3.1)
35.1(2.5)
37.1(4.)
39.7(2.5)
39.8(2.3)
Diff-HV
13.6
7.3
19.9
-11.0
2.9
1.6
0.8
4.2 TEXT STYLE TRANSFER (THREE-OBIECTIV TASK)
Following RLPrompt (Deng et al. 2022). we evaluate ParetoPrompt on unsupervised text style transform task. The goal is rewrite an input sentence into desired style while still keep the content similar. Two conflicting objectives are considered: style score and content similarity. We also include include fluency of the prompts as a third objective. We conduct the task using the Yelp sentiment dataset (Shen et al. 2017) to convert Yelp negative reviews into positive ones while main- taining the content similarity. For example, the sentence"i will never be back" might be transformed into "i will be back again". The dataset consists of Yelp restaurant reviews, labelled by star ratings, with three or above as positive and those below three as negative. We randomly select 50 negative reviews for training, 50 for evaluation, and separate set of 100 for testing.
Objective Settings. We adopt two model-based based metrics for content similarity and sentiment posi- tiveness evaluation. We set the content similarity objective as the content preservation reward func- tion with the Compression, Transduction, and Creation (CTC) metric introduced in Deng et al.) (022), which measures the embedding alignment between the input and output. The sentiment ob- jective is defined as the sentiment probability calculated using a BERT-based classifier fine-tuned on the Yelp dataset. For fluency objective, we kept using the CoLA scores calculated by the RoBERTa- based-CoLA model. Compared with few shot classification. this objective setting is more challeng- ing for prompt optimization. In few shot classification, outputs are chosen among verbalizer, while text generation introduces greater output uncertainty. As a result, the quality of the objective signal is noisier, making more difficult to identify prompts that achieve good average performance. To account for generative model randomness, we generate 128 outputs per prompt, averaging objective values for robust prompt evaluation.
Experimental Details. We use GPT-2 XL as the LM for the style transform task, while the prompt generation setting is the same previous experiments. For all algorithms except InstOptima, during the training stage, each iteration processes a minibatch size of two input negative reviews, and four prompts are sampled for each negative review, which are then used to transform the inputs into positive reviews. Each algorithm runs for 10K iterations for training, resulting in a total number of language model queries equal to 128 8 x 10,000. For InstOptima, of 16 manually designed prompts and run 130 generations of NSGA-II, ensuring a comparable running time with the RL algorithms. During the testing stage, we again generate 64 prompts for each instance to evaluate the multi-objective performance. To ensure robustness, we conduct three independent runs to obtain an average performance measure.
intialized with a population
Experimental Results The performance of generated prompts in the multi-objective space of single run is shown in Fig. 3 providing an intuitive illustration. Additionally, Table providdr the average performance across three runs with mean and standard deviation values, with all objectives
8

---

Published as a conference paper at ICLR 2025
set to maximize. Each algorithm generate 64 prompts from the policy model, and the Pareto Set Size represents the number non-dominated prompts generated by each algorithm reflecting its ability to generate effective prompts. Two other metrics are evaluated for the overall performance in the multi- objective space: the Dominated Volume (HV) and the Inverse Generation Distance (IGD). HV is the volume of the dominated region in the objective space with respect to the reference point (0, 0). IGD originally measures the average distance between the true Pareto front to the closest points in the objective space corresponding to generated prompts by different algorithms. A lower IGD value implies that the generated prompts achieve the performances closer to the Pareto front. Here since the ground truth Pareto front unknown, the Pareto optimal prompts of the combination of all the prompts act as the reference Pareto front.
Summation
90
Product
90
80
InstOpt
80
Content score
70
PP-
70
PR-IPO
60
60
50
10
20
30
40
5.0
0.1
0.5
0.8
0.9
0.95
0.98
Sentiment score
CoLA score
Figure 3: Visualization of three-objective prompt optimization results: Comparison of generated prompts from different algorithms in the objective space, with sentiment, content similarity, and CoLA score objectives to be maximized. Results show that our ParetoPrompt algorithms effec- tively explore the entire Pareto front.
Performance Comparison for Unsupervised Sentiment Transform. Pareto Set Size repre- sents the number of non-dominated prompts, HV measures Dominated HyperVolume, IGD indicates distance to the Pareto front.
Table 2:
Metric
Summation
Product
HVI
InstOpt
R-IPO
PP-DPO
PP-IPO
Max Style f
29.8(2.6)
17.2(2.0)
33.4(6.2)
30.5(2.4)
19.6(2.8)
30.7(2.9)
36.5(5.7)
Max Content
46.7(3.0)
44.7(3.4)
52.6(4.7)
77.5(11.4)
68.0(16.6)
69.9(13.2)
82.9(17.4)
Max Col.A
0.64(0.17)
0.95(0.01)
0.95(0.01)
0.96(0.0T)
0.96 (0.01)
0.98(0.0)
0.98(0.0)
Pareto Set Size
14.7(5.0)
6.6(3.1)
13.2(4.3)
11.6 (3.2)
26.8(5.4)
34.0(3.6)
34.8(10.9)
HV
572(135)
725(45)
1429(153)
1838 (228)
1166(241)
2007(242)
2406(361)
IGD L
31.6(2.5)
33.7(2.))
17.9(98)
17.1(3.7)
22.6(3.9)
15.7(7.6)
14.4(4.8)
From Fig. 3 we observe that Summation, Product and HVI do not perform well due to the fact that the inaccurate reward cannot guide the search effectively, R-IPO tends to form clusters because it lacks mechanism generate di- verse prompts but simply exploits the preference information provided by preference data. In contrast, our pro- posed ParetoPrompt algorithms (es- pecially PP-IPO) can cover the en- tire Pareto front due the intro- duction of non-dominance loss. Ta- ble 2 shows that our proposed Pare- toPrompt algorithms achieve higher HV and smaller IGD, reflecting that the prompts generated by Pareto- Prompt effectively cover the whole ob-
mon 138 Rosettge (50
0.59
025
Non Dor.Dominaattoo PaEl
Domburt Paina
Dom nonti.oo
Non mon-mnnnni--
Trns loce
0.15
0.5
Training Harad
Figure 4: Training Analysis of ParetoPrompt-IPo. Trends in dominant non-dominant prompt pair percentages, and total, dominance, and non-dominance loss changes.
Pareto front. Additionally, serve that PP-IPO performs better than PP-DPO, probably because the

---

Published as a conference paper at ICLR 2025
IPO loss does not suffer from the potential overfitting issues associated with DPO, which allows PP- IPO to explore the search space more efficiently. Overall, our ParetoPrompt algorithms demonstrate superior performances by effectively covering the entire Pareto front with diverse prompts.
4.3 PARETOPROMPT TRAINING ANALYSIS
We analyze the ParetoPrompt algorithms during the training process, specifically focusing on the text style transfer task. the ParetoPrompt-IPO algorithm, we present the percentage change of dominant and non-dominant prompt pairs in Fig. 4 Additionally, we include the non-dominated loss defined in Eq. 5). The figure reveals that as training progresses, the algorithm samples more non-dominant prompt pairs, indicating it learns to generate trade-off prompts. The non-dominated loss starts at value of and increases as training continues. This increasing non-dominated loss suggests that the algorithm approaches the Pareto front, it focuses more exploring non- dominated prompts. Our experimental results shown in Appendix [A.3 also indicate that without the non-dominated loss, the dominate-only algorithm tends to generate prompts in clusters.
5 CONCLUSION
We have developed ParetoPrompt, RL-based prompt optimization algorithm for multi-objective text generation. The algorithm's training relies solely on the multi-objective dominance relation- ships between pairs of prompts, and requires predefined scalarization function, thus allows us bypass assumptions about human preferences in text evaluation. ParetoPrompt defines separate loss functions for dominant and non-dominant prompt pairs. The combination effect of these loss func- tions encourages the generation of Pareto-optimal prompts but diversifies the prompts to cover the entire Pareto front. Moreover, by using only dominant relationships, the algorithm performs robustly even when there is mismatch between training and testing metrics. Additionally. it can incorpo- rate preference data for training. Overall, ParetoPrompt presents a preference learning approach for generating Pareto-optimal prompts, providing a promising direction for multi-objectin prompt optimization.
LIMITATION AND FUTURE WORK
highlighted Section 4.3, ParetoPrompt may become inefficient when addressing problems with large number of objectives. To address this limitation, future work can explore incorporat- ing relaxed Pareto dominance relations (López Jaimes & Coello Coello (009), which extends the definition of dominance to capture subtle preference information between non-dominant prompts. Examples include the (1 - --)--ommminnncee relation (Farina & Amato 2002) and the expansion relation that controls the dominance area of solutions (Sato et al. (007). By incorporating these relaxed lations, we can potentially use the information from non-dominant pairs to guide the policy model's updates in many-objective cases.
ACKNOWLEDGEMENT
This work was conducted part of the LUCID (Low-dose Understanding, Cellular Insights, and Molecular Discoveries) program, supported by the U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research, under Contract DE-ACO2-06CHI 1357. X.Q. ac- knowledges partial support from National Science Foundation (NSF) through grants SHF-2215573 and IIS- 221-291
REFERENCES
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand leam- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4457-4455. PMLR, 2024.
10

---

Published as a conference paper at ICLR 2025
Jill Baumann and Oliver Kramer. Evolutionary multi-objective optimization of large language model prompts for balancing sentiments. In International Conference on the Applications of Evolution- ary Computation (Part of EvoStar), pp. 212-224. Springer. 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-333 345, 1952.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205.12544 2022.
Marco Farina and Paolo Amato. On the optimal solution definition for many-criteria optimization problems. In 2002 annual meeting of the North American fuzzy information processing society proceedings. SAFIPS-FLINT 2002 (Cat. No. 02TH8622), pp. 233-238. IEEE, 2002.
Chrisantha Fernando. Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement arXiv:2309.16797, 2023.
prompt evolution. arXiv preprint
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532 2023.
Hisao Ishibuchi, Ryo Imada, Yu Setoguchi, and Yusuke Nojima. Reference point specification in hypervolume calculation for fair comparison and efficient search. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 585-592, 2017.
Yasaman Jafari, Dheeraj Mekala, Rose Yu, and Taylor Berg- Borg-Kirkpatrick. MORL--Prompt: em- pirical analysis of multi-objectivv reinforcement learning for discrete prompt optimization. arXiv preprint 11X1:22402.11711,, 2024.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannanch Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.1 1564, 2023.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 122853//1/20211aallooom wwtp://hwoooo...oo22....eeee.. long.
URL
Xiaoqiang Lin, Zhongxiang Dai, Arun Verma, Bee-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. Prompt optimization with human feedback. arXiv preprint arXiv:2405. 17346, 2024.
Zhuang Liu, Wayne Lin, Ya Shi, and Jun Zhao. A robustly optimized bert pre-training approach with post-training. In China National Conference on Chinese Computational Linguistics, pp. 41144771444 Springer, 2021.
Antonio López Jaimes and Carlos Artemio Coello Coello. Study of preference relations in many- objective optimization. In Proceedings of the I 11th Annual conference on Genetic and evolutionary computation, pp. 61 1-618, 2009.
John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A frame- work for adversarial attacks, data augmentation, and adversarial training in nip. arXiv preprint arXiv:2005.05000 055099 2020.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect rating scales. arXiv preprint cs/0506075, 2005.

---

Published as a conference paper at ICLR 2025
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with "gradient descent" and beam search. arXiv preprint arXiv:2305.03495, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pp. 1-7, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910.01 108, 2019.
Hiroyuki Sato, Hernán E Aguirre, and Kiyoshi Tanaka. Controlling dominance area of solutions and its impact on the performance of moeas. In International conference on evolutionary multi- criterion optimization, pp. 5-20. Springer, 2007.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. Advances in neural information processing systems, 30, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. Query-dependent prompt evaluation and optimization with offline inverse RL. In The Twelfth International Conference on Learning Rep- resentations, 2023.
Li Wang, Xi Chen, XiangWen Deng, Hao Wen, MingKe You, WeiZhi Liu, Qi Li, and Jian Li. Prompt engineering in consistency and reliability with the vrisence-based guideline for llms. npj Digital Medicine, 7(1):11, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thouge prompting elicits reasoning in large language models. Advances in neural information processing systems, 25:24824-24837, 2022.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36, 2024.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, VG Vydiswaran, and Hao Ma. Idpg: An instance-dependent prompt generation method. arXiv preprint 2004:2200.04449, 2022.
Heng Yang and Ke Li. Instoptima: Evolutionary multi-objective instruction optimization via large language model-based instruction operators. arXiv preprint urXiv:2310.17633 2023.
Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. Tempera: Test-time prompting via reinforcement learning. arXiv preprint ar. 18990, 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems. 28, 2015.
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang. Hongbo Xu, and Yongbin Li. Wider and deeper I1m networks are fairer IIm evaluators. arXiv preprint arXiv:2308.0 01862, 2023.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint 1rXiv:2211.0199 2022.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-frn aareall Multi-objective direct preference optimization. arXiv preprint 1000023300003700 2023.
12

---

Published as a conference paper at ICLR 2025
Eckart Zitzler, Marco Laumanns, and Lothar Thiele. Spea2: Improving the strength pareto evolu- tionary algorithm. TIK report, 103, 2001.
A
APPENDIX
This appendix presents supplementary materials, including the pseudo-code for the ParetoPrompp algorithm, an analysis of CoLA and perplexity scores, and results from two-objective prompt opti- mization experiments in text style transfer.
A.1 PSEUDO-CODE FOR PARETOPROMPT
Pseudo-code for ParetoPrompt is summarized Algorithm!
Algorithm ParetoPrompt
Require: Training dataset X, Reference model Tiet Loss Hyperparameters
Wref (angeenuueepoche do 3: for in X' do
1: Initialize policy model
2: for epoch
4:
21 2 x2 then loss La(21, 22; 5: else if =1 =2 then loss (2(2,, #1; else loss = Ind(22) 22;
6:
7:
end if
8:
Update To with gradient descent on loss end for
9:
10: if (epoch% .ppdate.period) == 0 then f-
11: end
12:
end for
A.2 ANALYSIS OF CoLA AND PERPLEXITY SCORES
We analyzed the CoLA and Perplexity scores prompts used our experiments. We generated 1000 prompts of tokens randomly and calculate their CoLA score and perplexity. And their rela- tionship is shown in Fig. S1. Notably, the CoLA score exhibits an approximately linear relationship with log perplexity.
The Spearman's Rank Correlation Coefficient between them is -0.473, indicating a moderate neg- ative correlation. In contrast, the linear correlation coefficient is -0.253, reflecting weak negative linear relationship. However, when we apply a logarithmic transformation to Perplexity, the linear correlation coefficient increases to -0.471. This suggests non-linear relationship between CoLA and Perplexity scores, which can be approximately described by logarithmic transformation.
A.3 TEXT STYLE TRANSFER WITH TWO OBJECTIVES
We also conduct two-objective prompt optimization experiments on the text style transfer task. The settings are the same as Sec. 4.2, except that we optimize prompts for two objectives: content similarity and sentiment positiveness. Fig. S2 provides intuitive illustration of the generated prompts in the objective space, Fig. S3 also provides the average performance across five runs.
Notably, the results shows that without the non-dominance loss, the Dominance-On Onlu algorithms tend to form clusters because of the lack of mechanism to diversify prompts, as shown in Fig. S2. In contrast, ParetoPrompt can cover the entire Pareto front due to the introduction of non-dominance loss. Fig. S3 shows that our proposed ParetoPrompt algorithms achieve higher D-HV with smaller variance because diverse prompts lead to higher and more robust D-HV. In contrast, the compet- ing algorithms generate prompts that cluster together, causing the D-HV varies with the cluster's position and resulting in larger variance. The smaller IGD ParetoPrompt reflects it has smaller distance from the reference Pareto front, reflecting that the prompts generated by ParetoPrompt ef- fectively cover the whole Pareto front. Overall, our ParetoPrompt algorithm demonstrates superior performance by effectively covering the entire Pareto front with diverse prompts.
13

---

Published as a conference paper at ICLR 2025
10a
Perplexity (GPT2)
10
10*
10a
0.0
1.0
Grammatical Acceptability Score (CoLA)
Figure S1: Relationship between CoLA and Perplexity scores.
100
RIPrompt
90
HVI
Reward-Guide DPO
80
Reward-Guided IPO
8
Dominance- Only DPO
sent eentiment score
70
Dominance- Oniiaacce--nlyy IPO
ParetoPrompt DPO
60
ParetoPrompt IPO
30
20
0
10
20
30
40
50
60
70
content score
Figure S2: Bi-objective Prompt Optimization Illustration. Comparison of generated prompts from different algorithms for a single instance in bi-objective space, with sentiment and content similarity as the two objectives be maximized. Results show that our proposed ParetoPrompt algorithms effectively explore the entire Pareto front.
A.4 TEXT STYLE TRANSFER WITH FOUR OBJECTIVES
In this section, we present preliminary results from an extended experiment evaluating our method on four-objective optimization task in text style transfer. While prior studies on multi-objective prompt optimization typically consider most three objectives, our method is fully capable of han- dling more complex scenarios. To demonstrate this, we evaluate performance under the following four objectives: Style score, Content similarity, Output fluency and Conciseness. The first three ob- jectives are the same as Sec. 4.2, while the fourth objective, Conciseness measured by the length of the output.
14

---

Published as a conference paper at ICLR 2025
A.Propp
Bewardd-Fuiiee pro Reword Guited IPO
Dominence-Dnll DFD Dominance Only IPO
Perelohhompi DPQ ParetoPrompciPP
HV1
2000
2300
2000
1300
20
1000
Figure S3: Performance Comparison for Unsupervised Sentiment Transform. We compares the performance of various algorithms in an unsupervised sentiment transform task. D-HV (higher better) measure Dominated HyperVolumm IGD (lower is better) indicate distance to the Pareto front. Our ParetoPrompt surpasses competing algorithms in both metries.
We present the available results for methods that could be easily adapted to this four-objective set- ting. These results provide insights into the capability of our method in handling larger number of objectives. The results, based on three independent runs per method, are summarized in Table SI.
Table S1: Performance Comparison on Four-Objective Text Style Transfer. Pareto Set Size rep- resents the number of non-dominated prompts, HV measures Dominated HyperVolume, and IGD indicates distance to the Pareto front.
Metric HV
Summation
Product
R-IPO
PP-DPO
PP-IPO
6279 (634)
6793 (659)
16895 (575)
23199 (170)
30861 (154)
IGD +
34.57 (1.35)
34.18 (2.07)
22.55 (2.55)
8.98 (0.55)
3.05 (0.44)
Pareto Set Size t
15.67 (1.70)
15.3 (0.47)
25.5 (6.5)
33.5 (1.5)
51.5 (1.5)
These preliminary results highlight the ability of our method to optimize across multiple objectives.
3000
ParetoPrompt IPO (20)
Deminaling typerVolure
30
ParetoPromet IPO (200)
ParetoPrompt IPO (500)
2300
2250
2000
ParetoPrompt IPO (20)
15
1750
ParetoPrompt IPO (200)
ParetoPrompt IPO (500)
1250
1000
2000
3000
4000
5000
6000
7000
1000
2000
3000
4000
5000
6000
7000
Training Iterations
Training Iterations
Figure $4: Ablation performance comparison with different update period choices.
15

---

Published as a conference paper at ICLR 2025
A.5 ABLATION: REFERENCE MODEL UPDATE PERIOD
The reference model update period (in Algorithm 1 line 10) plays a crucial role in balancing the convergence speed and stability during training. With short update period, the reference model closely tracks the current policy model, potentially leading to faster convergence; but such choice can also lead to unstable training. In contrast, long update period can lead to more stable training. However, the reference model may hinder the further improvement of the policy model. We conduct an ablation study with the different reference model update periods set 20, 200 and 500. The results are shown in Fig. $4. From the figure, we can observe that period of 200 achieves good balance between convergence speed and performance both D-HV and IGD. This setting allows the reference model to adapt sufficiently guide the policy's ' learning while maintaining stability for exploration.
16


Reviews:
[{"review_id": "yGqu6e9MWa", "reviewer": "Reviewer_UXSm", "summary": "This paper addresses the multi-objective prompt optimization challenge through a novel method called ParetoPrompt. In particular, ParetoPrompt introduces a reinforcement learning from human feedback (RLHF) approach that utilizes dominance preference data, enabling it to efficiently explore optimal trade-offs across multiple objectives. This approach is both innovative and promising, as it allows for nuanced optimization without relying on rigid scalarization functions.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Introducing reinforcement learning from human feedback (RLHF) into Pareto optimization is a novel and inspiring approach, adding a valuable dimension to multi-objective optimization.\n- The final results are promising, demonstrating the method's potential to achieve balanced and effective trade-offs across objectives.", "weaknesses": "- The motivation for using RLHF in Pareto optimization, as opposed to standard Pareto algorithms, could be further elaborated to strengthen the case for this approach.\n- The paper lacks a detailed report on querying budgets (or optimization efficiency), which is critical for assessing practical performance in prompt optimization.\n- The learning procedure for the reward model appears complex and may be challenging to implement or adapt across diverse applications.\n- Including more results with a greater number of objective functions would enhance the evaluation and demonstrate broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UBjQYqRK6f", "reviewer": "Reviewer_9fmx", "summary": "This paper introduces ParetoPrompt, a reinforcement learning method designed for multi-objective prompt optimization.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The reported experiments show the proposed algorithm outperform the baseline methods under a variety of metrics", "weaknesses": "1. More recent baseline methods should be compared. For example: https://arxiv.org/abs/2406.12845\n2. Any theoretical justifications that the proposed training process (in the end of section 3) is Pareto-optimal?\n3. In the proposed training process, how to estimate the objectives of the corresponding outputs y1 and y2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sERKE32FDC", "reviewer": "Reviewer_x6AH", "summary": "This paper proposed a Pareto prompt optimization algorithm. Instead of considering a single objective, this paper considers multi-objective in a real prompt optimization scenario and proposes an RL-based algorithm to find the Pareto optimal prompts in the Pareto front and achieve better performance than other algorithms.", "questions": "1. How does the approach proposed in this work different from [5]? Since [5] also considers the human preference in prompt optimization (in this paper's case, dominance relationship). Could the author provide some explanation? If indeed, there are some similarity, is it possible to compare this work as one of the baseline in the paper?\n\n[5] Lin, X., Dai, Z., Verma, A., Ng, S. K., Jaillet, P., & Low, B. K. H. (2024). Prompt Optimization with Human Feedback. arXiv preprint arXiv:2405.17346.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The Pareto prompt optimization problem proposed by the author is novel and practical for me\n2. The presentation of the algorithm design is great\n3. The experimental results shows that the Pareto prompt optimization algorithm outperforms other methods.", "weaknesses": "1. This paper did not provide enough discussion on the related works of prompt optimization, [1,2,3,4] are some of the works that I think should be included.\n\n2. In the comparison, the author did not provide any comparison with existing prompt optimization works like [1,2,3,4]. More justification of comparison on this is needed to position this paper in the area of prompt optimization.\n\n[1] Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2024). Large language models as optimizers. arXiv. https://arxiv.org/abs/2309.03409\n[2] Fernando, C., Banarse, D., Michalewski, H., Osindero, S., & Rocktäschel, T. (2023). Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797.\n[3] Lin, X., Wu, Z., Dai, Z., Hu, W., Shu, Y., Ng, S. K., ... & Low, B. K. H. (2024). Use your instinct: Instruction optimization using neural bandits coupled with transformers. ICML 2024.\n[4] Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., ... & Yang, Y. (2023). Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kz75AwQBHX", "reviewer": "Reviewer_yPkv", "summary": "The authors proposed a prompt optimization method relying on the pareto dominance relationships between prompts. This avoids a predefined aggregation among the multiple objectives. Instead, a loss containing separate components for dominance and non-dominance data is designed, optimizing for pareto optimal prompts on the pareto frontier. Experiments are conducted on real-world datasets and language models, showing the method’s capability of producing well-performing prompts across multiple objectives.", "questions": "1. Since non-dominated prompts consist of the majority of the sampled prompts, will they also dominate the loss function? For example, if most of the prompts are non-dominated, optimizing the loss function basically gives the same reward for almost all prompts. Please clarify.\n2. Clarify how having the loss in equation (4) encourages “diversifying non-dominated prompts” as stated in line 257?\n3. In the experiments, the fluency of the prompt is used as an objective. This is counterintuitive: I would imagine the fluency of the output/generation to be much more important than the fluency of the prompt itself. This makes the experiment results less convincing.\n4. From Figure 3, I can see that ParetoPrompt produces many prompts that lie on the “pareto front”. If I need to choose one of them eventually to use in the system, how should I choose the prompt?\n5. It seems strange to me that the percentage of non-dominated samples increases with training, but the loss also keeps increasing.\n6. When the number of objectives increases, it is expected to have much more conflicts (non-dominant pairs) as compared to dominant ones. Though this is discussed as a limitation in the last section, I always have this doubt in mind while reading the paper, do consider bringing this comment to an earlier section of the paper for clarity. On a side note, I would like to know the rough maximum number of objectives that ParetoPrompt can handle in practice?\n7. A missing related work [1] on using preference optimization for prompt optimization. \n8. [Typo] Line 404, “ouput” → “output”\n\nReferences:\n\n[1] Prompt Optimization with Human Feedback. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Studies an important practical problem of multi-objective metric design when the explicit structure is unknown.\n2. The special treatment for non-dominated prompts contributes to the novel design of the overall loss function.\n3. Clear visual illustration of the results lying on the pareto front.", "weaknesses": "1. The experiments are not extensive enough, e.g., no experiment beyond 3 objectives, and only models like BERT and GPT2 are used.\n2. The experiment design on the choice of objectives can be improved. Having prompt fluency as the objective makes less sense as compared to output fluency, conciseness, informativeness, style alignment, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)