QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning
Authors: Yilun Kong
Hangyu Mao
Zhao Qi
Bin Zhang
Jingqing Ruan
Li Shen
Yongzhe Chang
Xueqian Wang
Rui Zhao
Dacheng Tao
Venue: Accepted by TMLR ()

Abstract: Prompt engineering has demonstrated remarkable success in enhancing the performance of large language models (LLMs) across diverse tasks. However, most existing prompt optimization methods only focus on the task-level performance, overlooking the importance of query-preferred prompts, which leads to suboptimal performances. Additionally, these methods rely heavily on frequent interactions with LLMs to obtain feedback for guiding the optimization process, incurring substantial redundant interaction costs. In this paper, we introduce Query-dependent Prompt Optimization ($\textbf{QPO}$), which leverages multi-loop offline reinforcement learning to iteratively fine-tune a small pretrained language model to generate optimal prompts tailored to the input queries, thus significantly improving the prompting effect on the large target LLM. We derive insights from offline prompting demonstration data, which already exists in large quantities as a by-product of benchmarking diverse prompts on open-sourced tasks, thereby circumventing the expenses of online interactions. Furthermore, we continuously augment the offline dataset with the generated prompts in each loop, as the prompts from the fine-tuned model are supposed to outperform the source prompts in the original dataset. These iterative loops bootstrap the model towards generating optimal prompts. Experiments on various LLM scales and diverse NLP and math tasks demonstrate the efficacy and cost-efficiency of our method in both zero-shot and few-shot scenarios.

Published in Transactions on Machine Learning Research (02/2025)
QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning
Yilun Kong",, Hangyu Mao?i, Qi Zhao', Bin Zhang Jingqing Ruan Li Shen', Yongzhe Chang', Xueqian Wang't, Rui Zhao?, Dacheng Tao
TTinghua University; & SenseTime Research; 3 Institute of automation, Chinese academy of science; Schooo of Artificial Intelligence, University of Chinese Academy of Sciences; SSun Yat-Sen University; "Nanyang Technological University
Reviewed on OpenReview: https: // openreview. net/ pet/formm? id= bQMJTolkuT
Abstract
Prompt engineering has demonstrated remarkable success in enhancing the performance of large language models (LLMs) across diverse tasks. However, most existing prompt op- timization methods only focus on the task-level performance, overlooking the importance of query-preferred prompts, which leads to suboptimal performances. Additionally, these methods rely heavily on frequent interactions with LLMs to obtain feedback for guiding the optimization process, incurring substantial redundant interaction costs. In this paper, we in- troduce Query-dependent Prompt Optimization (QPO), which leverages multi-loop offline reinforcement learning to iteratively finc-tune a small pretrained language model to generate optimal prompts lailored to the inpul queries, thus significantly improving the prompting fect on the large target LLM. We derive insights from offline prompting demonstration data, which already exists in large quantities as by-product of benchmarking diverse prompts on open-sourced tasks, thereby circumventing the expenses of online interactions. Furthermore, continuously augment the offline dataset with the generated prompts in each loop, as the prompts from the fine-tuned model are supposed to outperform the source prompts in the original dataset. These iterative loops bootstrap the model towards gencrating optimal prompts. Experiments on various LLM scales and diverse NLP and math tasks demonstrate the efficacy and cost-efficiency our method both zero-shot and few-shot scenarios.
1 Introduction
Large Language Models (LLMs) have exhibited impressive prowess in various domains of natural language processing (NLP) (Ouyang et al., 2022 Touvron et al.) 2023 Achiam et al. 2023). ) Prompt engineering, a method that simply adds an instruction to the input query, emerges as a lightweight and promising solution
for adapting I.IMs to dowm tasks without the need for parameter tuning (Liu et al., 2023a Ajith
et a.. 2023 Since the performance of LLMs towards a particular task is significantly influenced by the quality of the prompt, the key challenge of prompting lies in how to design the optimal prompis.
Numerous prompt engineering algorithms have been proposed in recent years. Some algorithms (Zhou et al. 2022 Wang et al. 2023b Guo et al. 2023 Wang et al. 2024b leverage LLMs as a prompt optimizer, cm- ploying black-box optimization to derive the best prompts. Others utilize reinforcement learning (RL) (Deng et al. 2022 Zhang et al.) 2022 Kong et al. 2024) train policy model to generate the optimal prompts.
Despite these advances, prompt engineering still grapples with great challenges:
"Work doue during an internship SenseTTime Research. I fcorresponding authors: Hangyu Saaormmmccanggykkkeeeeetmm...om Xueqian Saanggwwww...deee...ssnnn......

---

Published in Transactions on Machine Learning Research (02/2025)
Zero-Shot Accuracy
Few-Shot Accuracy
News
Neoos
Bonl
BoolQ
85.6
76.5
Prompt-OIRI
86.8
71.0
ChanGPT-Rewrite
90.3
01.2
71,6
$1.4
GSHOK
INDB
HetaSwog
INDB
60.7
68/2
Hellewwag
65.3
CosmosQA
Tweet Emotion
Figure 1:
QPO delivers state-of-the-arl performance wide variely of lasks, superior existing query- dependent and query-agnostic prompt optimization methods.
(1) Most of previous algorithms only focus obtaining task-level optimal prompts, aiming to attain optinal average performance across all queries within specific task. However, they overlook the critical insight that no single prompt can be ideal for every possible query (Sun et al. 2023).
(2) All the aforementioned methods require frequent interactions with the target LLMs to obtain feedback the current prompts to guide optimization, incurring significant costs due to the high expense of inferences with target LLMs.
In fact, query-level optimal prompts can yield better performance (Zhang et al. 2022 Mcanwhile, offline optimization can serve as a promising method for reducing the costs associated with LLM interactions based on the existence of wealth of prompt optimization datasets, which can be easily available as by-products during the evaluation of existing prompting strategies. However, research the two important domains remains critically scarce.
To tackle the above challenges, we propose QPO, a Query-dependent Prompt Optimization method through multi-loop offline reinforcement learning. The overall process of our framework is illustrated in Figure 2 Firstly, we employ offline RL for the initial loop to finc-tune a small-scale pretrained language model (PLM) as a policy model on the pre-collected dataset, enabling to generate specific prompts based on input queries and given rewards for the target LLM. Subsequently, we utilize the trained policy model efficiently explore new queries and prompts to augment the datasct, the generated novel prompts are supposed to be more suitable for the current queries than other prompts in the dataset. This process eliminates the necessity to evaluate all collected prompts for each query, while suffices to assess only a few query-specific high-quality prompts, significantly reducing the required interactions with the target LLM. The augmented dataset, enriched with more queries and superior prompts, can further train the policy model and enhance performance. Consequently, we establish the iterative process wherein the policy model, fine-tuned via offline reinforcement learning, efficiently generates improved data, which then utilized further fine-tune the model, creating cyclical feedback loop. This bootstrapping learning process can achieve substantial improvements in limited loops with minimal interactions with the LLMs.
We evaluate our method across different LLM scales on various neural language understanding and math reasoning lasks using both zero-shot and few-shot settings. As shown in Figure 1 QPO reaches state-of- the-art performance, superior to both query-depeadent and query-agnostic prompting methods. Compared online optimization approaches, our method only needs a minimal number of interactions with the target LLM augment the dataset after each loop of offline optimization, which strikes great balance between performance and L.L interaction costs. In addition, perform extensive ablations on different aspects
2

---

Published in Transactions on Machine Learning Research (02/2025)
Step0: Dataset Construction
Metric Target Task
Target LLM
Query
Prompt
Output
Scorn
(query Prompt !)
Query
Prompt Ad
Output
Scure
Dataset
Step1: Training Policy Model with Offliin
The Train set Target Task
(Roward Query 3
Predicted Promat
Loss
(fromox
Design
PLM as Policy Model
Step2: Dataset Augmentation
Target LLM
Metric
x N'
Query I'
Novel Prompt 1'
Maximum Reward
(Qucry Promet it Soore 1
Query Ay'
Novel Prompt N!
Queries:
Query-level optimal prompts:
Maximum Reward
[Qusttion] is houss and property same
Easllatta whather the answer ol digs with the information presented the passage.
[Possage] Property tax house tax
[Question] is barg root beer peps product
The well-traines Policy Model
Ueternine the sosement comrepoonde to the decalls in the providdd test, determm too deterrie the aneover
Target LLM
Task-level Performance
[Possage] lanc's Footrks/ is an
The overview of our framework. We aim at training small pretrained language model (PLM) as the policy model, which can generate query-level optimal prompts based on the given queries, and ultimately improve the target LLM's task-level performance. In the training phase, depicted by the blue frame above, the blue arrows represent our primary iterative process, while the gray arrows indicate the initial steps. Specifically, in StepO, the offline dataset is constructed as by-product of evaluating existing prompts. The below frame with gray background indicates the testing phase.
Figure 2:
of the proposed algorithm. The policy model demonstrates great cross-model generalization ability, further enhancing the value of our approach. In conclusion, our contributions are threc-fold:
We identify the overlooked query-dependent prompt optimization problem, and introduce an Offline RL paradigm to fine-tune PLM to generate query-specific optimal prompts.
We propose a comprehensive framework, QPO, which leverages the existence of offline datasets and employs novel Multi-Loop Augmentation technique facilitate data augmentation with minimal interactions, and ultimately boolstrap the performance of prompt optimization through offline RL fine-tuning.
We benchmark our method across various LLMs on extensive datasets, demonstrating its superiority and great potential in broader applications.
2 Methodology
To automatically generate query-level optimal prompts in a cost-efficiency method, we propose QPO, a novel query-dependent prompt optimization framework. The overall process is shown in Figure In the first round, QPO harnesses the readily existing datasets to fine-tune pre-trained language model into policy model through offline reinforcement lcarning, enabling it to generate query-specific optimal prompts based on input queries. Subsequently, this policy model efficiently explores a broader query space and enriches the prompt space with more diverse and high-quality prompts, both of which contribute to the augmentation of the dataset. The augmented dataset, in turn, further refines the policy model for the next loop. Consequently, through multi-loop training and data augmentation, the performance of prompt optimization is bootstrapped for improvement.
3

---

Published in Transactions on Machine Learning Research (02/2025)
2.1 Problem Formulation
This study focuses on training the policy model to generale prompls, which can instruct target LLM to output expected answers that fulfill the query-dependent objective, based on the given queries.
Query and Answer. We consider the task of answering queries q E Q = v expressed in a natural language, where V denotes the vocabulary. Each query q is supposed to have an expected answer y² y as ground truth. Queries annotated with ground-truth answers can be employed as demonstrations d for few-shot scenario.
Prompt and Policy Model. The performance of an LLM can be significantly enhanced through appro- priate prompts. The prompt p P = y is a natural language instruction that explains to the LLM how to complete the query and output the predicted answer. In this paper, we utilize a fine-tuned policy model x : Q + P to generate query-specific prompts based on the given queries: p = x(q).
Target LLM. These tasks are performed by employing target LLM e: e + V. foeding queries q into the LLM to get answers. With prompls, the answers can be obtained by g(t...) = ((pl), d()), g())),, where g(,pp') ()) (h) denote the i-th query, j-th prompt, and k-th combination of demonstrations, respectively. For zero-shot setting, d is null, so we use g(t.) as shorthand unless otherwise specified.
Query-dependent Objective. Given a task with queries and ground-truth answers, the quality of the predicted answers can be evaluated by a metric p(y", 9), such as correctness. The objective of query- dependent prompt optimization can be formulated as below:
arg max (0
(9), {(T
(GEI)))
(1)
which aims to optimize the policy model to generate query-specific prompt that enhance the target LLM's performance on this particular query. As its performance improves across each single query, the overall task performance will consequently improve.
2.2 Model Query-dependent Prompt Optimization as Offline RL
Reinforcement Learning Formulation. As text generation problem, prompt generation can be formu- lated a Markov Decision Process (MDP) As,,, with a finile stale space S, action space A, reward function and state-transition probability function f, where a sparse reward can be obtained only after the complete prompt is generated. Instead of dedicateelyy designing intricate prozy rewards, we formulate the MDP as a single-step decision-making process. In offline RL (Levine et al. 2020 given the dataset: ( ()) (60),666) ")t,::0)))(l.)))))"))"))>...... (HEM))EEM)))
where the initial state q E S is the input query with n tokens q (g),..., 4n) and the corresponding action p E A represents the generated prompt with m tokens = (PI.... (m.) where each token q.P is from the vocabulary V. In single-step decision episode, the policy model generates a complete prompt based on the given expected reward and query p ~ (plr, q). The reward guides the quality of the generated prompt. To achieve this, we adopt Decision Transformer (DT) (Chen al. 2021] approach to fine-tune the policy model, which is aligned with the autoregressive predicting paradigm of language models. For further discussion on why prompt generation is formulated as single-step decision process, please refer to Appendix A.2
Reward Design. In our single-step prompt generation process, the reward plays crucial role in aligning with the true capabilities of the prompt. In this paper, we mainly focus on neural language understanding tasks and math reasoning tasks rather than generative tasks, which are challenging to assess objectively because of the poor correlation between the evaluation metrics and human preferences (Liang et al. 2022 Goyal et al. 2022). We design the reward based on two dimensions: query-level and task-level. Query-level reward measures whether the prompt can instruct LLM to answer the specific question correctly, while task- level rewand measures the prompt's average prompting ability for all queries in one task. In particular, we adopt the average accuracy of the prompt for all test questions as the task-level reward for both zero-shot and few-shot settings.
1 N N
(t) (p)))
Σ
1(005)) = (1).
(2)
=
4

---

Published in Transactions on Machine Learning Research (02/2025)
For query-level rewards, the design differs between zero-shot and few-shot scenarios due to the variations in evaluating approaches. For zero-shot evaluation, we utilize the prediction correct as a coarse-grained measure for the prompt-query pair and additionally introduce the output perplexity (PPL) as fine-grained penalty, which can be casily calculated by the Log-likelihood from the black-box API or directly obtained through the open-source LLM. During few-shot evaluation, the result of a query obtained by voling or averaging across K different in-context demonstration combinations to climinate the inherent instability and randomness of few-shot setup (Gao et al. 2020 Ajith et al., 2023). We the average correctness across demonstration combinations of the prompt-query pair the fine-graine seee-griied query-level reward in few- shot setting, which is more stable than PPL. As the query-level rewards are supposed to accurately reflect the distinctions in prompting effectiveness for various prompts across different queries, both the perplexity penalty in zero-shot setting and the averaged correct in few-shot setting, which are decimals rather than binary integers, exhibits greater variability across different query-prompy pairs compared to the binary correctness, thus enable a more fino-grained alignment:
1 (PL()( Zero-Shot, 10
-
Roquery ( (p(i))
1 K
Few-Shot.
The overall reward the sum of the query-level and task-level reward. Due to significant differences in reward scales across different tasks, we normalize the overall reward using the Min-Max Normalization method Patro 2015) for convenience, mapping it to the range of 0-100.
R(q (1) p())) Rquery (" (())) + Reast(g (()))
(4)
Training Objective. To enable the model to output appropriate prompts based on expected rewards and input questions, we maximize the likeliicl with teacher forcing through DT training paradigm:
Pprompt = -B log p(pr(p))
(5)
where inpul reward as return-to-go and the complete query as stale, and allow the policy model to autoregressively predict the entire prompt's tokens an action.
To further enhance the model to distinguish the prompting abilities of different prompts for the various queries, we introduce the reward prediction loss. We directly predict the expected reward based on the output of the transformer at the first token, regarding the model as an autoencoder (Liou et al. 2014) by encoding an reconstructing the input reward. This approach can extract more information from the given reward, impose constraints on the model weights, and significantly reduce training difficulty compared to predict reward based on the output prompt at the last token. Further discussion implementing autoencoder functionality is illustrated in Appendix A.4
f, = E ((g.))) - r2)²
(6)
We define the overall objective function by combining the above objectives with a balancing hyperparametor:
C Sprompt + XL,
Model Architecture. To achieve the aforementioned functionality, enabling the model to fully leverage reward information and temporal dynamics in reinforcement learning, also refine the architecture of the pro-trained language model, clevating it to serve as our policy model. We introduce distinct reward embedding layer, separate from the natural language token embeddings, to facilitate deeper comprehension of reward signals by the model. Furthermore, during training, we incorporate an additional reward prediction layer implement the reward prediction loss, thereby imposing constraints on the optimization of the model's primary network. The combined enhancements in both algorithm and model structure culminate in the successful implementation of offline prompt optimization.
5

---

Published in Transactions on Machine Learning Research (02/2025)
2.3 QPO Framework for Multi-Loop Augmentation
2.3.1 Initial Demonstration Construction
Dataset Construction. We start by emphasizing the presence and significance of prompt demonstrations generated by previous research of prompt optimization. In the domain of prompt engineering, abundant prompts have been proposed to improve the LLMs' performance on downstream tasks. Generally, their efficacy is assessed at the task level. However, different queries exhibit varied preferences for different prompts. Query-level score, obtained before calculating task-level score but ignored by previous research, can play a great role in teaching the query-dependent prompting ability. As the prompts have been evaluated on open-sourced datasets, the following query-level demonstrations can be constructed as by-products. The process for constructing the offline dataset is as follows: first, we sample queries from the task datasct, then combine them with the collected prompts and input the query-prompt pairs into the target LLM to obtain answers. Through the designed reward function, we calculate the reward value for each query-prompt pair. These query- preey--rompp-rewaatd triplets form the samples in our offline dataset:
D = (1) (p))) (1), g(اا)) R(y" ( HENN)NEMM)
where M, N denole the number of collected prompts and queries, respectively. To enrich the initial explo- ration of the prompt space and simplify prompt acquisition, we leverage ChatGPT-3.5 to rewrite prompts, resulting in a diverse vocabulary and yielding a substantial number of prompts in our dataset. Details of rewriting prompts are demonstrated in Appendix B.2, While as each query requires testing with all prompts, we only utilize a small collection of queries as the initial dataset. to strike a balance between algorithm per- formance and computational cost.
Data Filtering. Similar to popular offline reinforcement learning's need to differentiate expert, medium, and random datasets, in this work, simply remove the less-quality examples with rewards below the expert threshold, defined as the 66.7th percentile of the reward range in the dataset. Notably, this approach does not discard two-thirds of the data since we use rewards as the filtering metric rather than data volume, and most samples fall within the expert interval. Almost all prompts are retained without wastage; even a prompt performs poorly on average, may still enable the LLM to answer correctly on specific questions, and it is retained at this query. The specific data wastage resulting from data filtering is detailed in Section 3.3.
2.3.2
Multi-Loop Augmentation
After fine-tuning with the initial dataset, the policy model is supposed to generate prompts that are better suited for specific problems than any existing prompts in the dataset. Therefore, a natural and efficient approach is to use these prompts to enrich the dataset as a form of data augmentation. Moreover, as the model generates high-qualily prompts tailored to specific queries, there is little concern about filtering out evaluation samples and wasting data, which presents a great opportunity extensively assess various queries, expanding exploration of the query space.
Query Augmentation. As our initial dataset covers only a limited number of queries, we randomly collect both overlapped and uncovered queries from the task's training set, simultaneously focusing on optimizing prompts on existing problems and exploring unknown ones. Since only one specifically generated prompt needs to be evaluated for each problem, rather than testing all prompts, it significantly reduces computational overhead caused by the number of queries and candidate prompts. Hence, it enables large-scale exploration of new problems, avoiding the test performance degradation caused by the distribution shift of the limited initial training queries.
Prompt Augmentation. Based on the newly collected queries, we utilize the maximum expected reward to instruct the fine-tuned model to generate novel and superior prompts. In order to guarantee the quality and diversity of prompt exploration, we adopt the sampling generation approach with the policy model in this phase.
Dataset Augmentation. For augmented query-prompt pairs, the calculation of the query-level rewards remains consistent with the original formula (3) while task-level rewards are determined by the model's
6

---

Published in Transactions on Machine Learning Research (02/2025)
overall generation capability in the current loop, that is, the average accuracy of all prompts generated in the current augmentation stage. Then, we obtain the new training examples:
DA = {a(() p
p()
he(
where N' denotes the number of new queries. This datasct is finally supplemented into the original dataset for the next loop's training.
After data augmentation, the updated dataset is enriched with a greater variety of queries and higher-quality prompts, which can be utilized to fine-tune the model to further improve its performance. We structure these processes into a continuous bootstrapping loop, where both offline RL and data augmentation cyclically feedback to each other. The algorithm stops when the number of iterations reaches a predefined value, Our method achieves incremental improvements utilizing offline optimization with significantly minimal interactions with LLMs. The detailed algorithm of QPO are outlined in Algorithm in Appendix A.5
3 Experiments
3.1 Experimental Setup
Tasks. We perform experiments on 6 language understanding tasks and 2 math reasoning tasks to validate
our methods, including topic classification (AG's News (Zhang et al.] 2015)), 5) natural language inference
(BoolQ (Clark et al. )) sentiment classification (IMDB (Maas et al.
2011 TveetEval Emotion (Mo-
hammad et al. 2018)), ), multi-choice QA CA A (Huang et al.) 2019).
HellaSwag (Zellers et al., 2019)).
and math reasoning (GSM8K (Cobbe et al., 2021). SVAMP (Patel et al.
2021 ) These tasks are widely
studied in prompting scttings, and hence many expert-craftod and machino- -egiiinted prompts are available, which facilitates our oflline data collection procedure.
Baselines. We compare our method with three types of baselines, including manual prompt engineer- ing (PromptSource (Bach et al. 2022). Chain-of- Thought (Wei et al. 2022 online prompt optimization (Low Perplexity (Gonen et al. 2022). RLPrompl (Deng et al.] 2022), APE (Zhou et al. 2022) and oflline prompting approach (Prompt-OIRI) (Sun et al. 2023 ). We also compare the prompts rewritten by Chat- GPT. We reproduce the online methods based on InstructEval (Ajith et al. 2023). Rather than making a direct while unfair comparison between the performance of QPO and newer online algorithms, our primary focus is to demonstrate that QPO can further enhance performance by optimizing prompts at query-level based the prompts obtained through online optimization. Notably, for fair comparison, use a larger dataset in Prompt-OIRL than ours after final loop data augmentation to eliminate our advances in additional interactions.
LLMs. Our method is model-agnostic for policy model, and we use GPT-2 Radford et al. 2019) in this paper, which is compact while has sufficient text generation capability. For the larget LLMs, we use publicly available Llama2-7b-chat (Touvron et al., (003)) for natural language understanding tasks, while for the more challenging math reasoning tasks, opt for GPT-3.5-turbe (OpenAI 2023) and GPT-4o (OpenAI] 2024). Moreover, to evaluate the cross- moos--moeel generalization ability of our trained policy model, we employ models at different abilities, scaling from the GPTNeo-1.33 (Black et al. 2021 to TJama2-13b-chat (Touvron et al. 2023) and Vicuna-13b (Zheng et al. 2024).
Implementation Details. For the initial data collection, we utilize 30 expert prompts obtained from EnstructEval and 120 prompts rewritten from ChatGPT-3.5 to construct the dataset. We set the QPO with 3 loops. For all the tasks, we use a unified hyperparameter set and do not need complex hyperparameter design for specific task. We do not cherry-pick checkpoints and directly use the final checkpoint in each loop for evaluation and next loop's training. For NLU tasks, evaluate our method and few-shot settings, while for math reasoning tasks for GPT-3.5 and GPT-4o, we only test its zero-shot performance due to budget limitation. Both training and testing are conducted on 3 seeds. We set the maximum expected reward as 100 and pick the model with the highest score on the development set and report its score on the test set. For data augmentation, we adopt sampling generation with a top-k of 2 and top-p of 0.9, while for evaluation, we adopt greedy generation. Few-shot evaluations are conducted separately for 3-shot and 6-shot, where the shw-shot demonstrations are randomly sampled from the training set of the
both zero-shot
7

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: Main results (accuracy) QPO and baselines on Llama2 on NLU tasks.
Method
AG News
BoolQ
IMDB
Emotion
CosmosQA
HellaSwag
Avg.
Zero-shot Accuracy
PromptSource
68.3
59.0
86.3
57.3
42.9
68.8
63.8
ChatGPT
64.2(0.84)
62.0(2.41)
85.0(2.19)
62.4(2.68)
42.6(0.51)
68.8(0.51)
64.2
Low Perplexity
(0.3(
(1.0(2.65)
(8.7(2.21)
57.6(1.11)
(2.0(0.67)
69.6(0.69)
63.2
RLPrompt
47.8(4.67)
(4.9()
86.4(2.04)
49.8(1.17)
(4.7(0
68.6(0019)
60.4
APE
66.5(1.12)
64.5((150)
(7.0(3.84)
(2.7()
(3.7(1.7)
68.3(0
62.9
Prompt-OIRL
78.3
65.0
89.0
68.3
42.3
68.3
68.5
QPO
(9.4()
(1.0().02)
(2.2().12)
68.7(1.07)
(5.4)
69.7(0.71)
70.9
Few-shot Accuracy
PromptSource
83.3
67.1
89.4
66.4
45.6
70.3
70.4
ChatGPT
83.8().75)
(8.19)
(9.2()
(1.7().78)
45.5(0.62)
69.8(0.52)
70.6
Low Perplexity
82.9(0:20)
(8.5(0)47)
89.1 (0.09)
67.6(1.49)
45.4(0.41)
70.4(0.13)
70.7
RLPrompt
82.8(0.24)
(1.2(00.0)
(9.1(00544)
(5.7(0
46.2(0032)
70.2(0.57)
70.8
APE
83.1 (0.3()
68.4 (8.4().79)
(9.2()
(0.9()
45.8(0.70)
70.2(0.12)
70.8
Prompt- OIRL
83.8
73.3
89.5
67.6
48.1
70.4
72.1
QPO
85.6(0.29)
(6.6()
91.4(0016)
69.1(1.10
(1.2(
71.0(0.29)
74.2
Table 2: Main results (accuracy) of QPO and baselines on GPT-3.5 on math reasoning (MR) tasks.
Method
GSM8K
SVAMP
Avg.
CoT
88.0(0.49)
81.0(0.52)
84.5
ChatGPT
82.1(1.07)
79.5(2.24)
80.8
APE
85.8(0.74)
(1.3(
83.6
Prompt-OIRL
87.5
83.7
85.6
QPO
90.3()
86.8 (0.8().41)
88.6
tasks. We measure the algorithm's performance using accuracy, defined as the ratio of correctly answered queries by the LLM to the total number of tested queries. More details are demonstrated in Appendix B.1 The code is available at here
3.2 Main Results
Natural Understanding tasks. Table 1 presents comprehensive comparison of query-dependent opti- mized prompts generated by QPO against human prompts, query-agnostic prompt optimization methods, and SOTA offline query-dependent prompt selection method on alama2-7bbchhl on zero-shot and 6-shot set- tings. The results show that: (1) QPO delivers significantly better results on zero-shot metric (avg. +7.2% over other baselines) and also outperforms other basclines on few-shot setting (avg. +-33%), where the con- textual demonstrations slightly diminish the critical role of prompts, as evidenced by the standard deviation in few-shot scenarios being significantly smaller than in zero-shot scenarios. The 3-shot results in Table [I further illustrate that algorithm's 's advantage becomes
pronounced as the information provided by contextual demonstrations decrcases (avg. +6.1%). (2) Based on the dataset constructed from these online algorithrss, both QPO and Prompt-OIRL exhibit belter performance to those optimized at the task level on all three settings (avg. +7.2%, +2.3%, +4.9%, respectively), which underscores the effectiveness of optimiz- ing prompts the query level. (3) Despite the comparative strength of Prompt-OIRI, as discriminative model over our generative model, our algorithm outperforms Prompt-OIRL. demonstrating that learning generate optimal prompts for specific queries is superior to merely selecting the most suitable ones.
8

---

Published in Transactions on Machine Learning Research (02/2025)
Math Reasoning tasks. The experimental results of GPT-3.5 zero-shot math reasoning tasks are presented in Table 2 When applied to more powerful models for solving complex tasks, QPO exhibits the same leading capabilities. Specifically, QPO achieves up to 7.8% improvement with an average of 5.0% over other baselines, which demonstrates that QPO is a good mothod for prompting fancy LLMs for these challenging lasks. The results of GPT-4o are shown Appendix C.2
3.3 Analysis
Multi-loop augmentation can bootstrap to improve performance in interaction-cost- efficient way. Multi-Loop Augmentation (MLA) in our method is supposed to improve the exploration of the query space and prompt space efficiently, so that the augmented dataset can enhance the policy model in turn. The average number of queries in datasets
Averaged Performance
0.875
0.825
Accuracy
increases from 283 in original dataset to 673 in the final augmented dataset, and the average number of prompts rises from 150 to 829, which obtains 138%
0.000
0.775
Llama2-NlU-25
Jaam2--UU NLU-25 w/o MLA
and 453% increases in question coverage and prompt diversity with only a 17.1% increase in total data vol- ume. We limit the amount of data augmented in each loop, as research (Wang et al.) 2024c) indicates that gencrated data comprising 10% of the real data pro- vides the maximum training benefit, whereas exces- sive generated data can be counterproductive. To fur- ther quantify the effect of MLA, we conduct ablation
Llama2--LL-FS- MLA
0.725
GPT3.5 MR 25.25
GPT3.
Loops
Figure 3: Comparison of QPO and QPO w/o MLA.
experiments that finetune the model the same iterations without data augmentation. As shown in Figure 3. in 3 NL. tasks (AG News, BoolQ, IMDB) and 2 math reasoning (MR) lasks (GSM8K and SVAMP), MLA can consistently improve the performance in all settings, while training with more epochs without data augmentation results significant drops due to overfitting. To be specific, multi-loop augmentation and training lead to a performance improvement of 6.7% and 2.3% for Llama2 in zero-shot and few-shot set- tings in NLU tasks, respectively; while in the zero-shot math reasoning tasks for GPT-3.5, the performance improved by 3.5%.
Despite the introduction of online LLM interactions in MLA, our method: (1) only engages new interactions to supplement the datasot after the training convergence in cach loop, rather than requiring interactions at each step to provide real-time guidance for optimization direction; (2) requires significantly fewer inter- actions compared to other online methods. Therefore, QPO is considered an offiine approach. To demonstrate this, we pared our method with the classic online algorithm APF, which is & paradigm for many online prompt optimization methods. Table 3 presents the LLM inference time costs, which is operated locally on NVIDIA V100 GPU. To be specific, for loops of training, 3 data aug- mentation processes are employed, resulting in each MLA requiring approximately 0.097h of computational resources. Given an average performance improvement of around 2% per loop, this expense for interaction is relatively justified and valuable. Thus, employing QPO for prompt optimization is substantially more cost-efficient than methods reliant on LLMs critics. We analyse the specific number of required interactions for these methods in Appendix C.3
Table 3: Interaction costs of QPO and APE on Llama2- 7b-chat on AG News.
Method
Model Cost (GPU Hour)
APE Ours
1.761
0.291
Reward Matters. To validate the effects of RL and our proposed reward prediction loss, we conduct ablation experiments comparing supervised fine-tuning (SFT), RL, and QPO (RL+Reward Loss). Table ! presents that Rewadd Loss consistently surpasses others in all evaluation tasks, which confirms the effects of the additional reward loss. instruct the model focus more on the rewards, enabling to de- termine which query-prompt pairs can achieve higher rewards, allowing the policy model to generate optimal
9

---

Published in Transactions on Machine Learning Research (02/2025)
Table 4: Comparison between SFT, RL and QPO (RL+Reward Loss).
AG News
BoolQ
IMDB
Avg.
Method
zero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
SFT
76.0
84.2
70.7
74.9
90.7
89.6
79.1
82.9
RL
78.7
84.3
65.0
75.2
91.0
89.6
78.2
83.1
QPO
79.0
85.6
71.0
76.5
91.2
91.4
80.4
84.5
prompls given the maximum expected reward. Compared with SFT, RL also achieves better performance (5/6) because of the high data utilization, which is also thanks the introduction of the reward. Detailed reason is analysed in Appendix A.1 The reason for the drop between QPO and RL can be attributed to the underestination of rewards. As the reward and each query token have equal weight contributing to the prompt generation, excessively long queries may cause the model to overlook the single-token reward. No- tably, SFT also yields strong results, indicating that fine-tuning PLMs to generate optimal prompts can serve as a general paradigm, regardless of the specific fine-tuning method. The success of SFT also implics that each query does not necessarily need multiple prompts the dataset; as our data augmentation approach, to balance interaction frequency with query exploration, each new query might only be accessed minimal times, rosulting in a limited number of different corresponding prompts generated by sampling. Such augmented data can still positively contribute to the training process. For more details on ablation experiments related to reward design, including the contributions of Rquery and Rtask, the importance of fine-grained perplexity penalty in Rquery: and the difference between reward predicting positions, please refer to Appendix C.4
Table 5: Performance comparison between Nearest Neighbor and QPO.
AG News
BoolQ
IMDB
Avg.
Method
zero-shot
few-shot
zero-shot few-shot zero-shot
few-shot
zero-shot
few-shot
NN
70.3
82.4
64.3
76.7
83.7
89.1
72.8
82.7
QPO
79.0
85.6
71.0
76.5
91.2
91.4
80.4
84.5
Ablation on Prompt Generation. To further validate the rationality of using transformer for query encoding and prompt generation simultancously, additionally employ direct query-dependent bascline for comparison. Specifically, we ablate the text generation component of the policy model, only utilize it to encode queries into query embeddings, select the training query most similar to the testing query using Nearest Neighbor (NN) criterion, and directly choose the prompt with the highest reward associated with the matched training query. The results are depicted in Table 3 Generating specific prompts (QPO) significantly outperforms merely selecting suitable ones (NN), indicating the effectiveness of the main Training objective. Notably, NN also achieves higher scores than those query- prompts shown in Table [ which illustrates that the trained policy model can accurately obtain the input query embeddings, even without the specific training objective for encoding. And query-specific prompts, whether selected generated, indeed outperform query-agnostic prompts.
Ablation on Dataset. We investigate the effect of initial prompt quantity and overall data quality in the offline dataset on QPO. Firstly, we compare the performance with different num- bers of prompts shown in Table 0 where the 30 prompts are from online algorithms without ChatGPT rewriting and the 5 prompts randomly selected from the 30 prompts. The results reveal an overall trend that more prompts Icad to better fi- nal performance, while an exceptionally large number of initial prompts is not necessary, as a moderate amount can already yicld good results. This because once the datasct contains sufficient number of high-quality prompts, subsequent data aug- mentation phase can also effectively enhance prompt diversity.
Table 6: Performance under different prompt availability. "# prompts' demonstrates the number initial prompts used for training.
Avg.
# prompt
zero-shot
few-shot
scarce (5)
72.8
83.1
middle (30)
77.8
84.2
rich (150)
80.4
84.5
10

---

Published in Transactions on Machine Learning Research (02/2025)
It also validates the effectiveness of supplementing prompts with lower-quality GPT-rewritten prompts, prompt with lower average accuracy can still perform well on specific queries. This experiment demon- strates QPO's potential to perform well with a smaller number of prompts. Detailed results are shown in Table 16
Next, we discuss the impact of data quality on our method to illustrate the importance of data filtering. We view the filtered dataset expert dataset and the original dataset medium- expert one, for all the prompts within the dataset capable of achicving at least moderate performances. Table 7 demon- strates that QPO on expert dataset after data filtering out- performs that on medium-expert dataset. By filtering 34.4% of the lower-quality data, our method achieves a 2.1% per- formance improvement. While when trained on an unfiltered medium-experl dataset, our algorithm still achieves compara- ble or better results to the SOTA Prompt-Olie OIRL, which demon- strates that QPO exhibits strong robustness to prompt quality and data filtering can further enhance its performance. The detailed results are presented in Table 18
Table 7: Performance under different data quality. QPO with medium-expert dataset can already achieve comparable better results to SOTA Prompt-OIRT.
Avg.
Dataset
zero-shot
few-shot
expert medium-expper expert
80.4
84.5
77.9
82.6
Table 8: Cross-model generalization on different models.
AG News
BoolQ
IMDB
Model
Method
# wins
zero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
PronptSource
58.5
63.0
50.3
49.8
80.9
82.9
GPTNeo-L.3b
Prompt-OIRL
65.3
65.1
48.3
49.0
78.3
84.0
QPO
61.7
63.4
50.3
49.1
83.3
87.5
PromptSource
75.1
83.7
79.1
82.8
89.8
92.2
LlaMa2-13b-char
Prompt-OIRL
80.3
78.0
83.0
88.3
91.2
QPO
80.3
84.4
80.7
83.9
90.3
92.1
PromptSource
74.3
80.3
79.4
80.8
89.5
92.2
Vicuna- 13b
Prompt-OIRa
76.0
81.4
82.7
84.1
88.3
92.5
QPO
75.0
82.6
86.0
84.6
87.3
92.9
Cross-Model Generalization. We investigate whether QPO can be used to steer the target LLMs which are not involved in the dataset collection and augmentation. In this section, the policy model are trained on datasets collected by Llama2-7b-chat. As shown in Table QPO demonstrates excellent cross-model transferability compared with human designed prompts, which are generally considered high-quality and can be employed on any LLMs. Compared the results with GPTNeo-LL31, our algorithm demonstrates greater advantages on two larger LLMs (3 wins v.s. wins & 4 wins). This mooel generalization significantly enhances the value of our approach: We can train the policy model using existing data tested different models and then apply it to the target model. Additionally, we can collect data using cost-cffective smaller LLM and train the policy model, then directly apply it to a more expensive larger LLM.
Case Study. The prompts generated by QPO in NLU tasks are presented Table 19 which showcase the versatility of our approach. Notably, while their individual accuracies on all testing questions are relatively low, their collective application significantly euhances task-level performance when aligned with specific queries. Although single generated prompt may not be suitable for all queries, it is certainly the best for the specific query.
4 Related Works
Prompt Optimization. Automatic prompt optimization has become a pivotal issue in domain of LLMs. Recently, there has been growing interest in this area. Soft prompts (Tá & Liang 2021 Zhang et al.] 2021
11

---

Published in Transactions on Machine Learning Research (02/2025)
Lester et al. 2021 Liu et al. 2023b] achieve effective performance by tuning the parameters of the imput tokens, while exhibit two significant drawbacks. They require parameters of the target LLM,, which are inaccessible for black-box APIs, and their vector representation lacks interpretability for humans. Discrete prompts, also known as prompt enginecring, climinate the shortcomings and also achieve great success. Most of these methods are query-agnostic. Automatic Prompt Engineering (APE) (Zhou el al. 2022) induces prompts by instructing a LLM to describe the given task and refines the set of generated prompts. Low Perplexity (SPELL) Gonen et al. 2022) finds the perplexity is negatively correlated with the performance and select the least perplexity instructions. RLPrompt (Deng et al., 2022) employs RLL to optimize a prompt policy. Recently, more studies focus on leveraging L.M as prompt optimizer using black-box optimization
method for swift prompt engineering, such as evolution algorithm (Guo et al. 2023
Fernando et al.
2023
alternale sampling (Zhou el al., 2022 Xu et al. 2023 Pryzant et al. 2023 Kong el al. 2023), and RL
Kong
et al. 2024 Wang et al,, 2023b). However, studies (Wu et al. 2022 Jiang et al. 2022
Zhang et al. 2022;
Sun et al. 2023) have shown that query-dependent prompts can obtain better results. TEMPERA (Zhang
eet al. 2022) designs the action space to be editing operations to achieve test-time query-dependent edit.
Prompt- OIRL (Sun et al. 2023 utilize inverse RL to learn praxy reward model and thus select the best prompt based the query from candidate prompt set. Another advantage of Prompt-O1RL is it makes full use of offline dataset, significantly reducing the inference cost by interactions with LLMs. Promptist (Hao et al. 2023) also collect an offline dataset and employ SFT to fine-tune the policy model, while the main performance improvements are from the subsequent online RL.
Learning from Human Feedback. Our method is related the research on reinforcement learning from human feedback, which has shown promising results
wide range of tasks, including NLP (Rajpurkar
ct al. 2018 Wang ct al. 2023a; Ouyang ct al. 2022; Stiennon et al.] 2020 Klie et al.) 2020; Wang et al. 2024a and classical RL (Christiano et al. 2017 Ibarz et al. 2018 Xia et al. 2024). Another similar category is reinforcement learning form AI feedback (RLAIF) [Bai et al. 2022 Lee et al. 2023), which leverages the feedback typically from LLMs to facilitate RI. Differently, we directly use the feedback from LLM to shape rewards rather than learning a reward model. Our method is akin to imitation learning (Torabi et al., 2018) Chen al.] 2021 Janner al.] (22T), directly learning a policy from batch of behavioral demonsttratins and their corresponding rewards. This distinctive approach sets our method apart from existing prompt optimization techniques.
5 Conclusion and Future Work
We propose QPO, a novel query-dependent prompt optimization method through a multi-loop offline RL framework, which is cost-efficiett. This method leverages offline datasets from existing evaluations and employs offline RL to finc-tune a small-scale language model as the policy model, generating query-level optimal prompts. Experimental results demonstrate the superiority and cross-model generalization ability of QPO, significantly enhancing the value of our method. Besides, we conduct extensive ablation studies to analyse our method's insights and rationale. While our method has substantially reduced the required online interactions, we can entirely eliminate these interactions if necessary. For instance, in certain business scenarios where only fixed amount of reward-labeled data is available and the reward labeling rules are unknown, it is impossible to use LLMs to obtain new labeled data our exploration. We can employ inverse reinforcement learning to derive reward model from the original dataset to score the new query-prompt pairs obtained through the multiple-loop data augmentation. This approach completely eliminates the need for online interactions with LLMs, which we consider for future work. Furthermore, in the domains of text-to-image and text-to-video generation, where LLMs have slower inference speeds, interactions are more expensive, and user queries are more unique, our offline query-dependent prompt ipiimazaioon methods will show greater potential and advantages.
6 Broader Impact Statements
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
12

---

Published in Transactions on Machine Learning Research (02/2025)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ngee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprinl arXiv:2303.0877 08774, 2023.
Anirudh Ajith, Mengzhou Xia, Ameet Deshpande, and Karthik R Narasimhan. Instructeval: Systematic evaluation of instruction selection methods. In R0:FoMM: Robustness of Few-shol and Zero-shol Learning in Large Foundation Models, 2023.
Stephen H Bach, Victor Sanh, Zheng-Xin Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. Promptsource: An integrated development environment and repository for natural language prompts. arXiv preprind arXiv:2202. 0120022..22. 01279, 2022.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint ar arXi::2912.00073 2022.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata, 58: 2, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Loe, Aditya Grover, Misha Laskin, Picter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neurol information processing systems, 34:15084 15097, 2021.
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in neural information processing systems, 30, 2017.
Christopher Clark, Kenton Lee, Ming-Wey Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. ar preprint WVXiv:195.100444 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ar preprint arXiv:21100 10.02.1000000 14168, 2021.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3369-3391, 2022.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Prompl- breeder: Self-referential self-improvement via prompt evolution. arXiv preprint (xXi:2339.16690 2023.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint Xxxx:2012.15722 2020.
Hila Gonen, Srini Tyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. arXiv preprint 2rXic:2912.04039 2022.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of gpt-3.
arXiv preprint Xrxi:22009. 12356, 2022.
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang, Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Leurning Representations, 2023.
Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading compre- hension with contextual commonsense reasoning. Xiv preprint wpX::1900000000 2019.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geofirey lrving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations atari. Advances neural information processing systems, 31, 2018.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modcling problem. Advances in newral informalion processing systems, 34:1273 11117-1866,, 2021.
Yuezihan Jiang, Hao Yang, Junyang Lin, Hanyu Zhao, An Yang, Chang Zhou, Hongxia Yang, Zhi Yang, and Bin Cui. Instance-wise prompt tuning for pretrained language models. arXiv preprint or arXiv:2206. 01 958, 2022.
Jan-Christoph Klic, Richard Eckart De Castilho, and Tryna Gurovych. From zero to hero: Human-in-the- loop entity linking in low resource domains. In Proceedings of the 58th anmual meeling of the association for computational linguistics, pp. 6982 8982-6993, 2020.
Weize Kong, Spurthi Amba. Hombaiah, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Prewrite: Prompt rewriting with reinforcement learning. arXiv preprint arXic:2/01.088 2024.
Yilun Kong, Jingqing Ruan, Yihong Chen, Bin Zhang, Tianpeng Bao, Shiwei Shi, Guoqing Du, Xiaoru Hu, Hangyu Mao, Ziyue Li, et al. Tptu-v2: Boosting task planning and tool usage of large language model-based agents in real-world systems. arXiv preprinl arXiv:3311.13115 2023.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Car- bune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.0022 2023.
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-cfficient prompt tuning. arXiv preprinl orXic:2144.0809) 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXie preprint arXiv:2005.014 01643, 2020.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.001 2021.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Decpak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint orXiv:2211.0911 2022.
Cheng- Yuan Liou, Wei-Chen Cheng, Jiun- Wei Liou, and Daw-Ran Liou. Autoencoder for words. Neurocom- puling, 139:84-99 96, 2014.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1 35, 2023a.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AT Open, 2023b.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pp. 1-17, 2018.
14

---

Published in Transactions on Machine Learning Research (02/2025)
OpenAL. Gpt-3.5-turbo. 2023.
OpenAl. 4pt-
2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 27744, 2022.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are ulp models really able to solve simple math word problems? arXiv preprint arXit:2103.00 07191, 2021.
S Patro. Normalization: A preprocessing stage. arXiv preprinl XrXin: 06462, 2015.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with' gradient descent" and beam search. arXiv preprint arXiv:2905.0044 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nly Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. erXiv preprint arXiv: 186.0820, 2018.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Aduances in Neural Information Processing Systems, 33:3008-3000 3021, 2020.
Hao Sun, Alihan Hüyük, and Mihacla van der Schaar. Query-dependent prompt evaluation and optimization with oflline inverse rl. In The Twelfth International Conference on Learning Represenlations, 2023.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. ar preprint arXiv:1805. 01954, 2018
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXi:2200. 18971, 2023.
Haoyu Wang, Guozheng Ma, Cong Yu, Ning Gui, Linrui Zhang, Zhiqi Huang, Suwei Ma, Yongzhe Chang, Sen Zhang, Li Shen, al. Are large language models really robust to word-level perturbations? arXiv preprint ar Xix:22300 1100099.11110 11166, 2023a.
Haoyu Wang, Guozheng Ma, Ziqiao Meng, Zeyu Qin, Li Shen, Zhong Zhang, Bingzhe Wu, Liu Liu, Yatao Bian, Tingyang Xu, et al. Step-on-fect tuning: Scaling self-alignment 1lms via bootstrapping. arXiv preprinl ar Xiv:22402.00 07610, 2024a.
Ruochen Wang, Sohyun An, Minhao Cheng, Tianyi Zhou, Sung Ju Hwang, and Cho-Jui Hsich. One prompt is not enough: Automated construction of mixture-of-expert prompts. arXie preprint (.:::2207.00256 2024b.
Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric Xing, and Zhiting Hu. Promptagent: Strategic planning with language models enables expert-level prompt optimization. In The Twelfth Internotional Conference on Learning Representations, 2023b.
Yifei Wang, Jizhe Zhang, and Yisen Wang. Do generated data always help contrastive learning? arXiv preprint XrXi::2403.12. 12448, 2024c.
Jason Wei, Xuczhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. thouugt prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24883777--2877,, 2022.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, VG Vydiswaran, and Hao Ma. Idpg: An instance-dependent prompt generation method. arXiv preprint or arxi::2004 04497, 2022.
Bo Xia, Yilun Kong, Yongzhe Chang, Bo Yuan, Zhiheng Li, Xueqian Wang, and Bin Liang. Deer: A delay- resilient framework for reinforcement learning with variable delays. arXiv preprint arXiv:2406.031 031000 2024.
Weijia Xu, Andrzej Haaburski-Faheyy and Nebojsa Jojic. Reprompting: Automated chain-of-thought prompt inference through gibbs sampling, arXiv preprint vrXio:2305.09000 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprinl Xiv: 07830, 2019.
Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. ar preprinl arXiv:2108. 131611 2021.
Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph Gonzalez. Tempera: Test- time prompt editing via reinforcement learning. In The Eleventh International Conference on Learning Represenlations, 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neurol information processing systems, 28, 2015.
Boyuan Zheng, Sunny Verma, Jianlong Zhou, Ivor W Tsang, and Fang Chen. Imitation learning: Progress, taxonomies and challenges. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Newral Information Processing Systems, 36, 2024.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Represcnlations, 2022.
16

---

Therefore, we do not introduce additional proxy rewards and directly utilize the sparse rewards. At. this point. instead of modcling the prompt generation task multi-step mecision-making process {r, So,P1, r, 81; P2,...; where each r is the same sparse reward and additional methods are required to encode long texts (g1..., In. pl..... P:) into the single token state s;, we simplify the task modeling based on the principle of Occam's 's Razor. Thus, the prompt generation is formulated into a single-step decision process (r, P) Notably, the single-step decision-makinn process refers to the reward and state, taking only one action (i.e., generating the prompt), while the inner prompt generation process remains an autoregressive prediction Published in Transactions on Machine Learning Research (02/2025)
A Extended Discussion on QPO
A.1 Advantages of offline RL compared to SFT
Supervised fine-tuning (SFT) can be regarded as form of imitation learning (Zheng et al. 2022 similar to behavior cloning (Torabi et al. 2018), where it mimics expert actions (iee, the query-specili optimal prompts) based on given states (i.e., input questions). While offline reinforcement learning introduces reward values, guiding the generation of prompts based on the given queries and associated rewards. In prompt generation task, directly employing SFT results in low data utilization because it is unknown in advance that which prompt is optimal for specific problems. This necessitates massive prior evaluation of all prompts before selecting the best one as the expert behavior to construct the dataset, leading to significant resource wastage. Specifically, in the final SFT dataset, each query should ideally have only one prompt label. All M prompts must be evaluated but only one optimal prompt is retained as the label for SFT ultimately, wasting the remaining (M - 1)/M evaluation data. While for offline RL, the prodiction of prompt is based on both reward and query, leading to a much more diversity in input. Since different prompts have various rewards, most of them can be utilized for training. Offline RL offers significant advantages through the introduction of the rewards, which allows each query to be associated with multiple prompts and their corresponding rewards in the dataset. On one hand, the rewards improves the utilization of collected suboptimal prompts for training, significantly boosting exploration of the prompt space while avoiding resource wastage caused by prompt selection. On the other hand, rewards in RL enable the policy model to distinguish the prompting abilities of different prompts for various queries at the token level, leading to the generation of novel prompts for unknown queries.
A.2 Prompt Generation as Single-step Decision Process
In the reinforcement learning setup, prompt generation a sparse reward task. The complete prompt is evaluated with the target LLM to obtain a reward value only after it is fully generated. Given the availability of substantial amounts of such test data, offiine reinforcement learning is an effective approach for prompt optimization. For offline RL, there are two methods for collecting token-wise rewards which can be used for multi-step decision-making processes:
(1) Manually designing token-level rewards, which not only requires extensive prior expert knowledge but also introduces additional errors, making unreliable.
(2) Itcratively delcting the prompt's tokens from end to beginning and ro-evaluating to obtain rewards from 8m-1 (q.Pi,..PPm-i) to 8m = (q..P...,,P)), where Pm denoles the m-th token in the complete prompt. However, this approach significantly increases the cost of collecting datasets and is thus impractical.
p ~ II, (Pelr, q, P<.). The single-step decision process also resembles the offline contextual bandit setup, actions do not interact with the environment.
A.3 The Architecture of Policy Model
Compared to the standard GPT-2 model with 124M parameters, we make three modifications to the model architecture: a reward embedding layer, a reward prediction layer, and a timestep embedding layer. Both the reward embodding layer and reward prediction layer are single-layer MIPs at the first token of the policy model. The reward encoder into reward encoding through the reward embedding layer, then input as the first token into the transformer to obtain the output at the first. token position, which is then passed through
17

---

Published in Transactions on Machine Learning Research (02/2025)
the reward prediction layer to generate the predicted reward. This process can be viewed as autoencoder designed to extract more information from the reward. The additional timestep embedding layer encoding three timesteps into three embeddings, which are added with reward embedding, query token embedding, and prompt token embodding, respoctively. This helps the policy model better learn the positional relationships and distinctions belween reward, question and prompt.
A.4 Autoencoder with reward prediction loss
Autoencoder encodes the input into low-dimensional latent state through an encoder and subsequently learns to reconstruct the original input from this low-dimensional latent state via a decoder. In this process, the latent state can effectively represent the original input. In our approach, the reward embedding layer and the transformer map the reward value into hidden state. During training, the additionally introduced reward prediction layer acts as a decoder predict the original input reward based on this hidden state, and the encoder-decoder is updated through the reward prediction loss. In the inference phase, no longer utilize the reward prediction layer (decoder); instead, we leverage the hidden state (related to the reward) obtained from the encoder to perform another task, that is, predicting the prompt. This process mirrors the autoencoder's principle, thus introducing the reward prediction loss and the reward prediction layer into the PLM essentially makes serve as an autoencoder. Notably, the reward prediction loss does not directly contribute to reinforcement learning process, as optimization objective is not to generate actions.
A.5 Detailed Algorithm
Algorithm Query-Dependentt Prompt Optimization
Require:
Initial dataset Do, a collection set for task querics Q, the mumber of loops T, a pre-trained language model mas O-L(-) denoles the oflline RL fine-tuning process, R(-) presents evaluating the new query-prompt pairs on the target LLM and calculating the reward. Offline RL: マ < ORL(To, Do)
for t = 1 to T T do
3: Query Augmentation:
f- random
raadaaaannaaaaaaaaaa sample( Q)
4: Prompt Augmentation:
p + ne (g(g)
5: Dataset Augmentation: DA + {4,p, R(g,p)
6:
D. + (A)
7: Offline RI: M411 + ORI(
end for
return the policy model Tr, which can generate optimal query-dependent prompts during testing.
B Supplemental Experiment Detials
B.1
Hyperparameters and Details
For the task datasets with default testing or development set, we use their original split to obtain our testing set. If there is no official large set for stable evaluating and testing. Additionally, for all tasks, we split 10% training samples as collection set for initial query collection and subsequent query augmentation, cnsuring that the collected queries for training the policy model do not appear in the in-context examples during few-shot evaluation, and simultaneously simulating scenario where very few questions available. In each loop in query augmentation, 1,000 queries are sampled, which may include repeated queries. For repeated queries, the sampling prompt generation approach ensures different prompts are generated, allowing for the exploration of varying prompt. effectiveness for each query. We do not explicitly aim for uniqueness in the sampled 1,000 queries per loop, opting instead for a fully random sampling approach.
teestnintttttgg split, we randomly sample a reasonably
The parameters for the experiments are shown in Table g Notably, QPO consists of loops, indicating there are 4 times of offline reinforcement. learning fine-tuning stage and times of data augmentation ween
18

---

Published in Transactions on Machine Learning Research (02/2025)
each fine-tuning stage. Since introduce additional networks into the policy model, the number of training epochs and the learning rate required for the first loop of training are relatively high.
Table 9: Hyperparameter settings of QPO
Hyperparameters
Values
Loops
Batchsize
128
Learning Rate
1e-3 for the 1st loop, 1e-4 for others
Train Epochs
100 for the 1st loop, 20 for others
Optimizer
AdamW
Weight Decay
le-4
Balancing Parameter
0.1
B.2 Offline Dataset Collection
For the initial dataset collection, testing 150 prompts on too many queries incurs excessive computational costs. Therefore, we test 100 queries for every group of 40 prompts (with the last group containing 30 prompts). Each group sampled different queries, achieving a balance between query coverage and compu- tational cost. Since queries are randomly sampled for each test group, there are overlapped queries across different groups. In the end, our dataset consists of 15,000 paired examples, comprising 150 prompts and ap- proximately 350 queries. This dataset size is significantly lower than that required by other offline algorithms, such Prompt-OIRL for averages 56,900 examples and Promptist for averages 360,000 examples.
For the specific prompts, the initial 30 prompts are derived from various previous approaches, including PromptSource, ChatGPT Rewrite, Low Perplexity, RLPrompt and APE, which are all produced by In- structEval (jjihh et al. 2023), and we leverage ChatGPT-3.5 chat box to rewrite 120 new prompts based on these 30 prompts through in-context learning. Specifically, we provide existing prompts to ChatGPT-3.5, and ask it generate other 30 effective prompts for times. The prompt we use for rewriting is demonstrated in Table 10
Table 10: Template used for ChatGPT rewriting prompts (Sun et al. 2023
In practice, people find the following prompts help improving the classification abilities of language models like LlaMa-7b-chat on the <TASK> dataset. Rewrite the following instructions via rephrasing and/or adding specific requirements. Use illustrative description needed.
###*#####
Prompts:
<PROMPTI>
<PROMPT2>
<PROMPT3>
<PROMPT4>
<PROMPT5>
Please provide 30 more prompls different from those prompts that may improve the classification ability of language models. Diversity is much appriciated!
B.3 Codes and Hardware
Our code, as well as the offiine datasets, is now publicly available here All experiments are conducted on single NVIDIA V100 32g GPU.
19

---

Published in Transactions on Machine Learning Research (02/2025)
C Additional Results
C.1 Main Results on 3-shot Evaluation
Table 11: Main results of QPO and baselines on NLU tasks on 3-shot settings
Method
AG News
BoolQ
IMDB
Emotion
CosmosQA
HellaSwag
Avg.
PromptSource
76.0
58.5
90.3
42.2
39.3
69.8
62.7
Low Perplexity
53.5
58.0
90.1
48.0
39.0
70.0
59.8
RLPrompt
76.2
69.4
89.7
56.1
41.1
69.9
67.
APE
58.6
59.9
89.9
57.6
40.2
70.2
62.7
Prompt-OIRL
76.9
67.1
89.2
52.7
40.6
70.4
66.2
QPO
82.3
71.1
90.5
62.7
41.5
70.7
69.8
C.2
Main Results on GPT-4o.
As demonstrated in Table 12 our method also outperforms other baselines on most updated models.
C.3
Cost Analysis
In this soction, we analyse the specific number interactions roquired by different algorithms. APE requires Lotal of M +|T interactions with LLMs to obtain the leedback for optimization, while QPO only requires C interactions for data augmentation, where M is the candidate population under evaluation, [D] the size of development set, |C| is the size of collection set, and T and T' are the number of iterations needed. Though |C| is slightly larger than [D] for better exploration, QPO significantly reduces computational overhead due to its minimal iteration T and the absence of the need for evaluate M candidate prompts.
C.4 Analysis on reward design
Both Rquery and Rrask contribute to the performance. By separately removing Reuuery and Riask and compare the results with the full reward design R results in Table 13 indicate that for most of (asks, the combined design of R yields superior performance. Thus, both Rquery and Riask contribute to the performance. Notably, the use of either Rquery Reask alone also demonstrates commend- able results, which underscores the rationality of employing these two metrics as rewards. Furthermore, the comparison between using Rquery and Rtask individually highlights the advantage of Ryuerys suggesting that query-based prompt optimization is more effective than task-level prompt optimization.
Perplexity penalty is important. We compare the performance of rewards with and without perplexity (PPL) to validate the importance of the fine-grained penalty in Rquery. The experimental results are presented in Table 14] For all tasks, incorporating PPT. consistently yields better performance. This is because the introduction of PPL transforms the Rquery from a binary value into a fractional value, enhancing pairs. Consequently, the policy model can better learn
the discrimination among different which prompts are more effective for specific queries.
Predicting reward at the first token helps learn better. We introduce the reward prediction loss to enhance the model's ability to discern the reward values across different query-prompt pairs, while si- multaneously preventing the policy model from overemphasizing rewards to the extent that it neglects the distinctions between queries and prompts, potentially leading to overfitting on the prompt with the highest reward value across all samples in the dataset. Predicting the reward at the first. token can simplify the attention calculation process and mitigate excessive updates to the query and prompt embedding layers, thereby reducing the risk of overfitting. Our experimental validation in Table confirms the effectiveness of predicting the reward value the first token, evidenced by its better performance compared predicting at the last token.
20

---

Published in Transactions on Machine Learning Research (02/2025)
Table 12: Main results of QPO and baselines on GPT-4o on GSM8K.
Method
GSMSK
Co
95.8
ChatGPT
94.7
APE
95.0
Prompt-OIRI.
93.5
QPO
96.5
Table 13: Comparisons between Rguerys Reask and Rquery
P Reack-
AG News
BoolQ
IMDB
Avg.
Method
zero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
Rquery
78.3
84.5
71.4
77.2
89.0
90.9
79.6
84.2
Reask
76.3
81.6
71.0
75.3
84.7
91.2
77.3
82.7
Riquery + Rtask
79.0
85.6
71.0
76.5
91.2
91.4
80.4
84.5
C.5 Results on Different Prompt Quantity
The detailed results are shown in Table 16
C.6 Results on Different Prompt Quality
We conduct experiments on the dataset where all prompts were rewritten by ChatGPT to validate the robustness of QPO prompt quality within the dataset. The results are presented Table 17 The absence of initial high-quality prompts has minimal impact on QPO's performance, as the method focuses the score of cach query-prompt pair rather than the performance of individual prompts. Consequently, even without expert prompts, there are still other prompts perform well for each specific query.
The detailed results on different query peory-p pair quality is shown in Table 18
C.7 Case Study
21

---

Published in Transactions on Machine Learning Research (02/2025)
Table 14: Comparisons between with and without perplexity (PPL).
Method
AG News
BoolQ
IMDB
Emotion
CosmosQA
HellaSwag
Avg.
QPO (w/ PPL) 79.0
71.0
91.2
68.7
45.3
69.7
70.9
QPO (w/o PPL)
75.0
65.7
90.3
62.7
42.7
69.0
67.6
Table 15: Comparisons between reward predicting positions.
AG News
BoolQ
IMDB
Avg.
Method
zero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
QPO (first token)
79.0
85.6
71.0
76.5
91.2
91.4
80.4
84.5
QPO (last token)
79.3
81.2
69.7
76.1
89.3
90.9
79.4
82.7
Table 16: Performance under different prompt availability.
AG News
BoolQ
IMDB
Avg.
Prompt Num
zero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
zero-shol
few-shot
scarce
65.3
83.3
65.7
76.3
87.3
89.6
72.8
83.1
middle
70.7
83.9
70.7
78.4
91.5
89.7
77.6
84.0
rich
79.0
85.6
71.0
76.5
91.2
91.4
80.4
84.5
Table 17: Performance under different prompt quality in dataset.
AG News
BoolQ
IMDB
Avg.
Datase
zero-shot
lew-shot
zero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
QPO
79.0
85.6
71.0
76.5
91.2
91.4
80.4
84.5
only ChatGPT prompt
74.3
85.9
67.0
76.4
90.7
90.9
77.3
84.4
Table 18: Performance under different query-prompt pair quality.
AG News
BoolQ
IMDB
Avg.
Datase
wero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
zero-shot
few-shot
expert
79.0
85.6
71.0
76.5
91.2
91.4
80.4
84.5
medium-expert
75.0
83.1
68.3
75.9
90.3
88.9
77.9
82.6
22

---

Published in Transactions on Machine Learning Research (02/2025)
Table 19: Prompls generated by QPO. 'LAcc." denotes the individual accuracy.
Generated Query-level Prompt
I. Acc.
QPO Acc.
AG News
In which part of a newspaper, World News, Sports, Business, or Science and Technology? Innov., Sports.., or., or.,..: or.,
69.0
Assign this news report to the most suitable newspaper segment: World News, Sports, Business, or Technology. and Sports, or Sports, or
79.0
70.0
BoolQ
Analyze if the passage validates the true/false claim in the question.
64.0
71.0
Confirm the correctness of true/false question using the text excerpt.....
63.0
TMDB
Decipher the overall sentiment conveyed by the revicwer's critique of the movic.
89.3
or negative?
91.2
Interpret the reviewer's emotional stance lowards the film: positive or negative???
87.7
Emotion
receive full credit, choose the emotion that best fits the sentiment expressed in the tweet from the following oplions: anger, joy, optimism, or sadness. sadness.
49.7
From the given options, determine the emotion that best caplures the essence of the tweet's sentiment: anger, joy, optimism, or sadness... tweet. best
68.7
60.0
CosmosQA
Analyze the contextual details to select the response that offers the most compre- hensive understanding. the.
43.3
Synthesize the information provided below to formulate well well-supported response the upcoming question. question.
45.3
42.3
HellaSwag
Explore the passage's thematic depth and devise ending that explores those
68.0
themes further.......
Enlm with the passage and them and incorporate them into the ending. engaged.
69.7
67.7
engagedates. their comple their expectations.
23


Reviews:
[{"review_id": "xXXtiNuuqR", "reviewer": "Reviewer_rn9Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. This paper is well written and easy to follow. \n2. This paper is presented in a clean and elegant way.\n\nWeaknesses:\n\n1. Missing details on the offline dataset. The authors did not clearly explain how the offline dataset is constructed, making the quality of the experiments less trustworthy \n2. No code / dataset uploaded. With all missing details in dataset, the author did not upload the code & offline dataset, which makes the results even less trustworthy.\n3. Incorrect citations. This paper have some basic misuses of citations, which the reviewer considers as a red flag for insufficient literature review and not respecting prior efforts. For example, \n- When mentioning Chain-of-Thought, in page 7, paragraph baseline, the authors cited paper by Kojima et al, but not the most well known CoT paper by Wei et al \n- When mentioning GPT-3.5-turbo, the author cited Ouyang et al, 2022, which is instructGPT, not GPT-3.5 turbo\n- This paper applies offline RL in training, but no related papers about the offline RL field is mentioned, the author should at least mention [2], when talking about offline RL. \n4. The models being used in the paper are far from the most updated versions. For example, in page 7 implementation details, why not use ChatGPT-4 (or even 4o)?\n\n\n\n[1] Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in neural information processing systems 35 (2022): 24824-24837.\n\n[2] Levine, Sergey, et al. \"Offline reinforcement learning: Tutorial, review, and perspectives on open problems.\" arXiv preprint arXiv:2005.01643 (2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces QPO (Query-dependent Prompt Optimization), a novel query-dependent prompt optimization framework for LLMs. QPO leverages multi-loop offline reinforcement learning and a novel Multi-Loop Augmentation technique to iteratively fine-tune a prompt generation policy. The authors show improved performance across diverse NLP and math tasks in both zero-shot and few-shot settings, showcasing QPO's efficacy, cost-efficiency, and cross-model generalizability for improved prompt engineering.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please upload the dataset and codebase.\n2. All citations mentioned in the weaknesses section.\n3. If possible, please try to use the most updated model to collect the dataset. If not, explain why the paper did not use ChatGPT4 at the first place (even gpt4 is referred as “legacy model” by openai)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qFec1a1L9q", "reviewer": "Reviewer_xCNt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is relatively easy to follow (up to some missing bits, see below)\n* The empirical evaluation is thorough and shows good results\n\nMain questions / concerns: \n\n* The actual offline RL approach is not described. The reward loss is described, but then, how it’s used to fine tune the model seems to be implicit. But Offline RL can mean many things and the specific setting used here (single-step decision making process) would probably affect the learning algorithm used. Your task looks like an offline contextual bandit problem, is it what it actually is? \n\n* For the reward definition, you write that you use the “Min-Max Normalization Method” but there is no citation nor explanation. The equation below (Eq 4) does  not look like a normalization to me, it’s just a sum of 2 terms. I am not sure what you meant.\n \nMinor comments:\n\n* It is fairly clear that an LLM was used to polish the paper, which is totally fine indeed. Just a couple of remarks: “In the realm of prompt engineering” sounds quite funny, also you used about 15 times the verb “utilize”, which is most of the time a synonym of “use” and could be replaced. \n* The acronym SFT seem to never be introduced. \n* Page 5: “previous researches”  is incorrect -> previous research\n* The “accuracy” that you report in all the tables in the Experiments section is never really defined. I think you mean it’s the accuracy mentioned in the reward design section on page 4, but I have a doubt. It might be worth specifying exactly what you report.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a process to train a small language model to design prompts. Prompts are intermediate instructions given to the “target LLM” in order to generate a better answer to the original user query. The main idea is to use an offline RL algorithm to progressively tune the prompt model. Namely, they collect data via a multi-loop process where they augment the dataset in every loop by querying the target LLM with the newly created prompts. The new data is then used to fine-tune the prompt model (offline RL), and so on. \nI view the main contribution as the design of this data augmentation process and its evaluation on several language understanding tasks and against many strong baselines (to the best of my knowledge).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Add the details on the \"offline RL\" algorithm, for clarity but also for reproducibility of the results. For now, I tick 'No' in 'Claims and Evidence' below as I don't think the paper is reproducible as is.", "broader_impact_concerns": "Nothing to declare. This is a paper on a method to get better performance from LLMs without fine-tuning of the parameters. LLMs have a broad impact but this paper does not raise a flag.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pD79t05AiS", "reviewer": "Reviewer_giNB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Overall, the paper is well-written.\n- The proposed training objective is well designed and makes sense.\n- The main experiments show the superiority of QPO and ablation studies provide interesting findings about query-dependent prompt engineering.\n\nWeaknesses\n- There is no in-depth analysis on the training objective: e.g., ablation study to verify that both R_query and R_task contribute to the performance, the importance of perplexity penalty in R_query, comparison with the different implementation of reward prediction loss (predict reward at the last token, not the first token), etc.\n- There is a lack of explanation/analysis of the reward prediction loss. Please provide the evidence that the finetuned PLM with the reward prediction loss can serve as an autoencoder.\n- There are some missing details in the main paper\n  - It is not clear that how the initial 30 prompts are constructed. Are they the results of a single previous approach or various approaches?\n  - How many queries are sampled at the query augmentation stage in each loop?\n  - Please indicate which methods are human-made, query-agnostic, and offline query-dependent in Table 1.\n  - Is the data filtering applied after getting 150 prompts? If not, how many prompts remain after filtering?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main idea of this work are two-folds:\n- Most of prompt engineering approaches focus on single task-level prompts, which might lead to suboptimal performance. The proposed method, Queery-dependent Prompt Optimization (QPO) finetunes a small pre-trained language model (PLM) to generate prompts tailored to different queries in the same task.\n- To reduce the usage of LLM for prompt optimization, QPO leverages multi-loop offline reinforcement learning on the dataset initially consisting of query-prompt pairs from previous prompt engineering approaches and bootstrapped with the finetuned PLM between each training loop.\n\nThe experimental results show the superiority and cost-efficiency of QPO on several natural language understanding tasks and math reasoning tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, the quality of the paper is good. Please address the weaknesses above.\nAlso, minor questions:\n- The authors said that QPO is robust to prompt quality. Does QPO achieve competitive results even without initial high-quality prompts, that is only using the rewritten prompts by ChatGPT or the prompts with the rewards below the threshold (66.7th percentile)?", "broader_impact_concerns": "Please add the Broader Impact Statement in the final draft.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper touches on an interesting problem to the community, that is, how to do efficient prompt engineering. The paper provides details and experiments to support the claims. The method is easy to follow and reproducible. The authors did a great job in improving the paper during the rebuttal process.
Decision: Accept (poster)