Robust Prompt Optimization for Large Language Models Against Distribution Shifts
Authors: Moxin Li
Wenjie Wang
Fuli Feng
Yixin Cao
Jizhi Zhang
Tat-Seng Chua
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Model; prompt optimization; distribution shifts; sentiment analysis; question answering;

Abstract: Large Language Model (LLM) has demonstrated significant ability in various Natural Language Processing tasks. However, their effectiveness is highly dependent on the phrasing of the task prompt, leading to research on automatic prompt optimization using labeled task data. We reveal that these prompt optimization techniques are vulnerable to distribution shifts such as subpopulation shifts, which are common for LLMs in real-world scenarios such as customer reviews analysis. In this light, we propose a new problem of robust prompt optimization for LLMs against distribution shifts, which requires the prompt optimized over the labeled source group can simultaneously generalize to an unlabeled target group. To solve this problem, we propose Generalized Prompt Optimization framework , which incorporates the unlabeled data from the target group into prompt optimization. Extensive experimental results demonstrate the effectiveness of the proposed framework with significant performance improvement on the target group and comparable performance on the source group.

Robust Prompt Optimization for Large Language Models Against Distribution Shifts
Moxin Li', Wenjie Wang'; Fuli Feng2.3, Yixin Cao", Jizhi Zhang
Tat-Seng Chua
National University of Singapore, Uniieersity of Science and Technology of China Institute of Dataspace, Hefei, Anhui, China, "Singapore Management University
limoxin@u. nus. edu, wangwenjie@u, nus, edu, ful fulifeeng@@gmai..ccmm com, caoyixin281 1@gmmai com, waaponngjziiimman ustc. edu. cn, dcsctsênus. .g
Abstract
Prompt Optimization
Training Samples
Step
Large Language Model (LLM) has demon- strated significant ability in various Natural Language Processing tasks. However, their ef- fectiveness is highly dependent on the phrasing the task prompt, leading to research on auto- matic prompt optimization using labeled task data. We reveal that these prompt optimization techniques are vulnerable to distribution shifts such as subpopulation shifts, which com- mon for LLMs in real-world scenarios such as customer reviews analysis. In this light, we propose problem of robust prompt opti- mization for LLMs against distribution shifts, which requires the prompt optimized over the labeled source group can simultaneously gen- eralize to an unlabeled target group. To solve this problem, we propose Generalized Prompt Optimization framework, which incorporates the unlabeled data from the target group into prompt optimization. Extensive experimental results demonstrate the effectiveness of the pro- posed framework with significant performance improvement on the target group and compara- ble performance on the source group.
feedback varlous products
Light weight laplop with new amazing battery
[Prompt]
LLM
ar exparionces
update
Stop
Provide e sentiment analysia of given input
Accuracy
feedback
Deployment
Test samples same distribution (source group)
LLM
Task spectic optimized prompt
Accuracy = 85%
samples different distribution (target group)
Accuracy
Figure Illustration of prompt optimization under dis- tribution shifts. Existing prompt optimization solutions aim to improve the LLM performance on the training data, while unclear whether the optimized prompt can be generalized testing data of the same task but with distribution shifts.
McDonell, 2021:
Gao et al., 2021), making prompt optimization a promising research direction.
1 Introduction
Existing research has explored automatic prompt optimization methods eliminate manual effort in identifying effective prompts for a given task. These methods can be gradient-based or gradient- free, depending on the availability of model gradi- ents. Gradient-based methods optimize the prompt by calculating its gradients through the LLM (Schick and Schütze, 2021b,a: Hu et al., 2022). Gradient-free methods update prompts based on LLM outputs using techniques such as an itera- tive suurch-nnd-slles over the prompt space (Zhou et al., 2023; Prasad et al., 2022; Pryzant et al., 2023). This work focuses on gradient-free prompt optimization LLMs are evolving into black-box API services (Sun al., 2022).
LLMs have gained significant attention for their remarkable performance in broad range of Nat- ural Language Processing (NLP) tasks (Ouyang al., 2022; Chung et al., 2022; Brown al., 2020; Touvron al., 2023). This success has led shift in the paradigm of solving NLP tasks, mov- ing away from training task-specific deep models towards developing sask-spc strategics to effec- tively utilize LLMs (Wei et al., 2022; Kojima et al., 2022; Wang et al., 2022a; Ye et al., 2023b). In the new paradigm, the prompt becomes a crucial factor in ensuring the effeectiveness of LLM the NLP task, since even slight variations in prompt phras- ing can largely affect LLM output (Reynolds and
Current gradient-free prompt optimization meth- ods ignore distribution shifts (Wang et al., 2023),
*Corresponding author.

---

where the data an LLM serves may differ from the labeled data for prompt optimization. Real- world NLP applications often encounter distribu- tion shifts, such as new user groups with distinct linguistic habits customer review analysis. is unclear if prompts hinder the robustness of LLMs against distribution shifts. To answer this question, we conduct experiments with the representative gpt- 3.5-turbo-0301 model and prompts optimized by APE (Zhou et al., 2023) over paired data groups with distribution shifts. Results on 30 pairs of data groups from six tasks show the risk of significant performance gaps under certain distribution shifts. Based on this finding, we propose a new robust prompt optimization problem, which aims to opti- mize task-specific prompts with consideration of performance both source and target groups un- der different distributions. Given an NLP task such as sentiment analysis, our problem setting has labeled source group similar as the conventional prompt optimization setting and a unlabeled target group. We keep the target group unlabeled for the consideration that distribution shifts happen along time in practice. Labeling the newly coming target group will cause unnecessary labor cost and latency. Accordingly, the main challenge for solving this ro- bust prompt optimization problem is incorporating unlabeled data into prompt optimization.
tasks, validating the rationality and effectiveness of our proposed framework.
2 Preliminary Experiments
Prompt optimization aims to find the best prompt that can instruct LLMs to predict the output y based on the concatenation of p and task input where x, y and p are all sequences of tokens. Formally, given an NLP task with dataset y)x following a distribution P. the goal is obtain
pd arg max E (xx))- ().), x), (I) DEZ
where Z denotes the prompt optimization space and r the evaluation metric to compare the LLM output with the ground truth output y. e.g., Accu- racy. Existing studies usually leverage gradient- based or gradient-free methods to automatically optimize the prompts. Since LLMs are evolving into black-box API services, gradient-free methods become increasingly important. However, they ig- nore distribution shifts between training and testing data. In this light, we conduct controlled experi- ments to answer the following research question:
Are prompts optimized by existing gradient-free methods robust to distribution shifts?
To this end, we propose the Generalized Prompt Optimization (GPO) framework to obtain task- specific prompt for both source and target groups. To jointly considering the two groups in prompt op- timization, the key lies in labeling the target group in automatic and reliable manner by adapting knowledge from the labeled source group. Towards this goal, we leverage the strong power of LLM in zero-shot labeling, and prompt ensemble to en- hance the labeling robustness. Experimental results on three tasks demonstrate the effectiveness of our framework in improving the performance on the target group and simultaneously preserving com- parable performance on the source group. To sum up, our contributions threefold:
2.1 Evaluation Protocol
We conduct the controlled experiments between a pair of data groups with distribution shifts, i.e., a source group {x).,,)) following a distribution Ps, and a target group {(xx, with distribu- tion Pt. where P. * Ps. We intend to examine whether the prompt pa optimized
the source group can generalize to the target group. Specifi- cally, given p" and p' optimized on the target group, we compare the performance of pd on the target group E(x,)) [r(LLM(pp (r(LMpp x), with that of p' (rx.))-//[(LLMM y).
Datasets. We select 16 datasets from six popu- lar NL. tasks, where each pair of groups under the same task is treated as the source and tar- get groups Following recent out-of-distribution (OOD) research (Yang et al., 2022), we take each dataset group and regard different backgrounds and topies across the datasets the distribution shifts. For the sentiment analysis task, we adopt Yelp (Zhang et al., 2015), Flipkart (Vaghani and Thummar, 2023), IMDB (Maas et al., 2011 1) and Amazon (Zhang et al., 2015) of different topics.
We reveal the robustness issue of prompt opti- mization against distribution shifts and propose new robust prompt optimization problem.
We propose the Generalized Prompt Optimiza- tion framework, which generates robust prompts considering both labeled and unlabeled data.
We conduct extensive experiments on three NLP

---

Target
Target
Source
MILII
ANLI
Source
Yelp
Flipkart
IMDB
Yelp
MNLI
73.4 = 1.0 73.3 = 1.3
45.4 1.9
Flipkart
69.1 8.7
85.1
2.9
85.2 + 9.4
85.9
12.5
ANLI
46.0 = 1.5
IMDB
71.1 + 8.2
76.9 + 13.4
91.9
0.9
90.4
5.2
Amazon
75.5
1.5
85.6
2.1
91.5
0.8
93.5
1.4
Natural language inference
Target
(a) Sentiment analysis
RTF
HANS
Source
Target
SocialIQA
PIQA
OpenbookQA
RTE
78.3 = 0.8
67.2 = 1.1
Source
HANS
79.0 + 0.8
68.4 + 1.8
SocialIQA
75.6
82.0
5.2
6.9
83.6
69.2 5.1
(b) Textual entailment
OpenbookQA
84.5
80.1 244
Target
DSTC7
Ubuntu Dialog
MuTual
(b) Commonsense QA
Source
Target
DSTC7
58.4 + 0.8
78.9 = 0.3
74.2 + 2.2
Numher
Spans
Source
Ubuntu Dialog
56.9 = 1.3
78.7 0.5
74.4 2.1
MuTual
52.2 = 4.4
74.7 6.0
76.7 3.4
Number
51.9 = 2.8
20.1 63.1 2.2
Spans
57.7 +: 2.9
(c) Dialog
(c) DROP
Table 1: Results for tasks without large generalization performance gap across groups.
Table 2: Results for tasks with significant generalization performance gap across groups. Bold font indicates the largest value for each column.
For the natural language inference task, we uti- lize MNLI (Williams et al., 2018), and ANLI (Nie et al., 2020) which is an adversarial dataset for MNLI. For the textual entailment, we use RTE (Wang al., 2018) and its OOD dataset HANS (McCoy al., 2019). For commonsense QA, use SocialIQA (Sap et al., 2019), PIQA (Bisk et al., 2020), and OpenbookQA (Mihaylov et al., 2018), which focus on different types of commonsense knowledge. For the multi-turn dialog reasoning. we use DSTC7 (Gunasekara et al., 2019), Ubuntu Dialog (Lowe et al., 2015), and MuTual (Cui et al., 2020). Besides, for the numerical QA task, the samples of different answer types (i.e., nu- merical values and text spans) in DROP (Dua et al., 2019) as two groups. See Appendix 1 for details.
data to report the averaged results. More implemen- tation details can be found in Appendix A.2.
2.2 Experimental Results
Demonstration of Generalization Performance Gap. Table shows the tasks without a large generalization gap between the performance prompts pd and p', and Table shows the tasks with large gaps (Accuracy gap>>8.0) on some groups. The row headers refer to the source groups for prompt optimization while the column headers show the target groups test optimized prompts. The generalization performance gap between p* and p' can be observed by comparing the values the same column.
Experimental Setup. We adopt APE (Zhou et al., 2023), an effective gradient-free prompt optimiza- tion method, for prompt generalization analysis. To highlight the effect of prompts, we conduct ex- periments under the zero-shot setting without in- context examples. For the backbone LLMs, we leverage 3pt-3.5-turbo-0301 by calling the OpenAl API For all classification tasks (all tasks except for DROP), we use accuracy the evaluation met- ric. For DROP, we utilize its standard evaluation metric - Fl. Following the setting of APE, we randomly sample N-shot training and N-shot vali- dation samples for prompt optimization, and repeat the experiments for five runs with different sampled
From the tables, we can observe: 1) The gener- alization performance gap may not exist for pre- viously studied OOD and adversarial groups (see Table 1), including the groups of the natural lan- guage inference and the textual entailment tasks. This is possibly attributed to the strong generaliza- tion ability of LLMs. 2) However, under some data groups of Table 2 such as the sentiment analysis datasets (e.g., Flipkart and Yelp) and the common- sense QA datasets with different topics (e.g., PIQA and OpenbookQA). and the DROP groups with different answer types, there are still significant generalization performance gaps, demonstrating the existence of the generalization issue of prompt optimization. 3) Surprisingly. the prompt p¹ op-
Phttps://chat..r

---

Target
SocialIQA
PIQA
OpenbookQA
Yelp
Flipkart
IMDB
Amazon
Source
word I-gram char 4-gram
0.43
0.51
0.58
Yelp
0.33
1.62
1.62
0.50
0.60
0.65
Flipkart
0.30
0.57
0.56
(a) The n-gram diversity.
IMDB
0.25
0.29
Amazon
0.25
0.27
Target
SocialIQA
PIQA
OpenbookQA
(a) Label distribution shifts. Smaller values indicate less distri- bution shifts.
SocialIQQ
0.39
0.38
PIQA
0.47
0.46
OpeabookQA
0.51
0.52
Target
Yelp
Flipkart
IMDB
Amazon
Snurce
(b) The word I-gram coverage ratio between groups.
Yelp
0.65
0.73
0.76
Target
SocialIQA
PIQA
OpenbookQA
Flipkart
0.59
0.55
0.63
IMDB
0.70
0.81
SocialIQQ
0.51
0.51
Amazon
0.71
0.70
0.78
PIQA
0.60
0.58
OpeabookQA
0.66
0.64
(b) Input similarity. Larger values indicate less distribution shifts.
(c) The character 4-gram coverage ratio between groups.
Table 3: Results for (a) label distribution shifts (b) input similarity of the sentiment analysis datasets. Bold font indicates the least distribution shift for each column.
Table 4: Evaluation (a) the n-gram diversity and (b) word I-gram coverage ratio (c) character 4-gram coverage ratio of commonsense QA datasets to study the even higher generalization performance. Bold font indicates the largest value for each column.
timized from the source group does not always perform worse than the prompt p' optimized on the target group. In Table 2(b), pº from OpenbookQA performs even better than p' for SocialIQA. Be- sides, for DROP in Table 2(c), pº from Spans also performs better than p' from Number. the fol- lowing section, we try to explore the reasons for the above three observations.
formance. Nevertheless, the two metrics cannot perfectly explain the performance on all tasks (cf. Appendix A.3). Therefore, Q1 is still a challenging research question, requiring further exploration in future work.
For Q2, we conjecture that the outstanding gen- eralization performance is because a source group with large diversity covers heterogencous patterns in the target group, leading to a more robust prompt pa than p². To explore this, we measure the het- erogeneity of source and target groups by calcu- lating the percentage of unique n-grams, and the percentage of n-grams of the target group covered by the source group. For illustration, we present the results of the commonsense QA task in Table 4. From Table 4(a), we can observe that OpenbookQA has the most diverse input according to the n-gram statistics. Moreover, OpenbookQA covers a large proportion of n-grams of SocialIQA and PIQA. These partly explain the superiority of the prompts optimized on Openbook (see Table 2).
Exploration on the Factors Affecting Prompt Robustness. Based on the above observations, we further explore two research questions.
Q1: Why do the prompts optimized on source groups perform differently on target group? Q2: Why does the prompt optimized on the source group perform even better than the prompt opti- mized on the target group in some cases?
For Q1, we conjecture that the varied perfor- mance gaps are attributed to different distribution shifts between the source and target groups. To verify this, we examine two metrics measure kinds of distribution shifts: 1) the label shifts measured by the KL divergence, and 2) the input similarity quantified by the n-gram similarity of the input corpora of the two groups. Their detailed implementation is illustrated in Appendix A.3. We show the results of the sentiment analysis task as example in Table 3. We can observe that the smallest label distribution shifts and the largest in- put similarity in Table generally coincide with the best generalization performance on each get group in Table 2, indicating the correlation between distribution shifts and generalization per-
3 Robust Prompt Optimization
In this section, we first formulate robust prompt optimization problem and propose GPO frame- work enhance the robustness of the prompts.
3.1 Problem Definition
To enhance the generalization ability of prompts, we propose a robust prompt optimization prob-

---

Meta Prompt
more, the collected source group data cannot cover all potential target groups, and the prompts opti- mized on the source groups may inevitably test on the examples from previously unseen groups. Thus, we aim at improving the generalization ability of one task-specife prompt across different groups.
friend given instruction. Based on the instruction, gave him several inpuls, and he generated the corresponding oulputs. Here are the input- aulput examples: [Input: Output: The instruction is to
prompts
Provide feedback on various
Source group data G,
LLM
products expertences
Ensemble Labeling
3.2 GPO Framework
Labeled target group data G;
To obtain a robust prompt for both the source and target groups, it is natural to jointly consider G. and G2 for prompt optimization. However, G2 lacks the labels {yı} that are commonly required by the gradient-free optimization methods (refer to Ta- ble 5 for the inferior results without labeling). With the impressive capabilities of LLMs on zero-shot labeling, we propose to utilize LLMs to label {x.). Considering that noisy labels may damage the qual- ity of optimized prompts, we further present two strategies to improve labeling accuracy.
APE
Upsampling
Consistency >
Optimized Prompt
Unlabeled target group data (x)
Figure 2: The GPO Framework.
lem. Specifically, given an NLP task such as sen- timent analysis, it aims to optimize a task-specific prompt for the data groups with different distri- butions. We consider the popular scenario where a source group Ge = {x)x.,,)} following a dis- tribution Pg and {x:) } in a unlabeled target group Pt (P: * Pa) are available while (y,) is unseen during prompt optimization. The objective becomes utilizing G, = (xxx,,)) and {xr} } to optimize a task-specific prompt robust to the samples from either Ps or P..
As illustrated in Figure 2, we first propose a Meta Prompt to instruct LLMs to acquire knowledge from the labeled source group and generate a series of prompts. Thereafter, we utilize a prompt ensem- ble labeling strategy to apply generated prompts to an LLM for precise labeling of (x,). In detail, we derive three-step framework to perform the labeling with two strategies, and then conduct joint prompt optimization as shown in Figure 2.
G, y()
Reasons for Access to Unlabeled Target Group. real-world deployment, LLMs continually en- counter the testing data with distribution shifts. Col- lecting the input features [x,] of the target group is feasible. For example, when using LLMs as web services to solve user queries of certain NLP tasks, it is easy to collect extensive user queries as unla- beled target groups. However, labeling (xt) may be time-consuming and costly, and thus we intend to optimize robust prompts without the labels of the target group.
1. Prompt Generation via Meta Prompt. Fol- lowing APE, we utilize Meta Prompt to ask LLM to generate prompts for labeling by feed- ing the examples of G. (see an example in Fig- ure 2). Based on strong language understanding and reasoning abilities, LLMS can infer the re- lationships between the inputs and outputs the examples and provide general and precise task prompts. We use different splits of G2 to generate K different prompts in total.
A Task-Specific Prompt vs. One Prompt for Each Group. To tackle the generalization issue of optimized prompts, an intuitive approach is to optimize a separate prompt for each data group, yet this simplistic approach faces several limitations in real scenarios. In real-world deployment, it not only requires additional computation costs to con- struct more prompts, but also needs accurately classify each testing sample into the appropriate group of the same distribution, thereby resulting in increased computation costs, latency, and new challenges for precise group classification. Further-
2. Prompt Ensemble Labeling Strategy. Given K prompts, we utilize each of them to label (xx) with an LLM, and thus obtain K candidate la- bels for each example. We adopt an ensembling strategy and select the label with the highest consistency among the K candidate labels for each example. Besides, inspired from Wang et al. (2022a), we set consistency threshold T E [o, 1] to only accept the labeled examples that have more than percent of prompts agreed on the label. Eventually, we obtain a filtered la- beled set G; for the target group.

---

3. Joint Prompt Optimization. Finally, mix G3 and G. run APE for joint prompt opti- mization and obtain the final optimized prompt. As G7. may have fewer samples than G. after filtering with T, we perform a random upsam- pling on G. to have the same data number as G, before running APE. A brief illustration about APE can be found in Appendix A.2.
Yelp (Source)
Plipkart (Target)
Top
Ensemble
Top
Ensemble
APE
79.7 = 0.7
79.7 = 1.0
78.4 = 1.9
81.3 = 144
APO
78.9 0.5
79.7
0.8
74.7
1 3.0
76.4 1 144
APE+ut
78.9 1.4
78.8
= 1.4
80.3
+ 2.0
80.7 2.1
GPO
79.1 0.7
78.7
0.9
80.5
2.1
84,5
2.0
Upper Bound
85.1
1 2.9
87.2 1: 0.5
(a) Sentiment analysis.
SocialIQA (Source)
OpenbookQA (Target)
4 Experiments
Top 1
Ensemble
Top 1
Ensemble
APE
75.6 = 1.4
69.6 = 5.3
71.2 5.2
74,8 = 3.2
4.1 Setup
APO
76.1 = 2.7
72.3 = 2.6
72.4
66.1 = 7.2
APE+ul
77.9 1.3
78.9
0.8
77.5 +
79.2 +.2
Datasets. We experiment GPO with three tasks: sentiment analysis, commonsense QA, and DROP. For each task, we select pair of groups with gen- eralization performance gap source and target groups, and ablate the labels for the target groups. Compared Methods. We adopt the following base- line methods: 1) APE; 2) APO (Pryzant et al., 2023), the state-of-the-an gradient-free prompt op- timization method for LLM; 3) APE-ut, a naive generalization solution by incorporating the unla- beled target group input into APE; 4) the Upper Bound, which represents the performance of the prompt optimized the target group data with ground-truth labels by APE; and 5) our proposed GPO; We also show the results of simple human- written prompts that are general for the task, and the revised versions by PromptPerfect which is an automatic prompt engineering website.
GPO
76.7 = 2.0
78.9 = 1.2
78.7
79.7 = 0.8
Upper Bound
80.1 2.4
80.8 1.1
(b) Commonsense QA.
Number (Source)
Spans (Target)
Top
Ensemble
Top
Ensemble
APE
51.9 +
5.2
20.1 + 1.3
18.2 + 0.2
APO
56.7
54.5 2.1
20.2 2,4
20.0 :: 2,2
APE
52.0
1.8
53.1 1.2
16.1 16.1::355
17.7 4: 28
GPO
52.2 6.0
53.6 1.61330 3.0
27.7 + 12.0
26.7 + 4,9
Upper Bound
63.1 2.2
63.7 + 0.8
(c) DROP.
Table 5: Results of the compared methods. Bold font indicates the best performance for each column.
group does not largely hinder the source group. Compared with APE, GPO shows increased per- formance on the source groups of SocialIQA and Number by incorporating the target group data, which is in line with the finding in Table 2. 3) Across baselines, APO outperforms APE on the source groups of the last two tasks and achieve com- parable performance on sentiment analysis, show- ing its effectiveness for prompt optimization. How- ever, the generalization ability is only comparable to APE since APO performs worse than APE on several target groups. 4) APE-ut achieves improved target group performance for the first two task, indi- cating the benefit of incorporating unlabeled target group data for generalization. However, for Spans where obtaining accurate target labels is challeng- ing (as shown by the low Fl values), APE-ut largely underperforms GPO, showing the importance of target group labeling especially for difficult tasks. Compare to Human-written Prompts. From Table 6, we further observe that GPO outperforms human-written prompts and PromptPerfect for sen- timent analysis and commonsense QA tasks. How- ever, on the most difficult task DROP, GPO under- performs human- written prompts. This is poten-
Evaluation Protocol. We utilize two strategies for testing: Top 1 and Ensemble. Top refers to using the single optimized prompt with the best validation performance, while Ensemble refers to labeling with all obtained K prompts and accept the output with the most agreement on the prompts. We utilize the same -hohot data as the preliminary experiments and also report the averaged results for five runs. More implementation details are il- lustrated in Appendix A.4.
4.2 Performance Comparison
Compare to Generated Prompts. From Table 5, we can observe the followings: 1) GPO achieves superior performance for all target groups in both Top and Ensemble testing, validating its effective- ness. However, there is still space for improvement towards the Upper Bound for all tasks, showing the challenge of the generalization problem. 2) GPO achieves comparable source group performance for all tasks, showing its improvement on the target hnttps://ppomppterrfect jina.ai.

---

Yelp (Source)
Flipkart (Target)
SocialIQA (Source)
OpenbookQA (Target)
Number (Source)
Spans (Tärget)
Human
78.7
80.0
71.3
60.0
54.9
37.1
PromptPerfect
77.3
83.3
74.7
64.0
54.0
26.9
GPO best
78.7
84.5
78.9
79.7
52.2
27.7
Table 6: Performance comparison for the human-wrthe written prompts, PromptPerfect and the more effect testing strategy of GPO (Top or Ensemble, denoted as GPO best). Bold font indicates the best performance for each column.
tially because the inaccurate labels for Spans hinder the prompt optimization. Similarly, PromptPerfcet also fail to optimize human-writter prompts for DROP.
Flipkart
OpenbookQA
Spuns
w/o cons
81.9
69.8
3,6
GPO
94.2
84.3
3.7
The labeling accuracy comparison for the target group training and validation data GPO and w/o cons. The results for Spans here is accuracy instead of FI.
Table 8:
4.3
Ablation Study
Yelp
Flipkart
w/o cons and w/o cons+--train, removing the target group training data benefits the Top results of the source group, but harms the Ensemble results of the target groups. It has less effeet on the target group Top results since the two methods still use target group validation data.
Top
Ensemble
Topp
Ensemble
GPO
79.1 0.7
78.7 + 0.9
80,5 = 2.1
84,5 2.0
w/o cons
78.8 + 1.2
78.7 + 0.4
81.5 1.4
84.0 = 0.9
w/o I-train
79,9 + 0.8
79.7 1.0
80.3 3.2
81.3 1.4
(a) Sentiment analysis.
SocialIQA
OpenbookQA
Top
Ensemble
Top
Ensemble
GPO
16.7
79.7
4.4
In-depth Analysis
w/o
76.0 2.8
78.1 1.4
77.6
78.8
= 2.2
w/o conset-train
77.9 1 1.6
6.3
78.2 2.2
74.8 3.2
Analysis on the Effect the Consistency Thresh- old. To further reveal the effect of consistency threshold, we first show the labeling accuracy of the target group training and validation data for GPO and w/o cons in Table 8. We can observe that applying the consistency threshold can improve the labeling accuracy for all target groups. By examining the relationship between this labeling accuracy improvement and the performance differ- ence between GPO and woo cons in Table 7, can be explained that for Flipkart and OpenbookQA, where the labeling accuracy is already high under w/o cons, further improving the labeling accuracy by the consistency threshold is unlikely to achieve large performance gain. Conversely, in the case of Spans with low labeling accuracy, even minor improvement can result in significant performance gains. To explore the connection between labeling accuracy and target group performance further, we conducted an experiment where we manually as- signed incorrect labels to varying proportions (0%, 50%, and 90%) of the target training and valida- tion data. The results are illustrated in Figure 3. It can be observed that as the percentage of incor- rect labels increases, the overall performance on the target group generally decreases, emphasizing the importance of labeling accuracy for achieving effective generalization.
(b) Commonsense QA.
Number
Spans
Top 1
Ensemble
Top
Ensemble
GPO
52.2 1: 6.0
53.6 3.0
27.7 12,0
26.7 1: 4.9
cons
2.8
51.0
20.6 20.6
+
cons-t-train
$1.3
50.9 1.6
20.4
18.7 2.2
(c) DROP
Table 7: Ablation study. Bold-font and underline indi- cate the best and second-best results, respectively.
We study the effect of prompt ensemble labeling and joint prompt optimization by evaluating two modifications of GPO: (1) setting the consistency threshold 0, denoted as w/o cons; and (2) remov- ing the target group training data during the final prompt generation, denoted as w/ cons+t-train. From Table 7, we can observe that: 1) In all cases except for Flipkart with Top evaluation, GPO performs better than w/o cons on target groups, showing the effectiveness of the consistency thresh- old. 2) Among the three tasks, DROP has large improvement between w/o cons and GPO on both source and target groups then the other two tasks. We hypothesis that this discrepancy is related the different degrees of improvement in the labeling accuracy by the consistency threshold, which will be further discussed in Section 4.4. 3) Comparing

---

90
Flipkart
90
OpenbeokAA
100
Spans
and spurious correlation (Tang al., 2023; Stolfo et al., 2022). Moreover, LLMs remain vulnerable to adversarial perturbations and achieve inconsis- tent results (Wang et al., 2023; Ye et al., 2023a; Liang et al., 2022). Additionally, LLMs demon- strate high sensitivity to the prompt (Reynolds and McDonell, 2021; Zhu et al., 2023) and the selec- tion of in-context examples (Liu et al., 2022; Ru- bin et al., 2022). Lastly, instruction tuning allows LLMs to generalize novel tasks (Ouyang et al., 2022; Wang al., 2022b,a). We specifically focus on the generalization issue of prompt optimization on the distribution shifts within one task.
85
80
Accuraty
Accuracy
60
80
60
75
20
65
90
50
90
50 Incorrect Label
% Incorrect Label
Figure 3: Target group performance under different per- centage of wrong labels. The blue dotted line indicates the labeling accuracy of GPO as in Table
Top 1 APE
Ensemble
GPO
APE
GPO
Vicuna-7B
38.4 4: 25.3 63.5
43,9
21.3
71.9
Vicuna-13B 13B
66.8 + 18.4
68.3 13.7
60.7 9.5
70.7 10.8
Prompt Optimization. Obtaining effective prompts for applying LLM in NLP tasks is popu- lar research area. Prompt tuning methods (Li and Liang, 2021; ; Lester et al., 2021; Qin and Eisner, 2021; Gu et al., 2022) learn soft continuous vectors as prompts in the LLM input using gradients from the task objective. Recent studies have also focused on gradient-free prompt optimization for black-box LLM, such as reinforcement learning- based methods (Zhang al., 2023; Deng et al., 2022; Diao et al., 2022), search-based methods (Brown et al., 2020; Prasad al., 2022; Pryzant et al., 2023), and other gradient-free optimization techniques like evolutionary algorithms (Sun et al., 2022) and boosting (Hou et al., 2022). Among them, the state-of-the-art methods leverage the power of LLMs for prompt optimization, such as prompt generation and evaluation by LLM (APE (Zhou et al., 2023)) and prompt editing following critiques (APO (Pryzant al., 2023)), where we mainly compare with them. Notably, while some previous work on prompt tuning has addressed generalization across tasks and models (Su et al., 2022: Vu et al., 2021; Qin et al., 2023), and domain adaptation (Tam et al., 2022; Guo et al., 2022), this paper specifically focuses on the generalization issue of gradient-free prompt optimization.
GPT-3.5
78.4 + 199
80.5 2.11
81.3
- 1.4
2.0
GPT4
77.5 +: 13.7 85.3 1: 2.7
83.3
+ 0.0
85.4
2.4
Table 9: Performance comparison of APE and GPO on Flipkart of different backbone LLMs.
GPO with Different Backbone LLMs. We also conducted experiments with GPO using different backbone LLMs, including Vicuna 7B and 13B (Chiang al., 2023) which notable smaller- sized LLMs, and GPT-4 (OpenAI, 2023). Table shows the generalization results on Flipkart with Yelp as the source group for APE and GPO on dif- ferent backbone LLMs. Due to the small sizes of the Vicuna models, generating the exact sentiment label as the answer can be challenging. Therefore, we extract the sentiment labels from their outputs before calculating the accuracy. The results show that there is room for enhancing the generalization performance in APE across various LLMs, and GPO consistently outperforms APE in all cases. Notably, when applying GPO to the smaller Vicuna 7B model, there a significant improvement that allows it to reach the same performance level as the Vicuna 13B model. Across LLMs, the smaller- sized Vicuna models achieve relatively worse per- formance, and the powerful GPT-4 achieves the best performance GPO.
Conclusion
5 Related Work
In this paper, we revealed the generalization issue of prompt optimization for LLMs under distribu- tion shifts. We observed that the prompt optimized on the source data group may have performance drop on the target group with distribution shifts. We performed an initial analysis aiming identi- fying the factors that correlate the varied gen- cralization performance across groups, including label distribution shift and input distribution sim-
Generalization Ability and Robustness of LLM. Researchers have been investigating the gener- alization ability and robustness of LLMs since their recent breakthrough. LLMs like ChatGPT have shown significant improvement in out-of- distribution (OOD) and adversarial tasks (Wang et al., 2023), although they are still imperfect (Chen et al., 2023). Some LLMs still rely on shortcuts

---

ilarity. To enhance the generalization ability of LLMs, we proposed Generalized Prompt Opti- mization framework to jointly consider the source and target groups for robust prompt optimization. Experimental results validated the effectiveness of our proposed framework in boosting the robustness of the prompts on the source and target groups. In future work, we plan to study the prompt general- ization unseen target groups without available inputs {xr). and explore prompt generalization abil- ity with in co-oonext examples from different groups.

---

Limitations
Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Firstly, this work discusses the generalization abil- ity of prompts while ignoring the effect of other LLM inputs such as in-context examples. The choice of in-context examples might also affect the robustness of LLMs. Future work can look into the generalization issue of the prompt in combina- tion with in-context examples. Secondly, this work assumes the availability of the inputs {xr } of the target group. It is under-explored how to achieve generalized prompt optimization to completely un- seen groups without (x.). To improve the robust- ness these groups, we believe it is helpful to extend this work toward robust prompt optimiza- tion on multiple heterogencous groups. Thirdly, we acknowledge that the scope of our research is limited to black-box LLMs capable of under- standing instructions, where gradient-free prompt optimization with instructing LLM is a suitable choice. For smaller LMs without instruction under- standing abilitics, e.g., BERT (Devlin et al., 2019) and T5 (Raffel al., 2020), they are generally not black-box and are more advantageous utilize gradient-based prompt optimization methods.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, al. 2022. Sealing instruction-finett arXiv preprint
language models.
Leyang Cui. Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. Mutual: A dataset for multi-turn dia- logue reasoning. arXiv preprint arXiv:2004. 2004:2200.04444. 04494.
Mingkai Deng, Jianyu Wang. Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song. Eric P Xing, and Zhiting Hu. 2022. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv: 125488.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Shizhe Diao, Zhichao Huang, Ruijia Xu, Xuechun Li, Yong Lin, Xiao Zhou, and Tong Zhang, 2022. Black- box prompt learning for pre-trained language models. arXiv preprint (xXx:2200108855
Acknowledgements
This work supported by NExT Research Center, and the National Natural Science Foundation of China (62272437). We thank the reviewers for their constructive feedback.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark re- quiring discrete reasoning over paragraphs. arXiv preprint arXiv: 1903.00161.
References
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 3816-3830.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. 2020. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2022. Ppt: Pre-trained prompt tuning for few-shot learning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8410-8423.
Xuanting Chen, Junjie Ye, Can Zu, Nuo Xu, Rui Zheng. Minlong Peng. Jie Zhou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. How robust is gpt-3.5 to pre- decessors? comprehensive study on language un- derstanding tasks. arXiv preprint arXiv:2303.0022
Chulaka Gunasekara, Jonathan K Kummerfeld. Lazaros Polymenakos, and Walter Lasecki. 2019. Dstc7 task 1: Noetic end-to-end response selection. In Proceed- ings of the First Workshop NLP Conversational AI, pages 60-67.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion
Xu Guo, Boyang Li, and Han Yu. 2022. Improving the sample efficiency of prompt tuning with domain adaptation. arXiv preprint arXiv:2210.02952

---

Bairu Hou, Joe O'Connor, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2022. Promptboosting: Black- box text classification with ten forward passes. arXiv preprint arXiv:2212.09255
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct elec- tricity? new dataset for open book question answer- ing, arXiv preprint arXiv:1809.00 0279.9279.
Shengding Hu, Ning Ding, Huadong Wang. Zhiyuan Liu, Jingang Wang. Juanzi Li, Wei Wu, and Maosong Sun. 2022. Knowledgeable prompt-tuning: Incor- porating knowledge into prompt verbalizer for text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2225-2240.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language under- standing. In Proceedings of the 58th Anmual Meet- ing of the Association for Computational Linguisties. pages 4885-4901.
OpenAL. 2023. Gpt-4 technical report.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205. 11916.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3055-3059.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2022. Grips: Gradient-free, edit- based in- struction search for prompting large language models. arXiv prepriat arXiv:2203.00
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 4582- 4597.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chen- guang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint 2xXiv:3305.03499
Chengwei Qin, Shafiq Joty. Qian Li, and Ruochen Zhao. 2023. Learning to initialize: Can meta learning im- prove cross-task generalization in prompt tuning? arXiv preprint arXiv:2302.08144
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprint ar arXiv:2211.00 1.001100..
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying Ims with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 5203-5212.
Jiachang Liu, Dinghan Shen, Yizhe Zhang. William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (Dee- LIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555 5551.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dia- logue systems. arXiv preprint 1rXiv:7506.08000 08900.
Laria Reynolds and Kyle McDonell. 2021. Prompt pro- gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Comput- ing Systems, pages 1-7.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. 201 Learning word vectors for sentiment analysis. In Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655-2671.
R Thomas McCoy, Bllie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntac- tie heuristies natural language inference. arXiv preprini 1rXiv:1922.01000 090000
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv: 0904.0988..

---

Timo Schick and Hinrich Schützze. 2021a. Exploiting dozze-quettios for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics; Main Volume, pages 255-269.
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng. Yidong Wang, Linyi Yang, Hao- jun Huang, Wei Ye, Xiubo Geng, et al. 2023. On the robustmess of chatgpt: An adversarial and out-o--distribution perspective. arXiv preprint wwXiv22302.1299.
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2339-2352.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Fed Chi, and Denny Zhou. 2022a. Self-consisenncy improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi. Yeganeh Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An- jana Arunkumar, David Stap, et al. 2022b. Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 5085-5109.
Alessandro Stolfo, Zbijing Jin, Kumar Shridhar, Bern- hard Schölkopf, and Mrinmaya Sachan. 2022. A causal framework to quantify the robustness of math- ematical reasoning with language models. arXiv preprint arXiv:22 10.10010011 12023.
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang. Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. 2022. On transferabil- ity of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint 2rX11:2201.11003.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-cove coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022. Black-box tuning for manguag--moddlla-----eeeee
In International Con- ference on Machine Learning, pages 20841-20855. PMLR.
Weng Lam Tam, Xiao Liu, Kaixuan Ji, Lilong Xue, Xingjian Zhang. Yuxiao Dong, Jiahua Liu, Maodi Hu, and Jie Tang. 2022. Parameter-efficies prompt tuning makes generalized and calibrated neural text retrievers. arXiv preprint arXiv:2207. 07707.
Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yi- dong Wang. Hammeng Liu, Jindong Wang. Xing Xie, and Yue Zhang. 2022. Glue-x: Evaluating nat- ural language understanding models from out- of-distribution generalization perspective. arXiv preprint arXiv:22 0ete:22/1000000 1.08073.
Ruixiang Tang, Dehan Kong, Longtao Huang, and Hui Xue. 2023. Large language models can be lazy leam- ers: Analyze shorteuts in in-context learning, arXiv preprint arXiv:2305. 17256.
Wentao Ye, Mingfeng Ou, Tianyi Li, Xuetao Ma, Yi- fan Yanggong, Sai Wu, Jie Fu, Gang Chen, Junbo Zhao, et al. 2023a. Assessing hidden risks of 1lms: An empirical study on robustness, consistency, and credibility. arXiv preprini 2rXiv:2305.10235.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023b. Large language models are versatile decomposers: Decompose evi- dence and questions for table- reasoning. arXiv preprint arXiv:2301.
Nirali Vaghani and Mansi Thummar. 2023. BFlipkart product reviews with sentiment dataset. https:// www. www.kaggle. com/ dww/
Tianjun Zhang, Xuezhi Wang. Denny Zhou, Dale Schu- urmans, and Joseph E Gonzalez. 2023. Tempera: Test-time prompt editing via reinforcement learning. In The Eleventh International Conference on Learn- ing Representations.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2021. Spot: Better frozen model adap- tation through soft prompt transfer. arXiv preprint 10X.9:2100.07944.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 30Xiv:/804.07455
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015 Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.

---

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations.
Train& Val
Test
N Shot
K Prompt
Yelp
650000
150
36
Flipkart
75138
IMDB
25000
Amazon
100000
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang. Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023.
SocialIQA
33410
150
36
PIQA
16113
OpenbookQA
4957
Promptbench: Towards evaluating the robustness of large language models adversarial prompts. arXiv preprint arXiv:2306.0444 04528.
Number
2000
150
Spans
2000
MNLI
392702
1000
16
ANLI
45460
A Appendix
RTE
2490
277
16
4
A.1 Dataset Details
HANS
30000
1000
DSTC7
43824
150
9
9
For each dataset, we the original training set split into training and validation sets, and ran- domly sample a subset from the original validation set as our test set as sometimes the labels for the original test set are not available. Following the official implementation of APE 2, we split the orig- inal training set with 1000 training samples, and the rest as validation samples. For MNLI, we sam- ple the same number of matched and mismatched validation data as the test set. For ANLI, we use R2. For Yelp and Flipkart, we assign the review scores of and as negative, 3 as neutral, and 4, 5 as positive. For multi-tun turn dialog reasoning, we select the instances of MuTual within 5 dialog turns, Ubuntu and DSTC7 within 7 dialog turns, and reduce the number of choices to 4 for all three datasets. We show an example of LLM input for each task in Table 1, and the dataset statistics in Table 10.
Ubuntu Dialog
94107
MuTual
4783
Table 10: Statistics for the train, validation and test splits for each dataset, and the values of shot number N and prompt number K for each task. The Train& Val are further split into 1000 training samples and the rest as validation samples.
prompts by LLM for further selection. For each task, we try the value of N as 9, 16, 25, 36, and K as Nss factors, to ensure obtaining effective prompts, where APE is not very parameter sensi- tive. Moreover, we ablate the Monte Carlo search since it is optional and not significant for our tasks. Given the randomness of the backbone LLM, we set the temperature of the LLM as O, top as 1.0. We set the max tokens for prompt generation as 100 to try to ensure truncation, and keep other LLM parameters the same as the official APE implementation. The parameters N and K are shown in Table 10.
A.2
Additional Implementation Details for Preliminary Experiments
The APE performs prompt optimization by itera- tively generating and selecting the prompts lever- aging LLM. For prompt generation, it utilizes meta prompt instruct LLM to infer prompts from given outut- examples. Then, the generated prompts are evaluated on validation data to select the prompts with good task performance. After that, APE leverages LLM to perform Monte Carlo search by iteratively paraphrasing the current effec- tive prompts and performing evaluation on them to obtain optimized prompts.
A.3 Additional Details and Results for the Exploration on the Factors Affecting Prompt Robustness.
Calculation of Q1 Metries. The label distribu- tion shift quantifies the divergence of the label distributions between two groups for classification tasks, calculated by the KL divergence of their label distributions.
DKL = Σ Pr.(y)log( Prs(y) Pr((v)
Following the official implementation. for prompt generation, the sampled N-shot training data are divided into splits to generate K
DEY
where y is the label space of the task, and Pra(y) and Pry(y) denote the probability the label y in the source and target groups, respectively.
http:////wierp/aaaattt...p.... engineer/ttre/maiin

---

Dataset
Input Example
Labels
Yelp
Dr. Goldberg offers overything look for in a general practitioner. He's nice and casy to talk to without being patronizing; he's always on time in seeing his patients...
positive, negative, neutral
OpenhookQA
The sun is responsible for (A) puppies learning new tricks (B) children growing up and getting old (C) flowers wilting in a vase (D) plants sprouting, blooming and wilting.
B,
MNLI
Premise: One of our number will carry out your instructions minutely. Hypothesis: A member of my leam will execute your orders with immense precision.
entailment, neutral, contra- diction
HANS
Sentence I: The doctors supported the scientist. Sentence 2: The scientist supported the doctors.
entailment,
non entailment
DSTC7
S: Hello! A: Hello! S: I'm wondering for next semester what class should I take, A: Given your experience, I suggest you take EECS 280, S: Do you know what the size of that class is? Answer Choices: (A) EECS 481 covers dealing with structuring principles, pragmatic aspects of the production of software systems, design methodologies and informal analysis. (B) The class size normally around 167 students. Based the classes you't taken, this class shouldn't it be extremely demanding. (D) This course has discussion section.
A. B. C
Number
Question: How many pereent weren't 45 to 64? Context: In the city. the year 2010 e.g., 78.9 population was spread out with 26.3% under the age of 18, 13.6% from 18 to 24, 30.7% from 25 to 44, 21.1% 1s from 45 to 64, and 7.2% who were 65 years of age or older. The median age was 32 years. For every 100 females, there were 92.5 males. For every 100 females age 18 and over, there were 88.4 males.
Table 11: Dataset examples for each task. The output for classification tasks is one of the Labels, while for Number the output is string of numerical value.
The input similarity quantifies the n-gram simi- larity of the input corpuses of the two groups. Sup- pose that we sample M inputs from the source and target groups respectively, denoted 2i "Fa and ae (ait...., atar ), we calcu- late the Spearman's 's rank order correlation between the bag-of-word vectors of Kg and Ft.
For both metrics, the n-gram(-) is calculated as both word I I-gram and character 4-gram using scikit-learn.
Q1 Metrics for More Tasks. Table 12 and Ta- ble 13 show the two Q1 metric results for common- sense QA and Dialog tasks. Linking the results with the generalization performance in Table and Table 2, we have the following observations. 1) For each target group of the commonsense QA task, the largest value for input similarity coheres with the best generalization performance, but the smallest value of label distribution shifts does not correlate to the best generalization performance. 2) For the Dialog groups, the zero label distribution shifts and the close input similarities cohere with the subtle generalization performance difference on each target group. 3) The evaluation metries cannot be compared across target groups nor across tasks. e.g., the source group SocialIQA performs better on PIQA than OpenbookQA (cf. Table 2), but the input similarity is higher for OpenbookQA. Also, MuTual has smaller input similarity with Ubuntu (input similarity is 0.56, and generalization per- formance is 74.7) but better generalization perfor- mance than PIQA generalizing SocialIQA (input similarity is 0.57, and generalization performance is 68.9) (cf. Section 2). These findings reveals the benefits and limitations of the QI metrics.
P = cov Vs, Vi) ((V))
where Vg and V2 denotes the ranked bag-of-word vectors of na and 2t on the vocabulary of Iz.
Calculation of Q2 Metries. We sample the same amount of inputs from SocialIQA, PIQA and Open- bookQA, and denote the input corpuses as æ1, 22 and 13. Firstly, we calculate the proportion of unique n-grams for each group against the num- ber of all n-grams for the three corpuses as
[x:)) n- I-grams(( (r1, 22, x3))
i 1,2,3
where n-gram(-) returns the set of unique n-grams, and the braces denotes mixing the inputs. Secondly, we think the source group that has already covered a larger proportion of n-grams of the target group may promote better generaliza- tion, and we calculate the proportion of n-gram coverage between the source and target groups as
I-grram((z)) ( [x)r In-rrams(zz))

---

Target
ference for Vicuna models. We present the meta prompt APE and APE-ut, the initial prompt for APO, the human-written prompts, the revised ver- sions by PromptPerfect here.
S
PIQA
OpenbookQA
Source
SocialIQA
2.44
0.27
PIQA
0.38
0.59
OpenbookQA
1.59
3.17
(a) Commonsense QA
Tagget
APE meta prompt:
Mutual
DSTC7
Ubantu Dialog
Source
I provide my friend with an instruction. Based on the instruction, gave him several inputs, and he generated the corresponding outputs. Here the input-ouput examples:[DEMo Please briefly illustrate the instruction and describe the output format. The instruction is
Mutual
0
DSTC7
0
0
Ubantu Dialog
0
(b) Dialog
Table 12: Results for label distribution shifts. Smaller value indicates smaller distribution shift. Bold font indicates the smallest value for each column.
APE-ut meta prompt:
Target
I provide my friend with an instruction. Based on the instruction, I gave him several in- puts, and he generated the corresponding outputs. Here the input-output output exam- ples:[Source). Here also some unlabeled examples. Please consider these examples as well for prompt generation: Cemeration:Unnaabllee Unlabbeedd Tar- get]. Please briefly illustrate the instruction and describe the output format. The instruc- tion is to
SocialIQA
PIQA
OpenbookQA
Source
SocialIQA
0.59
0.62
PIQA
0.57
0.69
OpenbookQA
0.61
(a) Commonsense QA
Target
MuTual
DSTC7
Ubuntu Dialog
Source
MuTual
0.55
0.56
DSTC7
0.56
0.56
Ubuntu Dialog
0.57
0.57
(b) Dialog
Table 13: Results for input similarity. Larger value indicates smaller distribution shifts. Bold font indicates the largest value for each column.
APO initial Prompts:
For Yelp: Provide sentiment analysis of the following text. Answer Positive Neutral Negative as labels.
For SocialIQA: Give answer the following multi choice question. Provide only the single letter as labels.
A.4 Details for Baseline Implementation
For all compared methods, the LLM parameters such as temperature, top p. max tokens are the same as in Appendix A.2. The implementation and results for APE follow the preliminary experiments illustrated in Appendix A.2 and Section 2. For APO, we follow the original parameter setting ex- cept for number of optimization step as 1 because the three tasks do not need multi-round optimiza- tion. For GPO, the value K is unchanged from APE. The consistency threshold for GPO are 0.83 (5 out 6 prompts) for sentiment analysis and com- monsense QA, and 0.33 (2 out of prompts) for DROP. Note that APE and APO are not designed to utilize the unlabeled target group data so we only observe the direct generalization performance, while APE-ut and GPO utilize the N-shot source group data and N-shot target group data. All of the above methods do not need to apply Monte Carlo search following the official implementation of APE. We use one 32GB GPU to perform in-
For Number: Answer the following question based on the context which involves numerical calculation. Provide only the numerical value that directly answers the question.
Human Prompts:
For sentiment analysis: Provide a sentiment analysis of a given input text. The output for- mat a single word indicating whether the sentiment is positive, negative, or neutral. For commonsense QA: Give answer to the fol- lowing multi choice question which involves commonsense knowledge. Provide only the single letter (a, b, or d) labels.
For DROP: Answer the following question based on the context which involves numer- ical reasoning. Provide only the direct answer the question, which can be a numerical value or a short string.

---

PromptPerfect:
Table 2.
For sentiment analysis: Your task is to perform sentiment analysis on a given input texi and provide a single word indicating whether the sentiment is positive, negative, or neutral. The input text may contain any language or style of writing, Please ensure that your analysis takes into account the overall tone and con- text of the text. Your response should be con- cise and clear, providing a single word that accurately reflects the sentiment of the input text. If there are multiple sentiments present in the text, please choose the one that best represents the overall feeling conveyed by the author. note that your analysis should take into account all relevant factors, such as tone, language use, and content. Your re- sponse should also be flexible enough to allow for various types of imput texts.
Yelp
Provide feedbuck various experiences, such as dining, shopping, and service. The output format is a sentiment analysis, where the isput is analyzed to determine whether the experience was positive, negative, neutral. The output is single word indicating the sentiment of experience. Provide a sentiment analysia of customer reviews. The input consists of customer review of a prod- uci, and the outpui is a binary classiicctiio of the sentiment either positive or negative.
Flipkurt
GPO
provide sentiment analysis of a given text. The ouput format is single word indicating whether the sentiment is positive, negative, or neutral.
Number
Answer a specific question based given context. The ouput format is numerical value that directly answers the question asked.
Spans
Answer a specific question based on given context. The output format is a single word or phrase that di- rectly answers the question asked. Answer questions based
GPO
given con- text information. The output format is a manerical value or a single word answer.
For commonsense QA: Please choose the best answer for the following multiple choice ques- tion. Choose the one answer that best fits the given scenario. Please provide only the single letter (a, b, or d) as labels.
Table 14: Case study on the prompts optimized by APE from a source group, and GPO.
For DROP: Your task is to answer numeri- cal question based on given context involv- ing numerical reasoning. Please provide direct answer to the question, which can be a numerical value or a short string. Please note that your response should be concise and directly answer the question. The ques- tion may involve various numerical data, such as percentages, averages, or counts. You should focus on identifying the relevant in- formation and providing clear and accurate answerAdditionally, please ensure that your response is flexible enough to allow for var- ious relevant and creative answers based on the context provided.
K
Flipkart Ensemble
3
81.2 + 1.3
6
84.5 = 2.0
9
85.8 + 1.9
12
85.2 + 1.8
18
85.3 + 1.4
Table 15: Generalization performance of GPO on Flip- kart with different numbers of candidate prompts K.
A.6 Study on the Impact of the Number of Candidate Prompts
We examine the effect of varying the number of candidate prompts K on GPO performance in our 36-shot sentiment analysis task. We test the K values in (3, 6, 9, 12, 18). The results the target shown in Table 15. We observe group Flipkart that the generalization performance stabilizes as K reaches a specific value, in this case is 6, indicating that further generating more prompts are unlikely to yield significant improvements in performance.
A.5
Case Study
We present a case study by presenting the best prompt among the five runs for sentiment analysis and DROP as shown in Table 14. We can observe that the optimized prompt for a single group often contains group-specific background information as highlighted by underline which may hinder robust prompt generalization. On the contrary, the opti- mized prompts of GPO are more general and thus performs well on both groups. Note that for Spans, the optimized prompt is also general enough and thus can generalize well to Number as shown in


Reviews:
[{"review_id": "2gXVAOX5bi", "reviewer": "Reviewer_XdbA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback regarding the comprehensiveness of the paper and the substantiality of the results. \n\n**Reason to Reject 1**\n\nIn Table 9, we have explored combinations with GPT-3.5, Vicuna 7B, and Vicuna 13B, by comparing APE, the best baseline on the sentiment analysis dataset Flipkart, and GPO on Flipkart. In addition, we do new experiments with GPT-4. We will add the new results to our paper in the next version. Here we summarize the results of combining with various LLMs on Flipkart. \n\n| LLM  | APE ensemble | GPO ensemble |\n\n| Vicuna 7B | 43.9 $\\pm$ 21.3 | 71.9 $\\pm$ 13.1 |\n\n| Vicuna 13B | 68.3 $\\pm$ 13.7 | 70.7 $\\pm$ 10.8 | \n\n| GPT-3.5 | 80.5 $\\pm$ 2.1 | 84.5 $\\pm$ 2.0 | \n\n| GPT-4 | 83.3 $\\pm$ 0.0 | 85.4 $\\pm$ 2.4 | \n\nFrom the table, we can observe that 1) the performance gap between APE and GPO exists across LLMs, even for the state-of-the-art GPT-4, showing that the generalization issue exists across LLMs; 2) GPO largely outperforms APE, showing its effectiveness. \n\n**Reason to Reject 2**\n\nThanks for your insightful comments. As illustrated in Appendix A.2, our approach divides the N-shot data for prompt generation into K splits, where each split generates one candidate prompt, thus we experiment with different numbers of candidate prompts K in the range of N's factors. For example, on the 36-shot sentiment analysis task, we show the performance on Flipkart with the values of K in {3, 6, 9, 12, 18}. \n \n| values of K | 3 | 6 | 9 | 12 | 18 | \n\n| GPO ensemble | 81.2 $\\pm$ 1.3 | 84.5 $\\pm$ 2.0 | 85.8 $\\pm$ 1.9 | 85.2 $\\pm$ 1.8 | 85.3 $\\pm$ 1.4 | \n\nFrom the above table, we can observe that the performance stabilizes as K reaches a specific number, in this case, 6. This finding suggests that, beyond this point, further increases in K would not significantly enhance performance. We will add more analysis on hyperparameters into the Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The manuscript is well-structured and easily comprehensible.\n- The experimental results substantiate the efficacy of GPO.\n- This work examines the distribution shifts across diverse tasks.", "reasons_to_reject": "- The baseline method lacks experiments that explore combinations with various LLMs.\n- Certain hyperparameters seem to influence overall performance, such as the number of candidate prompts $K$, an aspect overlooked in the authors' analysis.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study presents the Generalized Prompt Optimization (GPO) framework for LLMs, aiming to enhance generalization performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NRtbxJWeZH", "reviewer": "Reviewer_P2wd", "summary": "", "questions": "A. What is the effect of the type of in-context examples on the upper bound of the accuracy? Was that evaluated at any point. Since the work has many real world applications it would be interesting to analyze this effect to see if the examples can be optimized to get better upper bound accuracy. The model might also be sensitive to the order of these examples. Was any similar experiment conducted?\nB. What are the parameters for generation like top p, temperature, frequency penalty and max tokens for the meta prompt and optimized prompt. What was the optimization approach for these parameters?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the importance of the problem, the extensiveness of the experiments, and the clarity of the paper writing. \n\n**Reason to Reject**\n\nThanks for your great question. As illustrated in the related work, there are several types of gradient-free prompt optimization techniques. The rationale for selecting compared methods is explained as below: \n- Search-based methods: This category of methods is the current state-of-the-art on black-box API LLMs, where we adopt APE and APO as baselines. \n- RL-basaed methods: Comparison with RL-based methods was omitted due to their substantial underperformance in relation to APE and APO, as evidenced in the APO paper (Pryzant et al., 2023). We conjecture that RL-based methods might not be suitable for prompt optimization with limited data examples due to the limited reward signals. \n- Other gradient-free methods: Although some methods claim suitability for black-box LLMs, they are not entirely compatible for black-box API LLMs like ChatGPT. The evolutionary algorithm (Sun et al., 2022) needs to input prompt embeddings to the LLM, and the boosting method (Hou et al., 2022) requires passing gradients through the LLM. Prompt embeddings and passing gradient are unavailable for the black-box API LLMs. \nTherefore, we mainly focus on the state-of-the-art search-based methods for investigation. \n\n**Question A**\n\nThanks for pointing out these insightful questions. \n- For the impact of the order of the in-context examples, we choose one data split of Yelp and Flipkart, and randomly shuffle the order of the in-context examples for five times. The averaged GPO performance is shown below. \n\n     | Dataset | Top 1| Ensemble| \n\n     | Flipkart | 80.6 $\\pm$ 1.2| 80.9 $\\pm$ 1.3| \n\n     | Yelp | 78.5 $\\pm$ 0.6 | 79.0 $ \\pm$ 0.9| \n\nWe can observe that the variances are all small, showing that the GPO performance is not very sensitive to the order of the in-context examples. This is probably due to the small differences in candidate prompts generated by varying in-context example orders, thereby yielding small impact on performance disparities.\n- We have not yet conducted experiments on different types of in-context examples. To achieve this, it is essential to classify existing in-context examples into different types, which are not directly available. We will try to design classification strategies and investigate their effects in the future. Additionally, in line with your insightful idea, optimizing in-context examples is a promising direction, and we have earmarked it for our future endeavors (mentioned in the future work and limitations). \n\n**Question B**\n\nAs illustrated in Appendix A.2, we set top p as 1.0, temperature as 0 to control the randomness of LLMs for better result replication. About the max tokens, for the input meta prompt with in-context examples, the length limit aligns with the LLM's input length limit, where truncation will not occur. For the generated optimized prompts, we set the max tokens as 100 to ensure no truncation of the generated prompts. For other parameters for generation, including the frequency penalty, we follow APE since APE is our base method with the same prompt generation and evaluation process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work identifies an important real world problem which can be deployed to production for multiple products with some great applications for difficult tasks. \n- This work tackles the lack of robustness of prompt optimization to distribution shifts for different prompt engineering tasks.\n- It is interesting how this works formulates the motivation and problem empirically.\n- GPO uses LLM capabilities for labeling and prompt optimization.\n- The authors demonstrate improve robustness to distributional shifts through extensive experiments across multiple datasets and tasks, over different baselines along with a detailed ablation study and analysis.\n- The paper is well-written with clarity of technical details and easy to understand.", "reasons_to_reject": "- Lack of comparison with other gradient-free prompt optimization techniques. What is the effect on robustness and upper bound of accuracy in comparison to baselines mentioned in the related works?", "questions_to_authors": "A. What is the effect of the type of in-context examples on the upper bound of the accuracy? Was that evaluated at any point. Since the work has many real world applications it would be interesting to analyze this effect to see if the examples can be optimized to get better upper bound accuracy. The model might also be sensitive to the order of these examples. Was any similar experiment conducted?\nB. What are the parameters for generation like top p, temperature, frequency penalty and max tokens for the meta prompt and optimized prompt. What was the optimization approach for these parameters?", "paper_topic_and_main_contributions": "- Existing prompt optimization techniques find the best prompt for the labeled data which doesn't necessarily generalize to test data with different distributions.The authors propose the GPO approach to overcome this.\n- They propose a Generalized Prompt Optimization (GPO) technique that incorporates LM generated labels for a portion of the target data and optimizes on both the source and LM labeled target data.\n- The GPO approach generates an ensemble of prompts based on examples, applies this ensemble to the target data, filters target examples based on prompt agreement and use these target examples mixed with source examples for prompt optimization.\n- The authors show that GPO improves target distribution performance on different benchmark datasets. \n- This approach has many great applications.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mVEcYEo7XQ", "reviewer": "Reviewer_Uxx8", "summary": "", "questions": "- How do you generate multiple outputs from these LLMs? is it the API or a specific decoding algorithm?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback regarding the quality of the paper writing, the clarity of the problem definition and experiments, as well as the potential for future research. \n\n**Reason to Reject**\n\nThanks for your perceptive comments. Firstly, our GPO with the meta prompt is tailored for black-box API LLMs such as ChatGPT which have garnered substantial attention recently. Their model parameters and gradients are not accessible, leading to gradient-free prompt optimization solutions such as APE and GPO. Secondly, for the medium-sized LMs, their model parameter and gradients are usually accessible. As such, there is no need to utilize these gradient-free prompt optimization methods. Instead, gradient-based approaches tend to yield better performance. In light of these, it is probably not necessary to develop a gradient-free method suitable for both black-box LLMs and medium-sized LMs. \n\n**Question**\n\nThanks for your question. We use the API to generate the outputs. To further clarify, we do not sample multiple output for one input. Instead, we create multiple inputs with different in-context examples, and generate one prompt for each input. \n\nFor the missing reference, thanks for pointing it out. We will discuss these related literature and revise the paper accordingly. We will also fix the typo.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is a well-written paper. The hypothesis and experiments are well defined and explained. The experimental analysis conducted by this paper over six different NLP tasks across 16 datasets clearly shows that with a recent gradient-free prompt optimization technique such as APE over a strong LLM such as GPT3.5, we may have a large performance gap if we optimize the prompts on the source group while testing on the another target group. This can form a standard benchmark for future research about the generalization and robustness of these prompt optimization techniques.", "reasons_to_reject": "- The proposed solution to solve the robustness issue under domain shift is not very ML grounded and cannot be generically applied to smaller LM models such as BERT-large or T5-large. The Meta prompt you have used to generate task instructions to be used for labeling the unlabeled target group, may not be applicable to these medium size LMs.", "questions_to_authors": "- How do you generate multiple outputs from these LLMs? is it the API or a specific decoding algorithm?", "paper_topic_and_main_contributions": "- The paper studies the robustness of the prompts for LLMs optimized using a recent gradient-free method. The analysis shows large performance gaps across famous NLP datasets. The solution to incorporate the unlabeled data from the target domain while optimizing the prompts on the source data improves the robustness under domain shift.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 047, 235 fix typo."}, "extra_text": {"Missing_References": "- Your solution for improving prompt robustness is similar to using any unlabeled data to augment the supervised examples for improving performance, although your unlabeled data is originating from a target group. The paper does not discuss the past literature of using unlabeled data in NLP tasks. At least such methods must be discussed as part of the related work section, although your noise reduction techniques are based on prompt ensemble and specific to Large LMs."}}]

Author rebuttal:

Meta-review: In light of the significant recognition garnered by LLMs for their exceptional performance across various Natural Language Processing tasks, the exploration of prompt optimization emerges as a highly promising avenue for research. 
This holds particular significance in the context of black-box LLMs such as ChatGPT.
This paper introduces a gradient-free prompt optimization technique, characterized by its robustness in mitigating the challenges posed by data domain shifts.
All reviewers have agreed that the proposed Generalized Prompt Optimization (GPO) framework is useful for improving generalization, especially under domain shifts.
While some concerns have also been raised that the comparison is a bit weak. 
And during the rebuttal period, the authors have made efforts to address the concerns.
Generally, the merits of this paper outweigh its flaws.
Decision: Accept-Main