Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models
Authors: Chengzhengxu Li
Xiaoming Liu
Zhaohan Zhang
Yichen Wang
Chen Liu
Yu Lan
Chao Shen
Venue: NeurIPS 2024 poster (2024)
Keywords: Prompt Optimization
Domain Generalization
Few-shot Learning
Pre-trained Language Models
Primary area: natural_language_processing
TL;DR: This paper explores the domain generalization ability of prompts in PLMs, discovering that prompts with higher "Concentration" are more generalizable, leading to new optimization methods that outperform existing techniques.

Abstract: Recent advances in prompt optimization have notably enhanced the performance of pre-trained language models (PLMs) on downstream tasks. However, the potential of optimized prompts on domain generalization has been under-explored. To explore the nature of prompt generalization on unknown domains, we conduct pilot experiments and find that (i) Prompts gaining more attention weight from PLMs’ deep layers are more generalizable and (ii) Prompts with more stable attention distributions in PLMs’ deep layers are more generalizable. Thus, we offer a fresh objective towards domain-generalizable prompts optimization named ''Concentration'', which represents the ''lookback'' attention from the current decoding token to the prompt tokens, to increase the attention strength on prompts and reduce the fluctuation of attention distribution.
We adapt this new objective to popular soft prompt and hard prompt optimization methods, respectively. Extensive experiments demonstrate that our idea improves comparison prompt optimization methods by 1.42% for soft prompt generalization and 2.16% for hard prompt generalization in accuracy on the multi-source domain generalization setting, while maintaining satisfying in-domain performance. The promising results validate the effectiveness of our proposed prompt optimization objective and provide key insights into domain-generalizable prompts.

Concentrate Attention: Towards DemainnGenerallzablee Prompt Optimization for Language Models
Chengzhengxu Li", Xiaoming Liu Zhaohan Zhang?, Yichen Wang", Chen Liu', Yu Lan', Chao Shen'
Faculty of Electronic and Information Engineering, Xi'an Jiaotong University
QQueen Mary University of London, London, UK Corresponding author
University of Chicago
{czx 1i, lcoder) Jastuxxssjjjie edu.cn {xm. liu, ylan2020, chaoshen]@xjtu, edu cn zhaohan. zhang@qmul ac.uk yichenzw@uchicago.ee edu
Abstract
Recent advances in prompt optimization have notably enhanced the performance of pre-trained language models (PLMs) on downstream tasks. However, the po- tential of optimized prompts on domain generalization has been under-explored. To explore the nature of prompt generalization on unknown domains, con- duct pilot experiments and find that (i) Prompts gaining more attention weight from PLMs' deep layers are more generalizable and (ii) Prompts with more stable attention distributions in PLMs' deep layers are more generalizable. Thus, we offer a fresh objective towards boominn-genealizzae "Concentration", which represents the "lookback" attention from the current de- coding token to the prompt tokens, to increase the attention strength on prompts and reduce the fluctuation of attention distribution. We adapt this new objec- tive to popular soft prompt and hard prompt optimization methods, respectively. Extensive experiments demonstrate that our idea improves comparison prompt optimization methods by 1.42% for soft prompt generalization and 2. 16% for hard prompt generalization in accuracy the multi-source domain generalization setting, while maintaining satisfying in-domain performance. The promising re- sults validate the effectiveness of proposed prompt optimization objective and provide key insights into domain demainngennrallizablee prompts. Our codes are available at https://gi thub. com/ czx- wtw.:///oooonettteeeeeeeee Attention
prompts optimization named
Introduction
Prompt optimization has emerged novel paradigm to effectively fine-tune pre-trained language models (PLMs), demonstrating impressive performance in natural language processing (NI.) tasks, especially under the few-shot setting [Schick and Schütze (220a b). Liu et al. 2023a Unlike traditional fine-tuning methods requiring training and saving entire model parameters Devlin et al. 2018), prompt optimization aims to explore well-performed prompts automatically discrete or continuous space as a context for model input, which boosts model performance on downstream tasks. The mainstream prompt optimization paradigms fall into two categories: hard prompt optimization and soft prompt optinization. Hard prompt optimization relies on selecting well well-performed prompts from pre-constructed prompt set by filtering Tiang
(2020). Haviv et al.] [2021), Davison
et al. [2019] or gradient-free optimization method Li et al. [2024] Sun et al. [2023], Prasad et al.
22022] Meanwhile, soft prompt optimization searches continuous embedding as prompts via gradient
information guided by task-specific loss function Vu et al. 2021 Li and Liang (021).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, while prompt optimization methods are becoming the mainstream of finetuning PLMs, the domain generalization ability of trained prompts still lacks exploration. Previous works Wu and Shi [2022]. Zhao et al. [2022). Ge et al. [023] Guo et al. [022] attempt to employ domain adaptation methods to address these challenges. These works are based on the assumption of target domain availability. They align the source domain and target domain by unsupervised feature learning. The data reliance on these methods becomes a serious limitation for broader applications because models are frequently exposed to unknown domains. Another branch to enhance the versatility of prompts is pre-training. Guu et al. [202T] pre-trains prompts with 10 GB textual data. Vu et al. [202T] uses three tasks across eight datasets for pre- tre-training obtain transferable prompts. As reported by Liu et al. 2024]. it requires 25-30 hours for pre-trainn pre-training prompts with Roberta-bast on a single NVIDIA A100. The inefficiency and high computational cost remain stumbling block for these methods to be widely used. More importantly, the aforementioned methods are parameterized and not applicable to hard prompt optimization, showing low readability. More studies refer Appendix A
Recognizing the problems mentioned above, we focus on improving the domain generalization ability of prompts with three constraints: (i) do so with no knowledge about the target domain, (it) do so with little training cost, (itt) do so with easy adaptation
Performance Comparison
100
Source Target
51
28
90
43.00
86.45
V2.61
both soft prompt and hard prompt optimization. To get started, we test the popular prompt optimization methods cross- domain setting (i.e., training prompts on one domain and testing them target domair") and show the results in Figure [ Interestingly, these optimized prompts exhibit (i) great performance drop in general (by an average of 8.49%) target domain, validating the existence of research gap mentioned above, (ii) different domain generalization ability in particular (Acc. drops by 2.61% % in best case and by 18.64% in worst case), indicating the existence of distinct prompt "nature" that contributes to its generalizability.
ACC %
80
72 15
3.00
cittof-diitribution
60-
50
ICLI
RLT
Soft! Prompting Methad
ICL
RL
Soft"
Figure 1: Domain generalization capabilities across
various prompting methods (ICL Brown et al. [2020). , RL Deng et al. [022], Soft Lester et al. (202T) in sentiment classification tasks.
Since prompts are functional in the model inference stage in which the model looks up contexts to generate new tokens through the attention mechanism, we probe the attention pattern on prompts during forward propagation with the question "what nature do well-generalized prompts have?" and get the following findings (F) via pilot experiments (:3):
Fi: Prompts gaining more attention weight from PLMs' deep layers are more generalizable.
F2: Prompts with more stable attention distributions in PLMs' deep layers generalize better.
Hence, we propose the idea of Concentration, representing the capability of prompts to get the attention stably from PLMs. We suggest that the concentration indicates the domain generalization ability for prompts, which can be a forebode ahead of the downstream tests.
With the principle of concentration 3 we propose two algorithms that could piggyback upon popular prompt optimization methods for both hard and soft prompts to improve the domain generalization ability of prompts. In the parameterized optimization process of soft prompt $4.1 where the loss function acts as objective, we introduce the concentration-Fs -ww.rightineiiiiingggggnn loss. It minimizes the attention weight on the original input sequence, so as to make the model concentrate on prompts stably for different inputs. the non- aun-paamettrized optimization process of hard prompt $4.2 where the prompt set is first filtered and matched with different inputs by trained agents, we propose the concentration- oriented metric and reward. They aim filter out and match the input with concentration- worthy hard prompts. Experiments show that our method respectively improves the target domain accuracy by 1.42% and 16% over the soft prompt and hard prompt optimized comparison methods, while maintaining in-domain capability.
'The represents using CR Hu and Liu 2004 as the source domain and the represents using SST-2 Socher al. 2013 the source domain. All results shown use MR Pang and Lee (2005) as the target domain.

---

2 Preliminary
This section briefly introduces definitions of the Multi-source Few-shot Domain Generalization (MFDG) problem, which is the primary application scenario of our work.
MFDG Setting. A text classification task, e.g., sentiment classification, is defined as T : - y, where y is the task's ss label space and X' is the feature space. We denote M() to be the marginal distribution over X, and P(Y) to be the prior distribution over y. The domain is then defined by Under the domain generalization setting, the source task is the
DT= = {X, M (X), P(Y), P((())))))))) same as the target task, i.e., To equals to T. But for the source domain DT, and target domain DT, at least one of the underlying probability distribution, i.e., (X), P(Y). or P(Y[X), is different.
In our MFDG problem, the training set is sampled from N source domains Darinn (Di, and the model is tested on unknown target domain D Dust DT.- Also, we follow Perez et al. [222]] to
simulate the few-shot learning setting, which means [Presl >
[Puain]-
MFDG Objective. Traditional prompting methods often rely on a crucial assumption that the training and testing sets come from the same underlying distribution Pusin Des ~ DT. In this context, the objective of prompting is to optimize high high-quality prompt z" that maximizes the expected metric of the prediction on the target domain DT,
arg max () (r(y,PM((
e +x)],,
where r a function that evaluates the quality of the predicted answers when using the prompts 2. For MFDG, the optimization objective is:
arg max BD EG B(a.y)Dr. (r(,,PM(( (z)]]
where G is the set of unknown target domains. In nutshell, Eq. [ searches the prompts well- performed within the known domain, while Eq. explores the prompts that perform well across unknown domains.
3 Concentration Benefits Generalization
In this section, we present pilot experiments to analyze the correlation between domain generalizability and attention concentration of prompts using RoBERTa-Large Liu et al. [2019] as the backbone. Appendix C.T shows the specific form of prompts used in the pilot experiment. From the effect of prompts in forward propagation, we analyze (i) how much each prompt is concentrated by the LM, and (ii) how stable the concentration is to formulate the correspondence.
Background. Attention mechanisms widely studied for PLM interpretability Wang al. 2022
Clark et al. (2019), [in et al. [2019] Hrut et al. 2019]. As for prompt optimization, Wang et al. 2023 provide insights that label words in in-context learning aggregate most of the attention weights in deep layers of PLM, which majorly determine the final prediction. Inspired by this, we further explore the attention weight on the whole prompt sequence and its impact on prompt generalizability from a global perspective.
Definition 3.1. Let 2 (21, 22, 2L)
21) and (e1, e...... eT) be prompt and original input with 2, E S. where S is the set of all possible textual sequences over the vocabulary. Let fe, be the attention block [7] in layer of PLM parameterized by 0,, Then concentration is function
Concentration : S IR+
Concentration(2 = @ x; = Σ Sa, (z5 #x).
2;E2
Heuristicly, concentration represents the "lookback" attention from current decoding token to prompt tokens, as shown in Figure 2
'The attention block refers to key-query attention mechanism which is broadly used in transformers based models. The normalized, inner product of "keys" & and "queries" is computed in the forward pass activations of attention block.
3

---

Prompt
Concentration
Input
Review
the
scripl
1
dark
Sentiment
negalive
Review
Sentment
<maakk
Figure Illustration of Concentration. The tokens the blue square are prompt, and those in yellow are input sequences. Concentration represents the model's attention on prompt tokens in forward pass when decoding <mask> token.
Definition 3.2. Let 2 = (21, 22, zL) and = (e1, e,, eT) be prompt and original input
with 2, where S is the set of all possible textual sequences over the vocabulary. Let #2, be the imput dataset. Let fo, be the attention block in layer / of a PLM. Then D = (z concentration strength a function Strength D R
Strength((2, D); 01) = DD]
Concentration(2 @ 2,; 61).
Concentration strength represents the average concentration across the input dataset.
Definition 3.3. Let D = (#1, 22, IM) be the set of textual sequences sampled from target domain Dr. where 2; S. Then the concentration fluctuation is function Fluctuation - R
(luctuation(::,
[Concentration(z @ m;; 0:)) Strength (z. D); 0:)). (5) Z.ED
Σ
[D]
Concentration fluctuttin demonstrates the variance of concentration strength for different inputs.
Concentration Stren Comparison
Concentration Fluctuation Comparison
15%
84.90%
83.85%
Layer Lager
85
0.7
86.15%
85
0.8
Lewe 22
84.90%
83.85%
Layer 23
80
80
Concentation Stength
Concentation Strongth
0.7
0.6
75
75
Gentrolizzaioo rratin A6
Gencrolization
0.6
70
70
0.5
63
63.15%
665
65
0.5
63.40%
63.15%
0.4
53%
-55
55.53%
55
0.3
50
0.3
50
ICL1
Soft!
Soft
ICL
Soft! RI
Soft"
Prompting Method
Prompting Method
Figure 3: Left: concentration strength of various prompting methods in the last layers (layers to 23). Right: boxplots of the concentration strength in the last layer. Overall, prompts that exhibit good domain generalization gain higher concentration strength and lower concentration fluctuation. The concentration strength of each layer is shown in Appendix C
Our pilot experiment unveils following insights: (7) Prompts with larger Concentration Strength achieve better performance domain generalization. For instance, Figure leff shows that tokens of ICL!, 1 the best-performed method, gain more than 0.8 of Concentration Strength at the 21 layer and over 0.7 at the 23rd layer. (ii) Prompts with lower Concentration Fluctuation tend to generalize to target domain better. As shown in Figure right), Soft and ICI." are concentrated at similar level, but ICL. generalizes better while its stability is better. (iii) High Concentration Strength and low Concentration Fluctuation together contribute most to prompt generalizability. The best-performed ICL. has most Concentration Strength and lowest Concentration Fluctuation across all comparison prompts. These discoveries inspire us to adjust the objective for soft prompt and hard prompt$4.2 optimization towards increasing Concentration Strength while decreasing Concentration Fluctuation.

---

Concentrative Prompt Optimization
4.1 Concentrative Soft Prompt Optimization
To devise soft prompt optimization with the guidance of concentration, first visit the optimization objective of mainstream meth-
Objective: Solve Eq.7
Finding f_1::
Finding More Stable Attention Distribution
Weight
ods (e.g., prompt tuning Lester al. (2021). prefix tuning Li and Liang [222] p-tuning
Prompted Inputs
Dased on 2 Minimize (Eq.))
(su (6) Deec
Cross- Entropy
v2 Liu et al. 2021)).
2 3
Cx
These methods optimize follow log- likelihood objective given a trainable prompt 2 and a fixed PLM parameterized by 0 for the input z:
Optimize Soft Prompt
Based on F_1:
(EQ.8)
Praf x.)
Figure Framework for Soft Prompt Optimization.
max Mog(((u(( @ z);
According to our findings in {3] domain seeaain-generalizbble prompts should be high in concentration strength and low in concentration fluctuation. Thus, we reformulate Eq. to get the objective for domain-generalizable prompts:
Mnxx(log((yle) E 2); 0) + Strength((2, Duain): 0)) s.t. min Fluctuation Duain); ); 0).
Towards the reformulated objective above, we propose the concentration-rewwiighnng loss for soft prompt optimization methods. The framework for soft prompt optimization is shown in Figure 4
First, we minimize the concentration strength on input to improve concentration strength on prompt by designing loss function Les
La = Strength Duain); 9).
(8)
In addition, to reduce concentration fluctuation of prompts, we propose to use every token's con- centration strength as hidden state feature of prompts. denoted as C, (c1, C2 ... CL) where L is the length of prompts. We design contrastive loss to cluster C with same label together to reduce concentration fluctuation:
iDeml
exp(sim CC, C,)/T)
Let
log
E
Dunial
1,z exp(sim(C.. (C, C2)/7)*
PEP(F)
where P(j) represents the input with the same label as the j-th input the dataset Dirain sim(() is used to calculate the cosine similarity between feature embeddings, 1izj is an indicator function, iee. 1igy E {0, 1} if and only if + j. and is a temperature parameter used to adjust the scale of the similarity score.
Also, utilize the cross-entropy classification loss Lee Mao al. 2023 reweighting loss for soft prompt optimization is formulated as:
The concentration-
La = Ace + Acc + AaLes
(10)
where Ages des and der weights different losses in training process. More details are in Appendix D
4.2 Concentrative Hard Prompt Optimization
In contrast to soft prompt optimization, hard prompt optimization searches suitable prompt in discrete space in ponattramtter fashion. Previous hard prompt optimization searches be divided into distribution-level |Prasad et al. [2022), Deng et al. [2022) and input-level Li et al. (2024), Lu al. 20222. Although distribution-leve prompt optimization can generally improve reasoning ability, motivated by the fact that no prompt is perfect for all inputs [Sun et al. (2023), we focus on improving the generalization ability of input-level optimization methods. Generally, the mainstream of lnput-level optimization technique for hard prompts could be encapsulated as: filter (by metric) and match (by RL agents). The findings of concentration could be applied to this optimization process by adjusting filter metric and agent reward. We illustrate the framework for hard prompt optimization in Figure
5

---

In order to Filter Metric. For previous filter metric only considering the overall accuracy on training set, we introduce a new metric called Global Concentration Score (GCS), which involves our ideas of concentration strength and con- centration fluctuation.
Our Objective: Eq.7
Finding 1: Attention Weight (ata)))
Finding F_2:
Distribution
Inputs Data
MARL Prompt Matching
(xx, Desis
Based on F_1: Strength (Bq.4)
PLM Backbone
Towards optimization objective Eq 7 we use
Based on F
Original Prompts
concentration strength as first metric filter out prompts which cannot get much concen- tration from model. Metric for reducing con- centration fluctuation could be regarded as
Set Manual, In-Context...
M_cl [1q.11]
Figure 5: Framework for Hard Prompt Optimization.
minimizing Kullback-Leibh Leibler (KI.) divergence between the concentration features C, of input with same label and the average of C; on whole inputs set Drrain
Ma(z, Duain) = Σ Σ
KL(Softmax(C.) Softmax (Cmgg)) (C )),
(11)
NEY IE Demll)
where y is label space and Darin(y) is the input set labeled in data set Dusin- Also, we follow the setting Li al. 2024] calculating the difference of the probability PLM that the x; is correctly labeled Yuure and wrongly labeled as Vlabe by base PLM to improve the overall accuracy:
Mace Dwain) =
(PLM + zz) PLM 22 @ x.)).
(12)
an E
Finally, we combine the above three metrics as one comprehensive metric, Le., Global Concentration Score (GCS), to assess the quality of prompts:
GCS(z, Punin) Clcce (z, Durin) + Strengtth Pimin)) ;) 0) aarMe((2, Danin), where O'ace. Oigs and Ote are the weights that balance accuracy, concentration strength, and concentra- tion fluctuation, respectively.
(13)
Prompt Matching, Previous methods mostly use a single RL agent to match appropriate prompts for each input Li et al. [2024], Lu et al. (2022), [unn et al. [2023]. Due to the large prompt space, the effective exploration of reinforcement learning agents is limited Dulac-Amold et al. (2019). Furthermore, the MFDG setting, inputs from different domains often have different state spaces, action spaces, and reward scales, then using a single agent often leads to the strategy converging to sub-optimality. To overcome these challenges, we redefine the discrete prompt matching problem in the MFDG setting multi-agent reinforcement learning (MARL) problem and propose new matching algorithm.
We build our matching algorithm based on the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm Yu et al. [2022). Specifically, we configure one reinforcement lcarning (RL) agent for each source domain, collectively forming a multi-agent ensemble A = {1. ,,...,N... .N. effectively share learning experience in different domains, all agents share the same value network v.(.) while having independent strategy networks {Tw,, ()) where & and w,, are the learnable parameters. Also, define set of prompts 2n for each source domain, which serves the action space for the corresponding RL agent. These prompts can come in various forms, including manual prompts Bach et al. [022]] original training inputs Brown al. [2020], Dong et al. [2022]. or LLM- Li et al. [2024). Lu et al. [2021]. Here, an action a" implies that agent n selects a specific prompt n from its designated prompt set Z".
At each step of the training phase, given a state s PLM(x,). which is the last hidden layer embedding of input It, the nithh agent selects an action af by policy Tu,, (a]ls?). This action corresponds to choosing prompt z!" We combine v; and 2/", feed them into the PLM for downstream tasks, and calculate the reward r: The agent's parameters are then optimized based on r..
The rewards received by the RL agent are used as feedback to directly guide the optimization direction of the strategy. In this work, we aim to ensure that the prompts selected by the RL agent have good generalization capabilities. Therefore, we reuse Strength 0) as part of our reward function, specifically rf is defined as:
r = Cacc Mace M2 {x+)) + Strenggt,,, (xx} ;).
(14)

---

In the testing phase, we use an ensemble decision- recision-making approach to apply the prompts, The prompts selected by each agent are input into the PLM perform downstream tasks, and the results are combined. For a given input x and its corresponding selected prompts {=" (=")==1> the final prediction obtained by PLM for label y can be expressed as:
P(ylz) softmax (LM(1)2,22))
(15)
Our intention is to divide the action space of agents into smaller, more manageable subspaces and make easier for agents to make the best decisions. The detailed training and testing processes, along with specific agent settings, are presented in Appendix L
5
Experiments
To demonstrate the effectiveness of our findings for domain generalization, we conduct extensive experiments on tasks of sentiment classification and natural language inference (NLI). We select the SST-2 Socher et al. [2013). MR Pang and Lee [2005], and CR Hu and Liu [2004] datasets for sentiment classification, and the WNLI. QNLI, and RTE datasets from GLUE Wang al. (2018) for
NLI t Each task involves designating one dataset as the target domain and the others as source domains. Detailed descriptions of the datasets and domain divisions are provided in Appendix [B.1
We choose RoBERTaallrree et al. 2019) for all downstream tasks for our hardware resources, and it has been widely used in previous prompt optimization works Li et al. (2024), Deng al.) 2022]. Zhang et al. 2022]. Admittedly, at the time of writing this article, various efforts to optimize prompts have surfaced. However, our goal is problems, but to pose new problem. egg., learning prompts with strong domain generalization ability. We therefore select three of the most well-known methods as baseline in the fields of soft prompt optimization and hard prompt optimization respectively. In addition, in order to more comprehensively demonstrate the performance of our method, we also select two distribution-level discrete prompt optimization methods as comparison methods. The baseline methods and their implementations are described in Appendix B and B.3
build better training method based on previous
5.1 Out-of-domain Performance Comparison
Domain Generalization Result for Soft Prompt. As shown in Table I the concentration strength loss Les and the concentration fluctuation loss Let, in most experimental settings, enhance the domain generalization of three soft prompt optimization methods. And, the combination of Les and Let, i.e. the concentratioo-reweijpe reweighting loss Le.. further improves the domain generalization ability of soft prompts, achieving the best results in all experimental settings. Specifically, Lar (both) boosts the average accuracy of Prompt Tuning, Prefix Tuning, and P-Tuning v2 by 1. 1.47%, 1. 1.78%, and 1.02%, respectively, highlighting its effectiveness in promoting the learning of -omain-invariant properties in soft prompts. In addition, using only Las or Let alone may sometimes impair the performance of soft prompts, such Prompt Tuning and P-Tuning v2 methods when QNLI data is used as the target domain. This indicates that concentration strength and concentration fluctuation are both indispensable for domain generalization ability of the prompts, and enhancing only one aspect may be harmful to the domain generalization performance of the prompts.
To more comprehensively illustrate the utility of the amncentration-reweighii reweighting loss, we delve into Appendix D for complete concentrative soft prompt optimization algorithm. extensive exploration on performance stability to prompt initialization and utility to decoder-only models of our method. Additionally, we provide quantitative analysis and visual representation illustrate the impact of concentration-rewiighinn loss for on soft prompts.
Domain Generalization Result for Hard Prompt. As shown in Table T. with the introduction of filtering metric and prompt matching framework, our approach effectively enhances the domain generalization capabilities of various existing methods. Among them, the improvements to the O method achieved the best performance in all experimental setups. Compared to the original
For simplicity, these datasets denoted by their initial letters (S. M. C. W. Q, and R respectively).
7

---

Sentiment
NLI
Paradigms
Methods
S+M-+C
C+M-S
S+C-M
Q+R-+W
W+R Q
Q+W-+R
Vanilla PT
62,73
65.51265
65. 65.12
41.20
49.83
49,661
Prompt
PT with C..
65.83
66.38242
65.332.7
41,53
49.60237
49,43
Tuning
PT with Cef
65.09).72
67.40243
65.40212
42.17
49.22
49.73
Lester al.
2021
PT with both
66.19x.9
69,542.s
65.89233
42.481.72
50.311.s3
50.421.34
Vanilla Prefix
65.91
83.25
75.51p91
50.266.1
51.88
50,02a28
Prefix
Prefix with Ges
66.231.37
84.32p.4
76.58ua2
50.690.33
$1,44
49.77a.2
Tuning
Prefix with Lef
66.82:.1
83,70p.3
77.17u75
51.730.52
52.12n.2
50.73
Li and Liang 2021
Prefix with both
68.29
85.07
77.53p.43
52.05e.30
53.32
51,26
Vanilla Pv2
65.92
83.84
75.89036
50.630.31
52.76
51.31
P-Tuning
Pv2 with C..
66.061.7
83.32)
75.07pas
51,370.7
50.93
50.20
Pv2 with Cef
66.721.6
84.12151
76.4
51.322.ss
52.64
51,281,22
Liu et 2021
Pv2 with both
67.071.53
84.861.42
77.26m.37
51.87e.2x
53.83
51.57
GrIPS
80.07
84.281 38
85.19
54.37
52.77
53.52
Prasad al. 2022
RI.Prompt
86.05
89.36
85,95
52.77
53.822.3
54.631.99
Deng 2022
Vanilla MP
52.734
55.81
50.85
41,70
50.80
51.60
Manual
MP with MARL
56.37
58.42
52.15
44.27
51.361.4
52,18
Prompt
MP with Metrie
54.632.2
57.84
11.79
42.86x
51.020.8
52.03
Bach et al.
2022
MP with both
56.76a.40
59.44n.3
53.15uas
45.05o.2s
52.03n.15
52.461.24
Vanilla IC
84.33215
84.811.9
80.21
50.861.28
52.63p
58.04223
In-Context Demo
IC with MARI.
85.33
87.022n4
82. 146
52.821.29
53.75
59.8720z
IC with Metric
84.70.
85.10
82.60ss1
51.194.80
52.72aa1
59,46464
Brown 2020
IC with both
87.29
88.49 1.52
83.52
52.94
54.24
60.32
Vanilla D
89. 79.06a7s
90.75p91
86.53p.s
54.84..2
54.85p37
59.78,79
DP O
DP:0 with MARL
87.36.
91.602.3
86.034ma
54,71221
53.131.77
60.62347
Li et al. 2024
DP;O with Metrie
86.79,.32
90.13
86.60a
53.21
54.02g
60.54
DP;O with both
89.63a.s2
92.87ma3
87.85u.47
56.42a.36
55.32n.ss
61.27am1
Table I: Performance comparison of text classification tasks in accuracy with MFDG setting. We double horizontal lines to separate soft prompt optimization and hard prompt optimization methods. denotes the distribution-level discrete prompt optimization methods which not considered in our concentrative hard prompt optimization method, as stated in $4.2
DP2O, our method improve the average accuracy on sentiment classification and NLI tasks by 1.34% and 0.85% respectively. These results demonstrate the effectiveness of our proposed filtering metries and surrogate rewards in selecting universal prompts from a pre-constructed set prompts. Additionally, we find that compared to filtering metrics, the prompt matching framework brings higher performance improvement to discrete prompts. This is because our reward function design adeptly guides the agent match inputs with prompts that have strong cross-domain capabilities, even when faced with an unfiltered set of prompts. We also analyze our method from multiple aspects in Appendix B
Overall Comparison. In the MFDG setting, hard prompts generally outperform soft prompts. As illustrated in Table I the best-performet hard prompt optimization method achieves a significant average accuracy of 73.88% compared only 64.61% for the best soft prompts. We hypnosis that hard prompts embed discrete tokens into the model input, providing precise guidance during testing and making easy for PL.Ms to associate semantics fo input text sequence with the task. And soft prompts rely on indirectly influencing model inference by searching in continuous space with only the guidance of objective function, which might cause overfitting on source domain.
5.2 In-domain Performance Comparison
We also compare the in-domain performance between our proposed optimization objective and traditional training objective. And we report not only model performance tested on in-domain
8

---

Sentiment
NLI
Paradigms
Methods
SST-2
CR
MR
Avg.
WNLI
QNLI
RTE
Avg.
Avg Gup
Prompt
Vanilla PT
63
75.89,
74.17233
74.63
47.64
49.71um
54.73
50.69
+6.66
Tuning
PT with both
72.61
76.07
74.372.2
74.35
46.79) se
49.50
54.21:44
50.17
+4.71
Prefix
Vanilla Prefix
87.39
71.37
82,65
82.47
55,88037
60.27
54.82
56,99
+6.93
Tuning
Prefix with both
87.29
76.73
83,32g33
82.45
56.18
59.74
55,38o.4
57.10
+5.19
P-Tuning
Vanilla Pv2
16.71
77.65
82.276.42
82.21
55, 73.77
60.73
55.16..3
57.15
+6.29
Pv2 with both
87.03
77.71
82,05
82.08
56.31pm
60.461
55,20 A9
57.32
+5.38
Manual
Vanilla MP
61 4222242
57.752.92
53. 53,13233
57.50
44.27pa:
$3/42
$2.63060
50.11
+3.22
Prompt
MP with both
61.33232
16.07
53,476,22
56.96
44.05
13.77
52.60ga9
50.14
+0.40
In Contenettt
Vanilla IC
45.91
85.57492
83.75
85.08
52.37
53.42arz
59.730.1
55.17
+1.65
Demo
86.33134
85. 85.14as7
84.312.1
52.25 oz
52, 52.966u4
59.36g33
54.86
+0.93
Vanilla DP.0
93.62am
90.76a.0
88.58
90.99
55.26,
55.13
61.07Gs1
57.15
+1.44
DP30
DP20
93.20uss
90.38p447
88.37212
56.4704:
55.42
61.29
57.73
+0.37
Table 2: In-domain comparison. The last column shows the average gap between test performance on in-domain and out-of-domain data.
dataset, but also the average gap between performance on in-domain and out-of-domain data. As shown in Table our method shows comparable accuracy with prompt optimization methods aiming only at maximizing log probability on correct label, demonstrating that taking concentration into consideration does not compromise model performance in in-domain data. Moreover, prompts optimized by concentration-drine -rrivviiiinndiinnnnnee objective shows better consistency when tested on both in-domain and out-of-domain data. Especially, for hard prompt optimization which searches for suitable prompts in a limited discrete space, the average performance gap is less than 1%, indicating our method always matches input sequence with proper prompts even if the prompts are not initially designed on target domain.
5.3 Applicability Larger Models and Other Tasks:
We also attempt to extend our method to larger models and more complex tasks. We validate the effectiveness of our method on Llama-2-777 2-b-chat Touvron al. [2023), Vicuma-7b-v1.55 Zheng et al. [2023]. and (ppac-7b-wdifff") Prefix Tuning and In-Context Demo on question-answering tasks. We evaluate our method on ROC, SCT, and COPA datasets from the TRAM Benchmark Wang and Zhao [2023) (referred as R, S, and C for simplicity), covering multiple choice question answering (MCQA) in reading comprehension and commonsense reasoning. The result is shown in Table 3
et al. 20033 models for improving domain generalization ability of
MCQA
Models
Methods
S+C
C+R S
R
+
Avg.
Ace Gap
Vanilla Prefix
62.322
66.302
73.15
67.26
Prefix with both
63.70 96
68.47a97
75.32..0
69.16
+1,90
Llama-2-7b-chat
Vanilla IC
23
65.501 18
17.59
68.74
IC with both
63.131.0
68.332.19
79.83n.s8
70.10
+1.36
Vanilla Prefix
67.721.9
81,092.17
88. 88.97264
79.26
Prefix with both
68.75 68.75
83,931.99
89.762.00
80.81
+1.55
Vicuna-7b-v1.5
Vanilla IC
68.37224
83.234.12
90.98
80.86
IC with both
69.67
85.50
93.39
82.85
+1.99
Vanilla Prefix
61.52
70.032.38
87.91
73.15
Prefix with both
63.89293
72.152.77
89.582.1
75.21
+2.06
Alpaca-7b-wdift
Vanilla IC 60.81
69.112.6
89.662.7
73.19
with both 63.161.56
70.57
91.19200
74.97
+1.78
Table Performance comparison of large models on MCQA task accuracy. The last column shows the average gap between test performance vanilla method and our method.

---

Experimental results show that our method significantly improves the performance of large models on question-ansee answering tasks across multiple domain generalization settings. For instance, for the Llama-7b model, our method improved the average accuracy of soft prompt generalization and hard prompt generalization comparisons by 1. 1.90% and 1.36%, respectively; similar improvements were observed for Vicuna-7h and Alpaca-7b models, ranging from 1.55% to 1.99% and 2.06% to 1.78% respectively.
Additionally, we would also like to discuss "why our method works well for large generative language models?". In Appendix F] we present the Concentration Strength Distribution of prompts using In Do-Centext Demo across three 7B-sized language models (Llama, Vicuna, Alpaca) on three different tasks (SA, NLI, MCQA). We observe that all three LLMs exhibit stronger concentration strength in deeper layers compared to shallower layers when confront with prompts for different tasks. We find that this phenomenon occurs earlier larger models (7B) compared to smaller models like Roberta-large. We speculate that this behavior is related to the alignment stage in pre-training of large models during Supervised Fine Tuning with a large number of prompts.
6 Conclusion
In this paper, we explore the nature of prompts with good domain generalization ability. By conducting experiments on model concentration prompts and concentration pattern stability, we find that well-generalized prompt attract more attention weights at deeper layers pre-trained language models (PLMs) and this pattern stably exists to different inputs. Inspired by these new findings, we propose optimization methods for soft prompt and hard prompt, respectively. For soft prompts, we design concentration-reewiie reweighting loss to search for prompts with strong domain generalization ability in continuous space. For hard prompts, we develop an attention aitention-weighted filter- taben-match framework. This framework first apply novel metric which takes model concentration and pattern stability into consideration to filter out low-quality prompts in candidate set. Then a multi-agent reinforcement learning method is used to match each input with optimized hard prompts from each source domain. Our extensive experiment on multiple datasets in different tasks demonstrates the superiority of our methods over existing comparison prompt optimization methods in terms of MFDG setting.
7 Limitations
In this study, we primarily focused on the performance of domain--eneralizable prompt optimization. Despite this, our research still faces limitations in some practical application scenarios. Firstly, our pilot experiments only covered a limited variety of prompts. In future studies, we plan to extend to more diverse types of prompts. Secondly, the current research mainly focuses on the prompt domain generalization capabilities in a small-sample environment; next, we will conduct more comprehensive performance evaluations on complete datasets. Additionally, our current discrete prompt optimization method is primarily applicable at the input-level; in the future, we plan to explore its potential applications the distribution-leyy Finally, although our method is designed to enhance the performance of PLMs in classification tasks, these methods cannot be directly applied to open-ended generation tasks.
Acknowledgements
We thank all the reviewers and the area chair for their helpful feedback, which aided us in greatly improving the paper. This work is supported by National Natural Science Foundation of China (62272371, 62103323, U21B2018), Initiative Postdocs Supporting Program (BX20190275. BX20200270),. China Postdoctoral Science Foundation (2019M663723, 1692555), Fundamen- tal Research Funds for the Central Universities under grant 200401120244444 144), and Shaanxi Province Key Industry Innovation Program (2021ZDLGY01-02),
10

---

References
S. An, Y. Li, Z. Lin, Q. Liu, B. Chen, Q. Fu, W. Chen, N. Zheng, and J.-G. Lou, Input-tuning: Adapting unfamiliar inputs to frozen pretrained models. arXiv preprint arXiv:2203.03331, 2022.
H. Bach, V. Sanh, Z.-X. Yong. A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Fevry, Z. Alyafeai, M. Dey, A. Santilli, Z. Sun, S. Ben-David, C. Xu, G. Chhablani, H. Wang, J. A. Fries, M. S. Al-shaibani, S. Sharma, U. Thakker, K. Almubarak, X. Tang, X. Tang, M. T.-J. Jiang, and A. M. Rush. Promptsource: An integrated development environment and repository for natural language prompts, 2022.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
H. Chen, X. Han, Z. Wu, and Y.-G. Jiang. Multi-prompt alignment for multi-source unsupervised domain adaptation. Advances in Neural Information Processing Systems, 36, 2024.
K. Clark, Khandelwal, O. Levy, and C. D. Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv: 1906.04341, 2019.
Davison, Feldman, and A. M. Rush. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 conference empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 1178-1178, 2019.
M. Deng, J. Wang. C.-P. Hsieh, Wang, H. Guo, Shu, M. Song, E. P. Xing, and Z. Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint 12555 2022.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint urXiv:1810.04805. 2018.
Q. Dong, L. Li, D. Dai, C. Zheng. Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui. A survey on in-context learning, arXiv preprint vrXiv:2301.00234 2022.
G. Dulac-Amold, D. Mankowitz, and T. Hester. Challenges of real-world reinforcement learning. arXiv preprint 120001 2019.
T. Gao, A. Fisch, and D. Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723. 2020.
C. Ge, Huang, M. Xie, Z. Lai, S. Song. Li, and G. Huang. Domain adaptation via prompt learning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Y. Gu, X. Han, Z. Liu, and M. Huang. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2 009.0433,, 2021.
X. Guo, Li, and H. Yu. Improving the sample efficiency of prompt tuning with domain adaptation. arXiv preprint arXiv:2210.02952, 2022.
A. Haviv, J. Berant, and A. Globerson. Bertese: Learning to speak bert. arXiv preprint 10Xiv:2103.05327. 2021.
P. M. Htut, J. Phang, S. Bordia, and S. R. Bowman. Do attention heads in bert track syntactic dependencies? arXiv preprint arXiv: 122466, 2019.
M. Hu and B. Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177, 2004.
Jiang, F. Xu, Araki, and G. Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
11

---

B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2004.0000 144.00691, 2021.
C. Li, X. Liu, Y. Wang, D. Li, Y. Lan, and C. Shen. Dialogue for prompting: A policy-gradient based discrete prompt generation for few-shot learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 18481-1844 2024.
X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
Y. Lin, Y. C. Tan, and R. Frank. Open sesame: Getting inside bert's linguistic knowledge. arXiv preprint arXiv: 1990609669969 2019.
Liu, J. Xiao, H. Ma. X. Li, Z. Qi, X. Meng, and L. Meng. Prompt learning with cross-modal feature alignment for visual domain adaptation. In CAAI International Conference on Artificial Intelligence, pages 416 428. Springer, 2022.
Liu, W. Yuan, Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9): 33, 2023a.
X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2 10.07602, 2021.
X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang. Gpt understands, too. Al Open, 2023b.
X. Liu, C. Liu. Z. Zhang, C. Li, L. Wang, Y. Lan, and C. Shen. Stablept: Towards stable prompting for few-shot learning via input separation. arXiv preprint arXiv:2404.19335, 2024.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy. M. Lewis, L. Zettlemoyer, and V. Stoy- anov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 190..11907.11699 2019.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051000 2017.
Lu, L. Qiu, K.-W. Chang, Y. N. Wu, S.-C. Zhu, T. Rajpurohit, P. Clark, and A. Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209, 14610, 2022.
Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.0877 087866 2021.
A. Mao, M. Mohri, and Y. Zhong. Cross-cntropy loss functions: Theoretical analysis and applications. In International Conference on Machine Learning, pages 23803-23822 PMLR, 2023.
OpenAl. Gpt-4 technical report. ArXiv, www//aa...0//77//// 2023.
B. Pang and L. Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cs/0506075, 2005.
E. Perez, D. Kiela, and K. Cho. True few-shot learning with language models. Advances in neural information processing systems, 34:1 1054-11070, 2021.
A. Prasad, P. Hase, X. Zhou, and M. Bansal. Grips: Gradient-free, bdst- instruction search for prompting large language models. arXiv preprint arXiv:2203.07281, 2022.
J. Qian, L.. Dong, Y. Shen, F. Wei, and W. Chen. Controllable natural language generation with contrastive prefixes. arXiv preprint arXiv:2202.13223 2022.
A. Radford, J. Wu, R. Child, D. Luan, Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
12

---

T. Schick and H. Schütze. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.0667, 2020a.
T. Schick and H. Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.077 18, 2020b.
R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods natural language processing, pages 1631-1642, 2013.
H. Sun, A. Hüyük, and M. van der Schaar. Query-dependent prompt evaluation and optimization with offline inverse rl. In The Twelfth International Conference on Learning Representations, 2023.
R. Taori, Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-followig laaaa model. https://github. cww/tatsu lab/ stanford_alpace 2023.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
J. Vig. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37-42, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v1/ P19-3007. /tRhstts::/// www. aclweb. ooog anthooogthogygooeeaaaoaoot P09-3007
T. Vu, B. Lester, N. Constant, R. Al-Rfou, and D. Cer. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint arXiv:2110.07904, 2021.
A. Wang, A. Singh, Michael, Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07444 2018.
L. Wang, Li, D. Dai, D. Chen, H. Zhou, Meng. Zhou, and X. Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023.
S. Wang, Z. Chen, Z. Ren, H. Liang, Q. Yan, and Ren. Paying more attention self-attention: Improving pre-trained language models via attention guiding. arXiv preprint arXiv:220.002922, 2022.
Y. Wang and Y. Zhao. Tram: Benchmarking temporal reasoning for large language models. arXiv preprint arXiv:2310.00835, 2023.
H. Wu and X. Shi. Adversarial soft prompt tuning for cross-domain sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1-38-2447, 2022.
Z. Xu, C. Wang, M. Qiu, F. Luo, R. Xu, S. Huang, and J. Huang. Making pre-trained language models end-to-end few-shot learners with contrastive prompt tuning. In Proceedings of the Sixteenth ACM International Conference
Web Search and Data Mining, pages 438-446, 2023.
C. Yu, A. Velu, F. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35: 24611-24622 2022.
Zhang, X. Wang, D. Zhou, D. Schuurmans, and E. Gonzalez. Tempera: Test-time prompting via reinforcement learning. arXiv preprint 11 2022.
L. Zhao, F. Zheng, W. Zeng, K. He, R. Geng, H. Jiang, W. Wu, and W. Xu. Adpl: Adversarial prompt-based domain adaptation for dialogue summarization with knowledge disentanglement. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 245-255 2022.
L. Zheng, W.-L. Chiang, Y. Sheng. S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging alm--s-a-judge with mt-hench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023,
13

---

Appendix
A
Related Work
Prompting for Few-shot Learning. Recent studies indicate that with pre-trained language models (PLMs) developing, prompt-based methods demonstrate significant competitiveness in downstream tasks with few-shot settings. For example, SShhick and Schütze (2020a b propose a semi-supervised training method that converts the text classification task into cloze task through word masking. Meanwhile. Brown al. [2020), Gao et al. [2020), Liu al. [2023b] find that manual prompts can guide large machines to perform NLP tasks without any training. Vu et al. [202T], Li and Liang (02TT. An al. [2022], Qian al. 2022] tune soft prompts using gradient descent with continuous embeddings instead of discrete prompts and achieve performance comparable to fine-tuning in few- shot setting. Although these methods have demonstrated impressive performance, they often rely on critical assumption, i.e., the training and testing sets come from the same underlying distribution. Unfortunately, this assumption frequently does not hold in real-world scenarios.
Domain Adaptation Prompting. To address the out-of-domain challenges, many studies employ domain adaptation (DA) methods acquire prompts that are effective the target domain Ge
et al. 2023]. Guo et al. [2022]. Liu et al. (2022), Chen et al. [2024]. For example, Ww and Shi
2022 propose novel domain adversarial training strategy to learn domain-invariant representations
between each source domain and the target domain. Zhao et al. [2022) introduce three kinds of
prompts learning task, source domain, and target domain features separately. However, these methods still need the involvement of unlabeled target domain samples during training. In contrast to current research, our method expands the exploration of prompt optimization the domain generalization problem where the target domain is entirely unknown during training.
B Experiment Setting Details
B.1 Datasets
In T we provide details of the original datasets used in the main experiments, including type, domain, and label words, for tasks of sentiment analysis and natural language inference (NLI).
Type
Datasets
Domain
Class
Label words
SST-2
Movie Reviews
positiveenegative
Sentiment Analysis
MR
Movie Reviews
positive/negative
CR
Product
2
positive/negative
RTE
News
Clearly/Yet
NLI
QNLI
Wikipedia
Okay/Nonethelesss
WNLI
Fiction Books
Rather/Alas
Table 4: Datasets in the main experiments.
Table 5 shows our specific division of the source and target domain under various settings of MFDG, as well as the sizes of the training and test set.
Type
Setting
Source
Target (Train//Vallddationl
iTest
S M + C
SST-2 & MR
CR
64
2K
Sentiment Analysis
C M + S
CR & MR
SST-2
64
1.8k
S C + M
SST-2 & CR
MR
64
2k
Q+R ++ W QNLI & RTE WNLI
64
0.7k
NLI
W R + Q WNLI & RTE QNLI
64
5.4k
Q+ W R QNLI & WNLI RTE
64
3K
Table 5: MFDG setting for the main experiments.
14

---

B.2 Baselines
We conduct extensive experiments, comparing main competitors, including representative soft and hard prompting methods.
For the soft prompt optimization methods: Soft Prompt Tuning Lester al. [202]
replaces discrete
prompt tokens with learnable embedding, and optimizes prompt through gradient information of
PLMS. Prefix Tuning Li and Liang [021] reparametrizes networks for soft prompts and integrates and adjusts soft prompts at every layer of the PLM. P-Tuning v2 Liu et al. [202T] is an improved
version of Prefix Tuning, which has the option reparameterize the network and use classification headers to adjust the soft prompts of each layer of PLM.
For the hard prompt optimization methods: Manual Prompt applies the prompt set designs of Bach ett al. [002]] randomly combines the prompt with the input for downstream tasks. In-Context Demo Brown et al. [2020) randomly selects training data as examples to prompt PLMs to process subsequent inputs. 2P Li et al. (2024) utilizes GPT-4 OpenAl (2023) to generation a in-context prompt set
and uses the reinforcement learning agent for prompt matching. GrIPS Prasad et al. [2022 optimizes distribution-leve hard prompts by editing on basic prompts, ie., substitution, deletion, and swapping, etc. Prommt (Dng et al. [0022 uses reinforcement learning techniques to individually train partial
parameters of PLMs to generate distribution-level discrete prompts for PLMs on downstream tasks,
B.3 Implementation Details
We provide experimental details for all baseline methods in the main experiment here. We choose RoBERTa-Laarge Liu et al. [2099] as our backbone model. We propose variant setting of the vanilla few-shot learning Perez et al. [2021] J. For all tasks, we randomly select 32 samples from each source domain as the training set to simulate MFDG setting. We use the same approach to build the validation set and ensure that the number of labels in the training and validation sets is balanced. For Soft Prompt Tuning, we replace the Manual Prompt tokens with five soft tokens in the same positions, and optimize them using Adam W Loshchilov and Hutter 2017 optimizer with learning rate 2x10-5 and batch size 32 for 300 epochs. For Prefix Tuning and P- P-Tuning v2, we apply the Adam W optimizer with learning rate of 2x10 and train for 100 epochs. The mini batch size is 8 and prompt length is set as 10. The setting of hard prompt optimization baselines (In- ConCenteet Demo, DP20, GrIPS and RLPrompt) follows Li al. (2024). All experimental results are the average results of 10 different random seeds on a single NVIDIA A100 GPU.
B.4 Training Details
In this subsection, we provide additional details for reproducing our method. In prompt matching framework, each agent's policy network consists of two fully connected layers, w E R 1004x6600 and wa E I2 12600x11 15 The shared value network included three fully connected layers, sized ø' e R. 10224%600 8² E 16000x000 600 and Ø² E R600x1 We use Adam W with eps of 0.00001 during training of 2000 epochs. The learning rate is 0.001, and mini-batch size is 32. Also, in Table 6 and Table 7 we provide the balance weight settings of the soft prompt and hard prompt methods respectively.
Method
Ace
Ags
det
Method
Ocee
Ocss
Oc
PT with both
0.3
0.3
MP with both
10
7
7.5
Prefix with both
0.3
0.5
0.15
IC with both
7.5
7.5
0.15
Pv2 with both
0.5
0.5
0.15
DP20 with both
10
6.5
6.5
Table 6: Weights for soft prompting methods.
Table 7: Weights for hard prompting methods.
C Pilot Experiments
C.1 Pilot Details
To ensure fair comparison between prompts of different lengths. we only select the top four tokens with the highest concentration strength in each prompt for experiment. We randomly select 1000 inputs from the target domain MR dataset for calculation. Results in Figure averages from ten random seeds. In Table g we show the specific methods and forms of the experimental comparison prompts in 3
15

---

ICL::
Review: the script smart and hallelujah for small favors. Sentiment: negative,
Review: Good times to he found here if you love Rockabilly music. I'll definitely be he
back here soon! Sentiment: positive. Review: 
 Sentiment: <mask>
Method:
In- Context Demo Brown et al. [2020)
ICL':
Review: it just not very smart. Sentiment: negative. Review: extraordinary debut from josh koury. Sentiment: positive.
Review: Sentiment: <mask>
Method:
In-Context Demo Brown et al.
2020
RL':

 AgeetMeediaGrraddlllicchllll Grade <mask>
Method:
PlPrmmpt Deng et al.
2022
RL":

 absoluteliterrlll absolute downright downright <mask>
Method:
RLPrompt Deng al. 2022
Soft!:
<soft> <soft> <soft> <softs <soft> <mask>,
Method:
Soft Prompt Lester
Lester et al. [2021]
Soft":
<soft> <soft> <soft> <soft> <soft> <mask>,
Method:
Soft Prompt Tuning Lester et al. [2021)
Table 8: Prompt details of the pilot experiment in
C.2 Attention Distribution Measurement
Figure 15 shows the distribution of concentration strength of various hints in each layer of the Robert-Laare model in the pilot experiment. We can find that PLMs, the concentration strength of almost all prompts is stronger in deep layers than in shallow layers, but there is clear difference in their maximum values. These findings prompt us to further investigate the properties of concentration strength.
Concentration Strength Distribution of ICLT
Concentration Strength Distribution of ICL'
Concentration Strength
Concentraaion neeeetttaiin Strength
Layer
Layer
Concentration Strength Distribution RL
Concentration Strength Distrbuttoo of RL'
Concentration Strength
Concentration Strength
Layer
Layer
Concentration Strength Distribution of Soft!
Concentration Strength Distribution of
Concentration Strength
Concentration Strength
Layer
Layer
Figure 6: Distribution concentration strength of various prompts in each layer of RoBERTa- LoBgRTa-Large.
16

---

D Details for Concentrative Soft Prompt Optimization
D.1 Optimization Process
Algorithm shows the detailed process of concentrative soft prompt optimization in $4.1 It also reveals that our method can be widely applied different soft prompt optimization methods to improve their domain generalization capabilities.
Algorithm 1 Concentrative Soft Prompt Optimization
1: Input: fixed PLM parameterized by 8, training dataset Dusin, learning rate 7), loss weight A Aet
and Ace
2: Initialize soft prompts Asoft with random values
3: while not converged do
4: for each input (Iz, Ms) in Duain do
5:
Construct input sequence (2 2 xz.
6:
Get final prediction P(u (00((2 (2 (5x)::)) Compute loss C.s by Eq.
7:
8:
Compute loss Let by Eq. (
9:
Compute loss Lee by cross-entropy classification loss Mao et al. 2023
10:
Compute final loss by Eq. (10)
11:
Compute gradients Zgoft Dzwn
DT.
12:
Update prompts 2soft f- 2soft 17 Zsoft
end for
14:
end while
Output: Trained domain generalzaeeee soft prompt Ssont
D.2 Stability to Soft Prompt Initialization
We adopt five different soft prompt initialization strategies Gu et al. [2021] to test the stability of our method. "Random" indicates that we randomly initialize the embedding of soft prompt. "Label" indicates that we use the embeddings of the label words. "Vocab" indicates that we randomly sample words from the vocabulary. "Top- Ik" indicates that we randomly sample words from the most frequent 1000 words in the pre-training corpus. "Task" indicates that we randomly sample words from the downstream data.
As shown in Table 9 the results validate that our method enhances the stability of soft prompts under various initialization strategies. The standard deviations of our method target domain SST-2 and QNLI are 1.11 and 0.37 lower than those of the vanilla soft prompt tuning, and the performance is better compared with the vanilla soft prompt tuning.
SST-2
QNLI
Methods
PT with both
Prompt Tuning
PT with both
Prompt Tuning
Random
69.36
65.51
50.31
49.60
Label
68.67
64.21
50.84
49.33
Vocab
66.88
62.03
51.20
50.45
Top- 1k
67.03
62.15
50.05
49.45
Task
68.92
67.29
50.10
48.04
Std.
1.14
2.25
0.50
0.87
Table 9: Comparison of stability to soft prompt initialization. The best result across different templates bold and the worst is double underline.
D.3 Extension to Decoder-only PI.Ms.
We explore the effectiveness of our approach on decoder-only PLMs. Keeping other experimental conditions unchanged, we replace the RoBERTa-Large with the GPT-2-Samll Radford et al. 2019 and perform the corresponding experiments.
17

---

Sentiment
NLI
Paradigms
Methods
S -
C+ M -
C+C
Q+R-> W
W+R- Q
Q+W R
Prompt
Vanilla PT
56.29 .03 57.85091
67.572m 68.52 68.522.7
58.502.1 59.73263
42.641.87 42.79
49.71 1.57
49,4 49,48
Tuning
PT with both
50.27
51.10
Table 10: Decoder-only PLM (GPT-2-Samll) backbone tests in accuracy.
The results in Table TO show that our method works well on the decoder-only PLMs backbone and successfully outperforms representative soft prompt tuning.
D.4 Attention Visualization
We show in Table IT the concentration strength (CS) and concentration fluctuation (CF) obtained in the last layer of RoBERTa-Larg before and after soft prompt are optimized using our method. The results indicate that in the SST-2 target domain, our method not only significantly enhances the concentration strength of soft prompt, but also effectively reduces the concentration fluctuation, thereby achieving significant performance improvements in the SST-2 target domain. However, in the QNLI target domain, the concentration fluctuation of soft prompt increases slightly, resulting in limited improvement in accuracy. This suggests that concenfration strength and concentration fluctuation jointly affect the generalization ability of prompts, which is consistent with observations from our pilot experiments in 3
Target
Method
CS
CF
ACC%
Prompt Tuning
0.523
0.062
65.51
SST-2
PT with both
0.578
0.054
69.36
Prompt Tuning 0.505
0.061
49.83
QNLI
PT with both
0.537
0.062
50.31
Table 11: Accuracy affected by concentration strength (the larger the better) and concentration fluctuation (the smaller the better) before and after using concentrative soft prompt optimization.
In addition, we also use bertviz Vig 2019 to visually display the continuous prompts before and after using concentrative soft prompt optimization. In Figure Figure I0 we show the attention distribution of vanilla soft prompt (left) and soft prompt trained with our method (right) on the same inputs at the last layer of the RoBERTa-Large model. It can be observed that concentrative soft prompt optimization improves the attention concentration and stability to soft prompts at predicted locations.
Layer: 23
Layer: 23
<>
<
<8OFT>
<8OFT>
<BOFT>
<SOFT>
<0FF>>
<SOFT>
<SOFT>
<BOFT>
<SOFT>
<OFT>>
<SOFT>
<masks
<mark>
<madk>
many
monty
meny
many
of
dis
dss
ds
dis
ney
ney
movies
es movica
play
pay on
on
this
hris
thls
Figure 7: Case for attention comparison visualization for soft prompt.
18

---

Layer: 23
Layer: 23
<BOFT>
<BOFT>
<SOFT>
<BOFT>
<5OFT>
<BOFT>
<SOFT>
<SOFT>
<8OFT>
<BOFT>
<30FT>
SOF
<5OFT>
+SOFT T
<5OFT>
+SC T>
<maak>
hed
had
hed
had
for
for
for
for
monthe monthw
muniss mordh
now now
now now
and
and
And and
am pleased
ploased
plaased
pleased
Figure 8: Case 2 for attention comparison visualization for soft prompt.
Layer: 23
Layer: 23
<>>
<>>
<BOFT>
<S0F T>
<SOF 1>
<SOFT>
<SOFT>
<BOFT>
<50FT>
<SOFT>
<SOFT>
<SOFT>
<SOFT>
<SOFT>
<8OFT>
<SOFT>
<S0 T>
<mark
<mast>
the
the
the
the
alows
allws
allos
a aloss
you
you
you
you
to
to
search saarch
search
search
for
for
for
fles fils
files filss
by
artist
antist
stiet
genre
gerre
gene
gonte
and
and
and
and
Figure 9: Case 3 for attention comparison visualization for soft prompt.
Layer: 23
Layer: 23
<>>>
<
<SOFT>
<SOFT>
<SOFT>
<SOFT>
<SOFT>
<SOFT>
<sOfT>
<SOFT>
<BOFT>
<BOFT>
<8OFT>
<SOFT>
<BOFTy
<SOFT>
<SOFT>
<SFF>>
<S0FT T>
<S0FT
use
U14
for
for
for
ripiin
appiig
ripping
rappin
which
which
does
does
does
does
protty
protly
profty
pratty
Figure 10: Case for attention comparison visualization for soft prompt.
19

---

E Details for Concentrative Hard Prompt Optimization
E.1 Optimization Process
We define the prompt matching problem under MFDG setting as multi-agent reinforcement learning (MARL) problem, as shown Algorithm 2
Algorithm 2 Concentrative Hard Prompt Optimization
I: Input: Training set Duiin of size T, testing set Disss fixed PLM parameterized by the prompt sets {2" | filtered by GCS 13 number of agents N.
training the multi-agent RI. model
2: Initialize policy networks Rui :......,
Jun for each agent with parameters wi...., ui and epoch +- 0.
3: while epoch < epochmor do for step f in [1...., T] de
4:
5:
for each agent in II.... N] do
Get state x < (Z. for agent 7.
7:
Run policy network Tw,, (a (s ) to take an action of to select a prompt 2f" from 2".
8:
Calculate reward for agent n1, i.e., Eq. [14]
Add (s af r?") transition to agent n's replay buffer. 10: end for
9:
11: Update of parameters Wiy...;wN using the MAPPO algorithm Yu et 2022
12:
end for
13:
end while
**** testing phase begins
14:
for each input (Ir, yt) in Denin do for each agent n in [1.... .] do
15:
16:
Get state s" PL,().)).
Run policy network Tu,, (a"|s") to take an action a" to select prompt x" from 2".
end for
19:
Get final prediction according to Eq. 15
20:
end for
21:
Output: trained policy network Twr Aww. predictions for test inputs.
E.2 Attention Visualization
We utilize bertviz Vig 2019 to visualize attention distribution the final layer RoBERTa- Large model when processing different inputs with hard prompts filtered by GCS. As illustrated in Figure [T to Figure 12 the filtered hard prompts demonstrate high attention concentration and stability at the predicted positions.
E.3 Stability to Hard Prompt Verbalizer Selection
Prompt-based methods require mapping the probabilities generated by PLMs the label space needed for downstream tasks. Thus, the selection of a verbalizer significantly impacts the performance of PLMs Liu et al. 2023b]. Previous research Xu et al. [023) explores the identification of appropriate verbalizers for these models. The experimental results in Table [2]sow that our method achieves the highest accuracy under different verbalizers settings, which shows that method can improve the
robustness of existing methods for verbalizers selection.
Verbalizer
In -Contex Demo
IC with both
DP:O
DP.O with both
bad/good
81.67
84.16 88.49
89.36
91.73%.43
negative/positive
84.81
90.75ax7
92.870.3
terrible/great
83.32
87.72kss
90.58
92.13%.91
Table 12: Analysis on stability to verbalizers.
20

---

Layer: 23-
Layer: 23-
Layer: 23-
Ravilu
14
best
more
mert
poodvc
looking
viuaily
heaps
siploee
dep sted
glad
entataring
Sonn
Figure 11: Case for attention visualization of three filtered hard prompts.
Layer. 23-
Layer: [23-]
Layer: 23
Redioo
Ravier
Radiow
The
comnady
cyruuyy
not
100
skp
and
parkery
perfedty Ineed
perfecty
tired
Ber
Biri
Banl
Bow
porth
imerd
zottie
postrve
Revive
posiiiee
Roview
Rocdow
intresting
Intorosi
medeinng
roronto
wien
voll
nxmpeing
10l
Sept
Sert
Set
Serd
inort
1947
<reak>
Figure 12: Case 2 for attention visualization of three filtered hard prompts.
21

---

E.4 Sensitivity to Number of Agents
We analyze the impact of the number of agents cue-matching framework. As shown in Figure the experimental results reveal that when the number of agents is small, adding agents can significantly improve the classification accuracy of the target domain. However, as the number of agents continues to increase, the accuracy gradually stabilizes. This suggests that as the number of prompts provided to the input gradually increases, the results of the ensemble decision will become more stable, and increasing or decreasing prompt alone will have less impact on the overall performance of the prompt matching framework.
100
Value
92.87
92.77
ACCS
80
75
Number Agents
Figure 13: Performance for the model with different number of agents.
F
Distribution of Concentration Strength on Larger Language Models
As shown in Figure 14 to Figure 16 we present the concentration strength distribution three prominent open-source LLMs: Llama-2-7b Vicuna-7b-vll 7b-v1.5, and Alpaca-7b- 7b-wdfff. Our findings reveal that almost all three LLMs demonstrate higher concentration strength in deeper layers compared shallower when processing prompts from different tasks. Moreover, larger models exhibit this concentration phenomenon earlier than smaller models.
Concantration Stringth Distibution of the Liama on the SA Task
Concentration Strungth Distribation of the Liema on the NL1 Task
Concentration Strangth
0.8
Concentration Strength
0.6
0.6
0.4
0.4
13
15 17 Layer
19
25
29
33
11
13
15
17 19 21 Layer
31
Concontration Strangth Distribttion of the Liama on the OA Task
0.8
0.4
0.2
0.0
11
13
15
17 Layer
19
21
27
31
Figure 14: Concentration strength distribution of each layer of Ilama in various tasks.
22

---

Concentration Strength Distibutionn the Viaua on the Task
Concentration Strength Distribution the Vicune on the NLI Task
Concentraton Streighh
0.8
Concentraiin Strength
0.8
0.6
0.4
0.2
19
25
33
21
Layer
Layer
Concentration Strangth Distriutioon of the Vicura on the DA Task
Strength
0.4
0.2
0.0
31
Layer
Figure 15: Concentration strength distribution of each layer of Vicuna in various tasks.
Concentration Strength Distribution the Alpaca on the SA Task
Concentration Strength Distribttio the Alpace on the NLI Task
Corcentrston Strength
0.8
Sirength
0.8
0.6
0.6
0.4
0.4
19
Layer
Layer
Concantration Strangth Distribution of the Alpaca on the OA Task
Strengh
0.8
0.8
0.4
0.2
0.0
31
Layer
Figure 16: Concentration strength distribution of each layer of Alpaca in various tasks.
23

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The contribution and scope of this work are discussed in detail in the abstract section and introduction (section Th in this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
24

---

Justification:
Section 7 in this paper discusses the limitations of our work
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: Our work is application based and doesn't 't include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the information to replicate the main experiment the paper in Appendix B And all information for the pilot experiment is provided in Appendix C.1
Guidelines:
25

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide all required code and datasets for this work in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppullc//guus http://////iie/////edububiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
26

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and testing details necessary to understand the results provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We show the random seed settings of the main experiment and pilot experiment of this article in Appendix [B3 (Line number: 518) and Appendix C.T (Line number: 531) respectively. In addition, we publish the standard deviation of all experimental results in Table Table Table and Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
27

---

Justification:
We introduce the types of computer workers in Appendix B.3 (Line Number:
519).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have checked that our work complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is conducted on general tasks the NL. field and has no social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
28

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all datasets (section 5 Line Number: 238, 239) and models (section Line Number: 109 and Appendix D.3) used in this article.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsswiteesseeeeeeeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
29

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "Lj6TauGrZ9", "reviewer": "Reviewer_rw17", "summary": "This paper investigates the domain generalization ability of prompts for pretrained language models (PLMs). The paper finds that prompts that receive higher attention weights from deeper PLM layers and those with stable attention distributions generalize better across domains. The authors introduce a novel objective called \"Concentration\" which implements a \"lookback\" attention from the current decoding token to prompt tokens, aiming to enhance both soft and hard prompt optimization methods. Their experiments demonstrate significant improvements in multi-source domain generalization accuracy—1.42% for soft prompts and 2.16% for hard prompts—while maintaining robust in-domain performance. These findings offer valuable insights into creating domain-generalizable prompts.", "questions": "Please see the Weaknesses above.", "limitations": "The author discussed several limitations of their approaches, including the limited variety of prompts, the focus on a few-shot setting, the restriction of discrete prompt optimization to the input level, and the inapplicability of their methods to generation tasks.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written.\n* The authors started with an initial analysis to inform a novel training objective, which is both insightful and methodologically sound.\n* The proposed method is simple and the authors demonstrate its effectiveness across several classification tasks.", "weaknesses": "I felt there were several obvious questions left unexplored, noted below, which raise concerns regarding the significance of the paper's contributions.\n\n* The authors only experimented with a rather small model, i.e., 355M RoBERTa, which raises concerns about whether the proposed method works with larger model sizes.\n\n* The authors focused solely on a classification tasks (sentiment classification and natural language inference). This raises concerns about the proposed approach's applicability and effectiveness for other tasks, like open-ended generation.\n\n* Finally, the improvements were shown over rather weak baselines. For example, prompt tuning, particularly with small models and limited training data, is a rather weak approach. I also felt that the authors compared their method against a weak implementation of this baseline, using only 5 soft prompt tokens and a learning rate of $2×10^{−5}$. For reference, the original prompt tuning paper used 100 prompt tokens and a learning rate of 0.3, which they found to be critical for prompt tuning's strong performance and faster convergence. These differences raise concerns about the significance of the proposed method's improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive comments and feedback.  We appreciate that our work is considered “well-written”, “novel”, and “insightful”. We hope our response can address your concerns.\n\n**Q1: Applicability to larger models and other tasks:**\n\nWe would like to thank the reviewer for drawing our attention to extending our method to broader applications. We are glad to share that our method is able to successfully be applied on mainstream **Large Language Models and question-answering tasks**. We validate the effectiveness of our method on Llama-2-7b-chat, Vicuna-7b-v1.5, and Alpaca-7b-wdiff models for improving domain generalization ability of prompts on question-answering tasks. We evaluate our method on ROC, SCT, and COPA datasets from the TRAM Benchmark [1] (referred as R, S, and C for simplicity), covering multiple-choice questions in reading comprehension and commonsense reasoning. The result is shown in Table 1.\n\n**Experimental results** show that our method significantly improves the performance of large models on question-answering tasks across multiple domain generalization settings. For instance, for the Llama-7b model, our method improved the average accuracy of soft prompt generalization and hard prompt generalization comparisons by 1.91% and 2.36%, respectively; similar improvements were observed for Vicuna-7b and Alpaca-7b models, ranging from 1.55% to 2.05% and 1.78% to 1.99% respectively. \n\nOur research primarily addresses the analysis and optimization of attention patterns from current decoding tokens to prompt tokens (line. 121-122). Thus, long sequence generation (e.g., open-end generation) is beyond the scope of this research, as we admit in Section 7 (line. 323-325). We will explore the possibility to improve domain generalizability on more generative tasks in the future.\n\n| Model | Method | S+C->R | C+R->S | R+S->C | Avg Gap |\n| --- | --- | --- | --- | --- | --- |\n| Llama-2-7b-chat | Vanilla Prefix | 62.32±2.15 | 66.30±2.30 | 73.15±2.53 | - |\n|  | Prefix with both | 63.70±1.96 | 68.47±0.97 | 75.32±1.09 | +1.91 |\n|  | Vanilla IC | 63.13±1.25 | 65.50±1.98 | 77.59±1.14 | - |\n|  | IC with both | 65.13±1.03 | 68.33±2.13 | 79.83±0.88 | +2.36 |\n| vicuna-7b-v1.5 | Vanilla Prefix | 67.72±1.79 | 81.09±2.17 | 88.97±2.64 | - |\n|  | Prefix with both | 68.75±1.04 | 83.93±1.79 | 89.76±2.60 | +1.55 |\n|  | Vanilla IC | 68.37±2.24 | 83.23±4.12 | 90.98±1.99 | - |\n|  | IC with both | 69.67±1.58 | 85.50±5.06 | 93.39±1.23 | +2.05 |\n| alpaca-7b-wdiff | Vanilla Prefix | 61.52±3.79 | 70.03±2.88 | 87.91±2.73 | - |\n|  | Prefix with both | 63.89±2.93 | 72.15±2.07 | 89.58±2.81 | +1.78 |\n|  | Vanilla IC | 60.81±1.14 | 69.11±2.46 | 89.66±2.37 | - |\n|  | IC with both | 63.16±1.56 | 70.57±1.95 | 91.19±2.00 | +1.99 |\n\nTable 1. Performance comparison of LLMs on multiple-choice tasks under MFDG settings. The last column shows the average gap between test performance on vanilla methods and our methods. Results are averages from 3 runs using different random seeds.\n\n**Q2: Baselines and Hyperparameter selection**\n\nWe thank the reviewer for the valuable comment. For starters, we would like to claim the objective of this work is not to design a brand-new framework, but to **improve domain generalization ability of current prompt optimization methods**, as stated in lines. 73-75:\n\n> With the principle of concentration §3, we propose two algorithms that could piggyback upon popular prompt optimization methods for both hard and soft prompts to improve the domain generalization ability of prompts.\n> \n\nThus, prompt tuning, as one of the most popular prompt optimization methods, serves as a good baseline to demonstrate that our proposed objective works well for improving the domain generalization ability within the framework of prompt tuning itself. In addition to prompt tuning, we also try to apply our proposed goal to several stronger baselines (such as Prefix Tuning and P-Tuningv2 for soft prompts, In-Context Demo and DP2O for hard prompts), and it consistently shows excellent performance in improving domain generalization ability of prompts on all experimental settings (as shown in Table 1 in our paper).\n\nAdditionally, we would like to address the reviewer’s concern in hyperparameter selection.  The suggested work [2] (100 prompt tokens and learning rate of 0.3) is based on full training data for the T5 model. In resource-limited scenarios, such settings may lead to severe overfitting issues.  \n\nWe also conduct experiments with the suggested hyperparameters by using T5-base model. The result is in Table 2. We find more prompt tokens and larger learning rate actually degrades performance of prompt tuning in the few-shot setting. We would like to clarify that our hyperparameter setting is similar with [3], where a learning rate of 1e-5 is used to train XLM-RoBERTa-base model in the few-shot setting with 4 soft prompt tokens.\n\n| Method |  | Sentiment |  |  | NLI |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | S+M→C | C+M→S | S+C→M | Q+R→W | W+R→Q | Q+W→R |\n| Vanilla PT | 55.70±0.98 | 52.37±2.05 | 52.67±1.55 | 41.57±0.93 | 51.83±0.79 | 51.70±2.25 |\n| PT with both | 57.17±0.99 | 54.80±1.77 | 53.35±1.73 | 43.72±1.33 | 53.46±1.53 | 53.79±1.47 |\n\nTable 2. Performance comparison of soft prompt learning on classification task accuracy for T5-base model. Results are averages from 3 runs using different random seeds.\n\n[1] Wang, Yuqing, and Yun Zhao. \"Tram: Benchmarking temporal reasoning for large language models.\" arXiv preprint arXiv:2310.00835 (2023).\n\n[2] Lester, Brian, Rami Al-Rfou, and Noah Constant. \"The power of scale for parameter-efficient prompt tuning.\" *arXiv preprint arXiv:2104.08691* (2021).\n\n[3] Zhao, Mengjie, and Hinrich Schütze. \"Discrete and soft prompting for multilingual models.\" arXiv preprint arXiv:2109.03630 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7MPmf9dy0G", "reviewer": "Reviewer_CLup", "summary": "This paper studies the problem of prompt optimziation for domain generalization.  Through a pilot experiment, they find that the domain generalization capability is tied to the attention concentration in later layers of the network.  Based on this finding, the authors design a set of regularizers to improve both soft and hard prompt optimziation procedures.  Empirically, the method is tested on sentiment and NLI tasks.  Results demonstrated a reduced generalization gap, and improved performance with the added losses over several prompt optimization methods.", "questions": "Results for the proposed approach primarily suggest that the phenomena happens in later parts of the network,  In practice, it is not clear without running the experiment, which layers will this be most prevalent and if it is task/prompt dependent.  For larger networks, would this happen earlier, or for shallow networks would this not happen at all?  Results here may be of interest in relation to the calibration of shallow models and lack thereof for deeper models.", "limitations": "- Proposed approach is only evaluated in fine-tuning settings.    This limits applicability to many settings where LLMs are evaluated such as in context learning, or zero-shot.  \n\n- The proposed approach is evaluated on the Roberta model, whereas there are a number of language models that could be investiinvestigated including the T5 encoder-decoder models, and GPT decoder only models.  Do we expect different behaviors based on the architectrues and attention?\n\n- The proposed approach is limited only to NLP applications, however could be applied for other variants such as vision transformers and robustness considerations.  Would we expect similar trends in other applications.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The exploration of prompt optimization is a highly important avenue for research as prompt optimization remains an efficient way to fine-tune language models. At the same time, it is improtant to consider the robustness of such approaches and how to improve this.\n\n- The paper is overall well-written and flows nicely.  The paper highlights the core properties for the findings and main results in the introduction of the paper, constructs an experiment to validate this in Section 3, subsequently intorduces their regularizer, and demonstrates results across two tasks.  \n\n- The proposed approach is simple to implement and can be applied irrespective of the architecture, as it is only an adjustment of the loss function thus having general applicability.\n\n- Authors have conducted thorough ablations studies in the proposed approach in the Appendices particularly around stability, visualizations, and initalizations.", "weaknesses": "- My biggest concerns with this paper are around the experimental results.  The performance improvements in experiments Tables 1 and 2 are small covering only 1-2% improvements over baselines.  The paper also does not make comparisons to existing approaches for domain adaptation of prompts such as \nhttps://arxiv.org/pdf/2207.07087\nhttps://arxiv.org/pdf/2210.02952\nhttps://arxiv.org/pdf/2305.13954\n\n- Further the paper only explores limited settings including a single architecture, task, and setting.  Exploring other modalities (such as vision transformers), additional tasks and settings, or other architectures would help solidify that the proposed approach is more general and would extend beyond the two tasks in this work.\n\n- Some minor typos such as hypnosis at line 284, and some hyperparameters seem to be missing from the main text for replication purposes including the value for lambda and how this is selected for the regularizers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive comments and feedback.  We appreciate that our work is considered “important”, “well-written”, and “thorough”. We hope our response can address your concerns.\n\n**Q1: Improvement and additional baselines**\n\n**Improvement:** We will try to address the reviewer's concerns about performance improvement in the following points.\n\n- The objective of this work is to enhance domain generalization in current prompt optimization methods, not to design a new framework. Our method acts as a plug-and-play module compatible with mainstream prompt optimization methods. As shown in Table 1, it improves performance in 3 soft and 3 hard prompt optimization methods.\n- Our method modifies the training objective or filter-match strategy within the original framework, so performance is limited by the original model structure or initial prompt set. For example, the quality of the candidate prompt set affects the performance of hard prompt optimization (as shown in Table 1 in our paper). Nonetheless, our method consistently enhances domain generalization with prompts from the same candidate set, demonstrating its effectiveness.\n\n**Additional Baselines:** Following the suggestion of the reviewer, we compare our method with the suggested domain adaptation methods — OPTIMA [1] and GPO [2] and two PEFT methods similar to [3] — IA3 and LoRA. The results are shown in Table 1. Compared with DP2O with both, the best performer in a series of prompt optimization methods piggybacked with our optimization strategy, we find that DP2O with both achieves accuracy improvements of 3.88%, 3.54%, 4.46%, 1.77%, 0.55%, and 3.62% compared to the second-best method in six experimental settings. \n\nAlthough our method still performs best, we would like to highlight that our work improves model performance in **domain generalization** setting where no information from target domain is available during training while data from target domain is accessible in **domain adaptation** methods like OPTIMA and GPO.  The Introduction section of our paper (**line 35-39**) discusses the shortcomings of domain adaptation methods, while the Preliminary section (**line 87-95**) introduces the basic definition of domain generalization problems. The Related Work section (**line 472-480**) contrasts the differences between these two types of problems.\n\n| Method |  | Sentiment |  |  | NLI |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | S+M→C | C+M→S | S+C→M | Q+R→W | W+R→Q | Q+W→R |\n| IA3 | 75.64±1.77 | 72.94±2.15 | 65.33±1.52 | 41.32±1.09 | 52.40±1.79 | 51.93±1.65 |\n| lora | 79.52±2.32 | 80.42±1.89 | 71.76±2.00 | 47.26±2.68 | 53.16±1.73 | 52.19±2.09 |\n| GPO | 82.57±1.73 | 89.32±2.09 | 83.39±0.88 | 54.65±0.71 | 54.77±2.20 | 55.90±2.65 |\n| OPTIMA | 85.75±2.77 | 85.01±4.4 | 80.61±4.46 | 53.33±3.26 | 54.19±2.70 | 57.65±5.17 |\n| DP2O with both | 89.63±0.52 | 92.87±0.33 | 87.85±0.47 | 56.42±0.36 | 55.32±0.33 | 61.27±0.81 |\n\nTable 1. Performance comparison of text classification task accuracy under MFDG settings. **Bold** indicates the best result for each column, and underline indicates the second-best result for each column. Results are averages from 3 runs using different random seeds.\n\n**Q2: Extension to other architectures, tasks and modalities**\n\nWe are glad to share that our method is able to successfully be applied on different architectures like **large decoder-only models** (Llama-2-7b-chat, Vicuna-7b-v1.5, and Alpaca-7b-wdiff) for additional tasks such as **question answering**. To be specific, we evaluate our method on ROC, SCT, and COPA datasets from the TRAM Benchmark [3] (referred as R, S, and C for simplicity), covering multiple-choice questions in reading comprehension and commonsense reasoning. The result is shown in Table 2.\n\nExperimental results show that our method significantly improves the performance of large decoder-only models on question-answering tasks across multiple domain generalization settings. For instance, for the Llama-7b model, our method improved the average accuracy of soft prompt generalization and hard prompt generalization comparisons by 1.91% and 2.36%, respectively. Similar improvements are observed in Vicuna-7b and Alpaca-7b models, ranging from 1.55% to 2.05% and 1.78% to 1.99% respectively. \n\nWe would like to clarify that our method is designed for language model, as we stated in the title of this paper. Incorporating other modalities into our method is beyond the scope of our research. We believe exploring the possibility to generalize our method to tasks in other modalities is interesting and we will do it in the future.\n\n| Model | Method | S+C->R | C+R->S | R+S->C |\n| --- | --- | --- | --- | --- |\n| Llama-2-7b-chat | Vanilla Prefix | 62.32±2.15 | 66.30±2.30 | 73.15±2.53 |\n|  | Prefix with both | 63.70±1.96 | 68.47±0.97 | 75.32±1.09 |\n|  | Vanilla IC | 63.13±1.25 | 65.50±1.98 | 77.59±1.14 |\n|  | IC with both | 65.13±1.03 | 68.33±2.13 | 79.83±0.88 |\n| vicuna-7b-v1.5 | Vanilla Prefix | 67.72±1.79 | 81.09±2.17 | 88.97±2.64 |\n|  | Prefix with both | 68.75±1.04 | 83.93±1.79 | 89.76±2.60 |\n|  | Vanilla IC | 68.37±2.24 | 83.23±4.12 | 90.98±1.99 |\n|  | IC with both | 69.67±1.58 | 85.50±5.06 | 93.39±1.23 |\n| alpaca-7b-wdiff | Vanilla Prefix | 61.52±3.79 | 70.03±2.88 | 87.91±2.73 |\n|  | Prefix with both | 63.89±2.93 | 72.15±2.07 | 89.58±2.81 |\n|  | Vanilla IC | 60.81±1.14 | 69.11±2.46 | 89.66±2.37 |\n|  | IC with both | 63.16±1.56 | 70.57±1.95 | 91.19±2.00 |\n\nTable 2. Performance comparison of decoder-only LLMs on multiple-choice task accuracy under MFDG settings. \n\n[1] Guo, et al. Improving the sample efficiency of prompt tuning with domain adaptation.\n\n[2] Li, et al. Robust prompt optimization for large language models against distribution shifts.\n\n[3] Tam, et al. Parameter-efficient prompt tuning makes generalized and calibrated neural text retrievers.\n\n[4] Wang, et al. Tram: Benchmarking temporal reasoning for large language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h9BgyfI35v", "reviewer": "Reviewer_a24D", "summary": "This paper focuses on improving the domain generalization of prompt tuning methods on LLMs. Specifically, this work claims that the concentration strength and concentration fluctuation of a candidate soft or hard prompt may indicate its generalization ability on new domains. By demonstrating the performance of various prompts with their concentration strength and fluctuation, the authors show that higher concentration strength and lower fluctuation may bring better prompt domain generalization. As a result, this paper proposes new objectives in soft and hard prompt tuning based on these observations. The experimental results show that the obtained prompts achieve better performance on new domains in NLI and sentiment classification tasks.", "questions": "Have you ever evaluated the effect of prompt attention concentration on any (recent) generative LLMs?", "limitations": "The authors have adequately addressed the limitations of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well organized, with very clear problem and methodology formulation, making the proposed approach easy to follow up.\n2. The observation on prompt attention concentration and its potential correlation to domain generalization are interesting and may encourage further research on this topic.\n3. The proposed objectives for soft and hard prompting methods are generalizable and are compatible with most recent prompting algorithms.", "weaknesses": "The main weakness of this work is the limited task types in the experiments. Only the sentiment classification and NLI tasks are considered in this work. It will be much better if more evidence or results on broader task types are obtained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive comments and feedback.  We appreciate that our work is considered “well organized”, “interesting”, and “generalizable”. We hope our response can address your concerns.\n\n**Q1: Extension to other tasks**\n\nWe would like to thank the reviewer for the valuable comment. We would like to thank the reviewer for drawing our attention to extending our method to broader applications. We are glad to share that our method is able to successfully be applied on mainstream Large Language Models and question-answering tasks. We validate the effectiveness of our method on Llama-2-7b-chat, Vicuna-7b-v1.5, and Alpaca-7b-wdiff models for improving domain generalization ability of prompts on question-answering tasks. We evaluate our method on ROC, SCT, and COPA datasets from the TRAM Benchmark [1] (referred as R, S, and C for simplicity), covering multiple-choice questions in reading comprehension and commonsense reasoning. The result is shown in Table 1.\n\n**Experimental results** show that our method significantly improves the performance of large models on question-answering tasks across multiple domain generalization settings. For instance, for the Llama-7b model, our method improved the average accuracy of soft prompt generalization and hard prompt generalization comparisons by 1.91% and 2.36%, respectively; similar improvements were observed for Vicuna-7b and Alpaca-7b models, ranging from 1.55% to 2.05% and 1.78% to 1.99% respectively. \n\n| Model | Method | S+C->R | C+R->S | R+S->C | Avg Gap |\n| --- | --- | --- | --- | --- | --- |\n| Llama-2-7b-chat | Vanilla Prefix | 62.32±2.15 | 66.30±2.30 | 73.15±2.53 | - |\n|  | Prefix with both | 63.70±1.96 | 68.47±0.97 | 75.32±1.09 | +1.91 |\n|  | Vanilla IC | 63.13±1.25 | 65.50±1.98 | 77.59±1.14 | - |\n|  | IC with both | 65.13±1.03 | 68.33±2.13 | 79.83±0.88 | +2.36 |\n| vicuna-7b-v1.5 | Vanilla Prefix | 67.72±1.79 | 81.09±2.17 | 88.97±2.64 | - |\n|  | Prefix with both | 68.75±1.04 | 83.93±1.79 | 89.76±2.60 | +1.55 |\n|  | Vanilla IC | 68.37±2.24 | 83.23±4.12 | 90.98±1.99 | - |\n|  | IC with both | 69.67±1.58 | 85.50±5.06 | 93.39±1.23 | +2.05 |\n| alpaca-7b-wdiff | Vanilla Prefix | 61.52±3.79 | 70.03±2.88 | 87.91±2.73 | - |\n|  | Prefix with both | 63.89±2.93 | 72.15±2.07 | 89.58±2.81 | +1.78 |\n|  | Vanilla IC | 60.81±1.14 | 69.11±2.46 | 89.66±2.37 | - |\n|  | IC with both | 63.16±1.56 | 70.57±1.95 | 91.19±2.00 | +1.99 |\n\nTable 1. Performance comparison of LLMs on multiple-choice tasks under MFDG settings. The last column shows the average gap between test performance on vanilla methods and our methods. Results are averages from 3 runs using different random seeds.\n\nAdditionally, we would also like to discuss ‘’**why our method works well for large generative language models?’’**.  In our **global response PDF**, we present the Concentration Strength Distribution of prompts using In-Context Demo across three 7B-sized language models (Llama, Vicuna, Alpaca) on three different tasks (SA, NLI, QA). We observed that all three LLMs exhibit stronger concentration strength in deeper layers compared to shallower layers when confronted with prompts for different tasks. Additionally, we found that this phenomenon occurs earlier in larger models (7B) compared to smaller models like Roberta-large. We speculate that this behavior is related to the alignment stage in pre-training of large models during Supervised Fine Tuning with a large number of prompts.\n\n[1] Wang, Yuqing, and Yun Zhao. \"Tram: Benchmarking temporal reasoning for large language models.\" arXiv preprint arXiv:2310.00835 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OfL1zAjN7C", "reviewer": "Reviewer_pV2m", "summary": "This paper studies prompt optimization methods for finetuning language models. While previous methods mainly concern with in domain performance, this paper brings awareness of the domain generalization issue presented in existing PO methods, under the setting where the target domain is unknown. Two empirical findings that link the domain generality of the prompt to the behavior in attention map are presented regarding this setting. Building on top of these findings, the paper proposes new objectives that account for domain generability for both soft and hard prompt optimization settings. Empirical study on BERT-size transformers and standard NLP datasets reveals that the proposed method achieves fairly consistent and substantial gain over vanilla.", "questions": "Both NLI and sentiment analysis are classification tasks, I wonder if the proposed method generalize to tasks beyond classification?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a pioneering study of the domain generalization issue of prompt optimization for PLM finetuning under the more practical setting where the target domain is unknown.\n2. The findings connecting the generalization ability of prompts to the attention patterns are interesting and of value to the community.\n3. Translating the findings into loss objectives are nontrivial technical challenges, and the paper presents neat solutions to these problems.\n4. Empirical gain are fairly substantial and consistent.", "weaknesses": "1. The main findings that motivate the proposed method are empirical. The paper can benefit from addressing the intuitions behind why such prompts are more generalizable.\n2. The models of choice are mainly small-scale transformers rather than LLMs. It is unclear whether the findings on the attention patterns generalize to bigger models. The paper could benefit from further verifying them on open-source LLMs.\n3. Presentation: the in-context citation formats are incorrect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive comments and feedback.  We appreciate that our work is considered “pioneering”, “interesting”, and “neat”. We hope our response can address your concerns.\n\n**Q1: More intuitive explanation**\n\nWe thank the reviewer for the valuable comment. Our intuition is that more concentration on prompts would lead to less attention on inputs. Thus, domain shifts (change of the domain that inputs are sampled from) cause less negative effect as the model pays more attention to knowledge-intensive prompts instead of constantly changing inputs.\n\nMoreover, it is also interesting to investigate the reason why prompts optimized by our method are more generalizable from the perspective of calibration. Calibration refers to the model's ability to provide class probabilities that correspond to its likelihood of being true. A well-calibrated model exhibits better domain generalization ability, as claimed in [1]:\n\n> In this paper we highlight a novel connection between multi-domain calibration and OOD generalization, arguing that such calibration can be viewed as an invariant representation.\n> \n\nWe use a common metric Expected Calibration Error (ECE) [2] for evaluating the calibration:\n\n$$\n\\mathrm{ECE}=\\sum_{m=1}^{M} \\frac{\\left | B_{m} \\right | }{n} \\left | \\frac{1}{B_{m}}\\sum_{i\\in B_{m}}\\left [ \\mathbb{I}(\\widehat{y_{i}}= y_{i})-\\widehat{p_{i}}  \\right ]    \\right | \n$$\n\nHere $n$ is the number of samples, $M$ is the number of bins. Each sample has its own label $y_{i}$, predicted label $\\widehat{y_i}$, and predicted probability $\\widehat{p_i}$. A lower ECE value indicates better calibration. As shown in Table 1, we demonstrate the change in prompt calibration before and after training using our method. We find that our method significantly improves model calibration compared with vanilla method. We infer that good calibration functions as an important factor to avoid significant inductive bias in domain generalization setting, which explains the performance improvement brought by our method.\n\n| Method |  | Sentiment |  |  | NLI |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | S+M→C | C+M→S | S+C→M | Q+R→W | W+R→Q | Q+W→R |\n| Vanilla Prefix | 0.145 | 0.160 | 0.104 | 0.297 | 0.193 | 0.159 |\n| Prefix w both | 0.128 | 0.095 | 0.072 | 0.206 | 0.162 | 0.127 |\n\nTable 1. Expected Calibration Error of Vanilla Prefix Tuning and Prefix Tuning with our method. smaller the better.\n\n**Q2: Extension to other tasks**\n\nWe would like to thank the reviewer for the valuable comment. We are glad to share that our method is able to successfully be applied on mainstream Large Language Models and question-answering tasks. We validate the effectiveness of our method on Llama-2-7b-chat, Vicuna-7b-v1.5, and Alpaca-7b-wdiff models for improving domain generalization ability of prompts on question-answering tasks. We evaluate our method on ROC, SCT, and COPA datasets from the TRAM Benchmark [3] (referred as R, S, and C for simplicity), covering multiple-choice questions in reading comprehension and commonsense reasoning. The result is shown in Table 2.\n\nExperimental results show that our method significantly improves the performance of large models on question-answering tasks across multiple domain generalization settings. For instance, for the Llama-7b model, our method improved the average accuracy of soft prompt generalization and hard prompt generalization comparisons by 1.91% and 2.36%, respectively; similar improvements were observed for Vicuna-7b and Alpaca-7b models, ranging from 1.55% to 2.05% and 1.78% to 1.99% respectively. \n\n| Model | Method | S+C->R | C+R->S | R+S->C | Avg Gap |\n| --- | --- | --- | --- | --- | --- |\n| Llama-2-7b-chat | Vanilla Prefix | 62.32±2.15 | 66.30±2.30 | 73.15±2.53 | - |\n|  | Prefix with both | 63.70±1.96 | 68.47±0.97 | 75.32±1.09 | +1.91 |\n|  | Vanilla IC | 63.13±1.25 | 65.50±1.98 | 77.59±1.14 | - |\n|  | IC with both | 65.13±1.03 | 68.33±2.13 | 79.83±0.88 | +2.36 |\n| vicuna-7b-v1.5 | Vanilla Prefix | 67.72±1.79 | 81.09±2.17 | 88.97±2.64 | - |\n|  | Prefix with both | 68.75±1.04 | 83.93±1.79 | 89.76±2.60 | +1.55 |\n|  | Vanilla IC | 68.37±2.24 | 83.23±4.12 | 90.98±1.99 | - |\n|  | IC with both | 69.67±1.58 | 85.50±5.06 | 93.39±1.23 | +2.05 |\n| alpaca-7b-wdiff | Vanilla Prefix | 61.52±3.79 | 70.03±2.88 | 87.91±2.73 | - |\n|  | Prefix with both | 63.89±2.93 | 72.15±2.07 | 89.58±2.81 | +1.78 |\n|  | Vanilla IC | 60.81±1.14 | 69.11±2.46 | 89.66±2.37 | - |\n|  | IC with both | 63.16±1.56 | 70.57±1.95 | 91.19±2.00 | +1.99 |\n\nTable 2. Performance comparison of LLMs on multiple-choice task accuracy under MFDG settings. \n\nIn our **global response PDF**, we illustrate the Concentration Strength Distribution of prompts in the In-Context Demo format for three 7B-sized language models (Llama, Vicuna, Alpaca) across three different tasks (SA, NLI, MCQA). A common observation is that the concentration strength is stronger in deep layers instead of shallow layers. To be more specific, compared to smaller models (Roberta-large), concentration phenomena occur earlier in larger models. Inductively, we get the conclusion that this phenomenon — higher concentration in deep layers, happens independent of model sizes, tasks or prompts. \n\n**Q3: In-context Citation Errors**\n\nWe would like to thank the reviewers for reading our work carefully. We sincerely apologize for the in-context citation errors in our paper. We will check and revise these errors in the next version.\n\n[1] Wald, Yoav, et al. \"On calibration and out-of-domain generalization.\" Advances in neural information processing systems 34 (2021): 2215-2227.\n\n[2] Naeini, Mahdi Pakdaman, Gregory Cooper, and Milos Hauskrecht. \"Obtaining well calibrated probabilities using bayesian binning.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 29. No. 1. 2015.\n\n[3] Wang, Yuqing, and Yun Zhao. \"Tram: Benchmarking temporal reasoning for large language models.\" arXiv preprint arXiv:2310.00835 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Global Response to All Reviewers**
---
We illustrate the Concentration Strength Distribution of prompts in the In-Context Demo format for three 7B-sized language models (Llama, Vicuna, Alpaca) across three different tasks (SA, NLI, QA). A common observation is that the concentration strength is stronger in deeper layers instead of shallower layers. To be more specific, compared to smaller models (Roberta-large), concentration phenomena occurs earlier in larger models and keeps till the deep layers.

Meta-review: This paper offers a detailed study of domain generalization of the learned prompts for PLMs. The paper receives a consensus of ratings after the discussion periods. All reviewers found that their concerns were sufficiently addressed during discussion and recommend acceptance. The AC shares the majority opinion. It's recommended that the authors integrate the additional experiments/discussions into the final paper, e.g. results of more tasks and architectures, and comparisons with other domain adaptation methods.
Decision: Accept (poster)