Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization
Authors: Kaihang Pan
Juncheng Li
Hongye SONG
Jun Lin
Xiaozhong Liu
Siliang Tang
Venue: EMNLP 2023 Findings (2023)
Keywords: prompt tuning
self-supervised meta-learning
meta-gradient regularization

Abstract: Prompt tuning is a parameter-efficient method, which learns soft prompts and conditions frozen language models to perform specific downstream tasks. Though effective, prompt tuning under few-shot settings on the one hand heavily relies on a good initialization of soft prompts. On the other hand, it can easily overfit to few-shot training samples, thereby undermining generalizability. Existing works leverage pre-training or supervised meta-learning to initialize soft prompts but they fail to data-efficiently generalize to unseen downstream tasks. To address the above problems, this paper proposes a novel Self-sUpervised meta-Prompt learning framework with MEta-gradient Regularization for few-shot generalization (SUPMER). SUPMER leverages self-supervised meta-learning with a diverse set of well-designed meta-tasks to learn a universal prompt initialization for efficient adaptation using only unlabeled data. Additionally, it jointly meta-learns a gradient regularization function to transform raw gradients into a domain-generalizable direction, thus alleviating the problem of overfitting. Extensive experiments show that SUPMER achieves better performance for different few-shot downstream tasks, and also exhibits a stronger domain generalization ability. The code for SUPMER will be available at https://github.com/beepkh/SUPMER.

Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization
Kaihang Pan 12+ Juncheng Li", Hongye Song', Jun Lin', Xiaozhong Liu?, Siliang Tang' ZZeejiagg University, DAMO Academy, Alibaba Group,
Worcester Polytechnic Institute (kaihangpan, junchengli, siliang]ezju. wdw. cn
(hongye.shy, linjun. www.111jjlabbbbbae com, x1 xll 4@wpi edu
Abstract
Prompt tuning is parameter-efficient method, which learns soft prompis and conditions frozen language models to perform specific downstream tasks. Though effective, prompt tuning under few-shot settings on one hand heavily relies a good initialization soft prompts. On the other hand, it can easily overfit few-shot training samples, thereby undermining generalizability. Exist- ing works leverage pre-training or supervised meta-learning to initialize soft prompts but they fail to data-efficiently generalize to un- seen downstream tasks. To address the above problems, this paper proposes novel Self- sUpervised meta- Peeta-prompt learning framework with MEta-gradien Regularization for few- shot generalization (SUPMER). SUPMER leverages self-supervised meta- meta-learning with diverse of well-designed meta-training tasks to learn universal prompt initialization for efficient adaptation using only unlabeled data. Additionally, jointly meta-learns gra- dient regularization function to transform raw gradients into domain-gente -ennraingeneaalizallle direc- tion, thus alleviating the problem of overfit- ting. Extensive experiments show that SUP- MER achieves better performance for differ- ent few-shot downstream tasks, and also ex- hibits a stronger domain generalization abil- ity. The code for SUPMER will be available https://gittee http://ggttuub.ccob/ee//////////
Aecurnty
Figure (a) Performance PT with different prompt initialization. (b) Performance after different training steps for vanilla PT and SUPMER.
would be sometimes unaffordable. More recently, prompt-based methods, which simply insert piece of carefully designed text to the input (e.g.. "It was (X).") and predict target words (e.g.. "great" or "terrible") at the mask position with frozen PLMs, have demonstrated remarkable effectiveness. But it has been observed that the performance of prompt- based methods is greatly affected by the design of prompts. light of this, prompt tuning (PT Lester et al., 2021), as parameter-efficces tuning method, proposed to only prepend some additional learn- able tokens called soft prompts to the input text, with all PLM parameters freezing.
Though prompt tuning is an efficient and effec- tive paradigm, Gu et al. (2022) shows it performs much worse than fine-tuning under few-shot set- tings. We argue that the performance is not satisfac- tory mainly due to two limitations: 1) The perfor- mance of PT is highly sensitive to the soft prompt initialization, especially for few-shot tasks. As shown in Figure (a), different soft prompt initial- ization leads significant performance variations. 2) Few-shot PT risks overfitting to some spurious correlations as soft prompts are tuned on limited training samples, thus undermining the generaliz- ability of PLMs. As shown in Figure (b), the performance of few-shot vanilla PT degrades sig- nificantly in the final training steps.
1 Introduction
Recent NLP accomplishments witnessed the rapid development of pre-trained language models (PLMs) (e.g., BERT Devlin et al., 2019; T5 Raf- fel et al., 2020; GPT3 Brown et al., 2020). Fine- tuming, which tunes the entire PLM parameters, has achieved outstanding performances in various NLP tasks. However, as the pre-trained model scale increases, tuning the entire set of parameters
Work done when interning at Alibaba DAMO Academy. Conresponding Author.
Recent research mainly focused on the first limi-

---

tation, leveraging pre-training or supervised meta- learning for soft prompt initialization. A pre- trained prompt tuning method (PPT) (Gu et al., 2022) is proposed from the beginning, which uti- lizes self-supervised tasks to pre-train soft prompts and then applies them in the few-shot scenario. However, without explicitly optimizing the fast adaptation abiliiy of the model, PPT suffers from train-test mismatch between the pre-training data and the downstream data. So limits generaliza- tion to unseen few-shot tasks, especially when there is a significant disparity in task domains or for- mats. MctaPrompting (Hou et al., 2022), as an- other effort, seeks assistance from model-agnostic meta-learning (MAML. Finn et al., 2017) for fast adaptation in few-shot settings. However, in each task, MetaPrompting requires plenty of labeled data within certain classes to perform supervised meta-learning for prompt initialization, which is often inaccessible in practical few-shot scenarios. And the learned initialization can only generalize to the remaining classes of the same task a few- shot manner, exhibiting weak task transferability. Furthermore, all these existing works ignore the second limitation, i.e., the propensity for few-shot prompt tuning to lead to overfitting.
learning. As we simulate distribution shift through task augmentation, the meta-gradient regularization parameters are jointly optimized to align gradient directions across different distributions within our proposed meta-prompt learning paradigm. Con- sequently, in downstream tasks, these optimized parameters can be directly utilized to transform raw gradients over few-shot samples into domain- generalizable direction, preventing prompt tuning overfitting some domain domain-speciftt specific correlations.
Overall, our contributions are mainly three-fold:
(1) We propose novel self-superriied meta- prompt learning framework to better initialize soft prompts, where only unlabeled pre-training data are used to construct different meta-training tasks with curriculum- task augmentation for fur- ther task enrichment.
(2) We incorporate a novel meta- gett-grradent regu- larization function into our meta-prompp learning framework, which meta-learns to transform the raw gradient during few-shot learning into domain- generalizable direction, thus preventing prompt tun- ing overfitting to domain-specific correlations. (3) Comprehensive experiments on few-shot learning and domain generalization validate the superiority of our method, which even outperforms full-model tuning in few-shot learning. It also ex- hibits stronger domain generalization ability.
To address the shortcomings of existing works, we propose SUPMER, a Seee-sUee meta- Prompt learning framework with MEta-gradient Regularization. It leverages self-supervised meta- learning to universally learn an efficient soft prompt initialization, also with a meta-gradient regulariza- tion function mitigate overfitting. This compre- hensive process only requires a one-time execu- tion and enables seamless adaptation different downstream few-shot tasks, while also facilitating faster convergence for downstream prompt tuning.
2 Related Work
Soft Prompt Tuning. Soft prompt tuning is of the most parameter-efficient tuning methods widely used in NLP (Liu al., 2023) and vision- language tasks (Zhou al., 2022; Li al., 2023a), which only tunes a small number of (extra) pa- rameters to attain strong performance. Specifi- cally, it freezes the PLM parameters and prepends some trainable continuous embeddings (i.e., soft prompts) to the input sequence (Lester et al., 2021) or every layer of the pre-trained model (Li and Liang, 2021; Liu et al., 2022).
Specifically, to address the first limitation, we de- sign novel self-supervised meta- meta-learning method for prompt initialization, which automatically gen- erates a diverse set of meta-training tasks from large-scale unlabeled corpora and explicitly learns to fast adapt across these tasks. To ensure task di- versity, we initially design a collection of anchor self-supervised meta-training tasks with different formats. And then a curriculum based task aug- mentation method is further proposed to enrich the task distribution dynamically in terms of the cur- rent model capability.
To efficiently train task-adaptive soft prompts in few-shot scenarios, some studies ( (Vu et al., 2022: Asai et al., 2022; Sun et al., 2022) employ task adaptation techniques, obtaining source prompts from source tasks in supervised way and interpo- lating them into the target prompts. Other works focus on training improved prompt initializations. PPT (Gu et al., 2022) pre-trains the soft prompts with some supervupe tasks on unlabeled cor- pora, but it doesn't 't explicitly optimize the fast adap-
For the second issue, we integrate a meta- gradient regularization function into meta-prompt

---

otcchr
tork diviriaaiion
Sostenne classfirutin
6g
Updrrn
Maltti Maltii-choice claznifccatin
reneee-entence clareifccttinn
unicbeled
cata
pawrr
Compute
papport
Regulartzaiin
ponry
Compule Gracient
98
Gradient
Teak interpolaion
Mixing Rario
Gradient Simibrity
seff-xxprviser kook dloss
Meca Meca-training Data
Meta- Poont--rrmpttioo Learning
Gradient Regularization.
Mete-trainiig
fook
ditrilatiion
tearnablee
Sampling
borwn
Diamoutioon Stile
Sampling
primano
Regulariation
Compule Gradient
evaluation
Curriculum-based Augmontation
Adaptive
Few-shot Ceneralization
Figure 2: The framework of SUPMER. We employ task interpolation enrich the distribution self- supervised meta-training tasks. Concurrently, we integrate meta-gradient regularization function into meta-prompi learning. Furthermore, during meta meta-prompt learning we also dynamically adapt the mixing ratio of task interpolation. upgrading the vanilla task augmentation into curriculum-bart one.
tation ability of the model. MetaPrompting(Hou et al., 2022) utilizes supervised meta-learning for soft prompt initialization, splitting each dataset into two sets with disjoint data classes. One split is used to initialize soft prompts while the other serves as the downstream task. In comparison, SUPMER dif- fers from MetaPrompting in the following ways: 1) for each downstream task MetaPrompting focuses on fixed supervised dataset to reinitialize soft prompts, whereas SUPMER can universally gener- alize different unseen tasks with large-scale un- labeled corpora for initialization; 2) MetaPrompt- ing doesn't t freeze PLM parameters, while SUP- MER only tunes the soft prompts as the general soft prompt tuning methods do.
ents towards a domain Domain-generalizabbl direction.
Method
In this section, we describe the whole framework of SUPMER (shown in Figure 2). With pre-defined preliminaries, we first introduce the way to con- struct anchor self- self-supervised meta tasks and the foundation of task augmentation to densify task distributions. Then elaborate on the SUPMER model, including the meta-gradient regularization function. Finally, we upgrade the original task aug- mentation method into a curriculum-based one. Be- sides, we formalize all tasks in a text-to-text format following the T5 fashion (Raffel et al., 2020).
3.1 Preliminaries
Meta-Learning. Meta-learning, also known as learning to learn, optimizes the ability to learn new tasks quickly and efficiently, utilizing expe- rience from previously seen tasks. It can be classi- fied into three types: metric-based methods (Koch al., 2015; Vinyals et al., 2016; Snell et al., 2017), model-based methods (Graves et al., 2014; Mishra et al., 2018; Qiao et al., 2018), and gradient- based methods (Hochreiter et al., 2001; Ravi and Larochelle, 2017; Nichol et al., 2018; Li et al., 2020). In this work, we focus on gradient-base based meta-learning algorithm (i.e., MAML Finn al., 2017). Compared to typical meta-learning methods that rely on numan-annotated meta-training tasks, we automatically generate abundant tasks in self- supervised way, also integrating a meta- gredien regularization function into MAML to steer gradi-
Prompt Tuning. In prompt tuning (Lester et al., 2021), given training sample (æt, y/i from task D,, we apply prompt template P converting x; into new sequence P(xx) and then concatenate set of soft prompts 0 to the beginning of P(a:). And verbalizer V plays role in mapping y to corresponding label tokens V(H)) in the vocabulary of PLMs. So the objective of prompt tuning can be formulated as follows:
arg min Lr, (0)
arg max
Σ
loge( (X) P(x)))((
where denotes the soft prompt embedding (the only tunable parameters in prompt tuning). (X) let PLMs predict target tokens at the masked positions and [ is the concatenation operation.

---

Model-Agnostic Meta-Learning. Assuming ac- cess to a task distribution p(7), the goal of meta- learning to utilize tasks Tj ~ p(T), referred to as meta-training tasks or meta tasks, to train learn- ing procedure that generalizes to unseen tasks from the distribution. Model- Model-Agnostic Agnostic Meta-Learning (MAML) (Finn et al., 2017) is gradient-base based bi-level optimization meta-lcarning method, which consists of inner loop task task-specific learning and outer loop fast adaptation across tasks.
by K-means. On this basis, for each task format, we distribute meta-training data into different tasks to construct anchor meta-training tasks with well- balanced task distributions. We group samples with similar embeddings into the same task based on the results of K-means. And we give a more detailed description of anchor meta-training task construc- tion in Appendix A.2.
3.3 Vanilla Task Augmentation
Specifically, a task is composed of the support set D and the query set D4. In the inner loop of MAML, a model learns to adapt to a new task T using its support set in the following way:
With a set of anchor meta-training tasks, in this section we first introduce the vanilla task augmen- tation densify the task distribution. Extending the idea of mixup (Zhang al., 2018), we augment the task set through task interpolation, which lin- early combines features and corresponding labels of samples from the query set in different tasks. In $3.5 we further upgrade the vanilla task augmen- tation method into curriculum crrrsiuluu---eedd one, which dynamically controls the task interpolation in terms of the current model capability.
0 = VeCre, (0)
(2)
where an is the inner loop learning rate and 0 the model's parameters. And the optimized parameters Of is then evaluated on the query set of task with the loss function Lpz. In the outer loop, this loss across meta-training tasks is treated as the final training loss update A:
Specifically, for task composed of the support and the query set, we denote the hidden repre- sentations of the query set samples in task & as H Given an anchor task T₁, first we randomly se- lect another task Tj. While retaining the support set of Tj we reconstruct its query set by interpolating on the hidden representations (Hą H and corre- sponding labels (Y, Y19) from the query sets in т and Tj, which can be accomplished using mixup:
- 8 Vi
Loz, (8))
(3)
TemPp(T)
where 31 is the outer loop learning rate.
3.2 Constructing Anchor Meta Tasks
Supervised datasets with a large amount of la- beled data are often unavailable in many NLP tasks. While unlabeled data is more easily accessible and generally covers broader semantic concepts. So we utilize the unlabeled data from a large corpus to create anchor self-super meta-training tasks.
H = (1- -) AHY. Y, (1 - A)YY - AY
(4)
where the mixing ratio X E [o, 1] is drawn from Beta distribution Beta(a, a),, and a is a hyper- parameter. The process of task augmentation not only enriches the task distribution, but also sim- ulates the distribution shift between the support set and the query set within one task, as we only leverage interpolation between the query sets of dif- ferent anchor meta-training tasks. And in $3.4 we will show the effect of this distribution deviation.
The unlabeled data are first grouped into differ- ent clusters. We utilize PLMs to derive semanti- cally meaningful embeddings for sentences in the corpus, and then apply unsupervised K-means to cluster these unlabeled sentences. Based on the re- sults of K-means, we design three different formats of sulf-supervieed meta-training tasks: sentence- pair classification, multi-choice classification, and single-sentente classification.
3.4 Meta-Prompt Learning with Meta-Gradient Regularization
Specifically, sentence-pair classification in- volves predicting whether two sentences are adja- cent in the same document or from the same cluster after K-means clustering. Multi-choice classifi- cation identifies the correct sentence among sev- eral candidates, which is either adjacent query sentence or from its same cluster. And Single- sentence classification aims to associate each sen- tence with its correct cluster label, as determined
In this section we introduce the algorithm of our meta-promp learning framework, which is a bi-level meta-lcarning paradigm learning a task- universal soft prompt initialization 0 for efficient adaptation. And it jointly meta-learss meta- gradient regularization function vig that transforms raw gradients into a domain generalizable direction to prevent prompt tuning from overfitting.

---

Specifically, considering that the inner loop up- date of MAML (i.e., Eq. (2)) over limited samples might overfit to some domain-specifie correlations, we propose to learn gradient regularization func- tion (*), making direct transformation to the raw gradients obtained from the support set D The function first performs affine transformation l(() (e.g.. rotation) to modulate the raw gradients 9. and then an update gate vector is employed to combine g and h(g) into the final gradients:
zo to reformulate Eq. (8) as:
V. Lov(x) = Cov (zo 4. arg arg min Lov (1t
(9)
arg min VoCrv (VoCo.
Based the aforementioned discussion, reach the following conclusions: (1) The update of O minimizes the expected loss on the query set. (2) The optimization of both and & maximizes the inner product between the regulated gradients from the support set and the gradients from the query set. The inner product of two vectors is larger if they are in a similar direction. Recalling that we simulate the distribution shift between the support set and the query set, the optimization of A and á tries align the gradient directions across different distributions. To improve the alignment between the Nomain-specifii gradients, the gradi- ent regularization parameters are optimized to retain some domain-invariant information of meta- training data and then can be utilized to regulate raw gradients obtained from few-shot samples into domain-generaan generalizable direction in downstream prompt tuming, thus avoiding overfitting to some spurious correlations.
(و(ي) k(y) + (1 (1==)) y
(5)
Obviously, the value of z can be used control how much the transformed gradients h(g) con- tribute to the output of (g) We hope to deter- mine this weight based on the input samples them- selves, setting 2 = aW H + b). where H is the hidden representations of input samples. Formally, now we transform Eq. (2) into:
# = -= an- (VoCoe (0))
(6)
After adapting the soft prompt embeddings to the support set D in the outer loop we optimize the prompt initialization 9 based on these adapted embeddings 0' via Eq. (3). Besides, meta-gradient regularization parameters @ are also optimized us- ing the same loss to learn better gradient transfor- mation, with B2 as the learning rate:
3.5 Curriculun-based Task Augmentation
In $3.4 we show that SUPMER can help align the optimization direction across two distributions with deviation, which is simulated by performing task augmentation exclusively on the support sets. From Eq. (4) it is evident that the mixing ratio À of mixup controls the extent of the distribution deviation, with a larger resulting in a more noticeable devia- tion. However, in the previously discussed method, is sampled from a fixed Beta distribution. In this section, we propose more flexible sampling ap- proach, which upgrades the original task augmenta- tion method into muriculum-basee one, gradually increasing the task difficulty and achieving more reasonable distribution shift.
Σ Coe, (0)) T(pp(T)
Overall, the total meta- pottappppppt learning obejec- tive can be formulated as follows:
arg min Bich
Σ
Cug (0 an a00-a (Vooo (0))
(8)
((P)
Downstream Prompt Tuning. The above meta- prompt learning framework only requires a one- time execution. The optimized prompt initialization 6* and meta- graddeg regularization parameters o* are then universal for different downstream tasks. During downstream prompt tuning, we fix o² and further adapt 0* to testing tasks as Eq. (6).
The curriculum-bast based task augmentation dynam- ically adjusts the parameters of the Beta distribu- tion, from which we sample the mixing ratio X. Specifically, batch of meta tasks is sampled in each training epoch. For each task, we can obtain gradients on the support set of and gradients on the query set o/ along with their cosine similarity. We leverage the average cosine similarity 8k-1 of all tasks in a batch during the last epoch to derive the
Analysis of SUPMER. Here give some anal- ysis of how SUPMER could enhance generalizabil- ity, with more complete proof in Appendix A. Given that x a th (V (f)) and 2g = 6, focusing on a single meta- teta-trainnig task, we can ap- ply a first-order Taylor expansion around the point

---

Model: TS-base (220M)
Methods
SST-2
SST-
MR
CR
SUBJ
TREC
RTE
QNLI
WIC
MRPC
QQP
AVG
Prefix Tuning
78.2,7
38.02u
74.0g.2
84.5a,
66.2g9
70.129
70.42a
64.42.
56.74.1
63:41
68.11a
64.9La
64.826
P-Tuning-22
53.10s
41.71a
82.Jos
88.76.5
74.824
12.
67.564
53.91s
61.4.4
54.522
70.7
68.0
68.71V
63.6
41.2g
81.7ea
86.3pg
50.0
79.8
7
56.9a1
62.3a.n
64.61.
70.2g7
69.5
70.01.4
PT
T1.9..
37.9ka
73.315
$1.41.
61.56a
65.31a
58.96.
53.2ax
55.2
53.12
66.6s,3
63.02m
62.0c.
PPT
21.22.0
40.26.4
83.Gr.a
66.8%.7
73-4g4
60.7g
66.41,
60.43.
63.61.
68.0gs
63.1o7
66-6ga
Unified-PP PPT
76.8r.
4..7
79.0s.s
87.70a
64.2
68.4p.
63.9ca
54.4..
56.7a.1
64.5
67.8
67.62M
65.5gx
MetaPT
5..7
45.3
82.5
88.50.3
73.2
78.7pa
65.4e.
56.19.
58.2g.
54.11.
69.6
68.8gg
68.81.
SUPMER
87.30.5
16.7a.s
84.0p.a
89.3g.3
79.622
80.
72:41.4
57.33.o
61.7a.
5/..1
71.3a.s
70.51.a
71.30.9
Model:
Flau-15-XL
(38)
Methods
SST-2
SST-
MR
CR
SUBJ
TREC
CB
RTE
QNLI
WIC
MRPC
QQP
AVG
Zoro-shot Inforence
80.1
62.3
83.3
80.6
57.4
87.2
76.8
75.8
86.0
50.5
77.2
77.5
74.4
Fow-shot Inferance
53.3
88.5
87.8
58.6
64.3
Prefix Tuning
88.6g.1
45.42.
88.7pa
80.1
84.6g9
90.6g4
65.40.4
73.4g6
72.3
66.61s
T3.9a
75.0g
76-22.
2-Tuning-22
01.2
5
90.0ga
89.8;.
87.7
92.3o.
60.12a
62.3c.
76.81a
63.1gg
17.2
77.3k
77.5g0
,2.9,4
53.6g
89.61,
91.3o.8
88.7pa
93.62.0
77.8
76.0g,
84.4
63.01s
77.71.
82.9g2
81.01.
PT
88.2
45.7a.4
85.02.6
88.65.1
81.00.
85.
65.4p.
61.66.
69.3s.
54.811
71.
7
72.36,
PPT
81.02.6
49.12.
88.81,
88.0g.0
87.61
90.8
67.148
67.1g2
81.5g.
66.11a
75.82.
71.6g.1
70.2g8
Uni
89.9..3
47.42x
89.Taa
89.0..
86.423
88.43.2
68.3a.
68.2
74.9
39.917
T1.02.
76.32n
75.729
MetaPT
93-7ps
52.61,
80.767
80.70.
88.1g4
91.81,
72.322
74.1
77.525
68.51.
76.61.
81.424
78.91,
SUPMER
95.
65.3p.7
91.44..
90.76.7
90..3p.
93.00,s
87.6
81.44.0
88.3p.o
65.0).7
78.
88..1..4
83..5..
Table 1: Results of few-shot learning. For each dataset report the average accuracy and standard deviation over five random seeds (zero-shot & few-shot inference produce nearly consistent results each time as they do not require parameter tuning). Bold fonts indicate the best results. We can see SUPMER achieves better performance.
mixing ratio A for the current epoch
classification dataset TREC (Voorhees and Tice, 2000); 4) the natural language inference datasets CB (De Marneffe et al., 2019) and RTE (Wang et al., 2019); 5) the question answering dataset QNLI (Rajpurkar et al., 2016); 6) the word sense disambiguation dataset WIC (Pilehvar and Camacho-Colladoo 2019); 7) the paraphrase de- tection datasets MRPC (Dolan and Brockett, 2005) and QQP. Following Karimi Mahabadi et al. (2022), for each dataset we sample 16 instances per label from the original training set to form training and validation sets for few-shot learning.
b (eta(a; ba)))
by m
(10)
19]
08 9
where 8k-1
1 |9
where m is the curve parameter. In this way, when our model is not capable of aligning the optimiza- tion directions across different distributions at the beginning, a smaller À is preferable to create a smaller distribution deviation. Then À tends to gradually increase as the model's capability im- proves, resulting in larger distribution deviation and corresponding increase in task difficulty.
Domain Generalization. Then we design a more challenging problem about zero-shot domain gen- cralization in the sentiment analysis task. Our ex- periments include 6 domains across 3 datasets: 1) the Amazon review dataset (Blitzer et al., 2007) containing reviews about Books (B), DVDs (D), Electronics (E) and Kitchen appliances (K); 2) the airline review dataset (A) (Nguyen, 2015; Ziser and Reichart, 2018); 3) the restaurant (R) domain obtained from the Yelp dataset (Zhang et al., 2015).
We present the pseudo-codes of SUPMER in Appendix A.4.
4 Experiments
4.1 Experimental Setup
We evaluate our approach in two problem settings: 1) Few-shot learning with different NLP down- stream tasks; 2) domain generalization.
We choose A as the source domain and the other five (B, D, E, K, R) constitute the target domains. On this basis, we sample 16 instances per label from the training set of the source domain to tune soft prompts. And then we directly use the soft prompts learned from the source domain to evaluate performance on the test set of each domain.
Few-shot Learning. We consider 6 downstream tasks with 12 datasets: 1) the sentiment analy- sis datasets SST-2, SST-5 (Socher et al., 2013), MR (Pang and Lee, 2005) and CR (Hu and Liu, 2004); 2) the subjectivity classification dataset SUBJ (Pang and Lee, 2004); 3) the question

---

Method
Siurc
Tagas
AYG
79
A
E
II
Posis Tuxny
82.0
62.23
Aucurscy (86) 8
PT
54.4
68.9%
63.0vv
85.061
10-8aa
7600g
United
Milb
Merift
60.1a
Viills FPT
Va
SI
K5.3ca
81.3aa
90.4aa
227
VethiT
23
Method
SoureE
Thead
AWGG
A
E
R
Trairing
Trairing
don
77.8
infeence
Piefs Tuxing
S
65.b.
83.4.1
ST.
45.45
89.5..
45.
Figure 3: The performance after different training steps on CB and MRPC.
Tuxing
86.5
00:25
85.0
87.5
$6.30
57.
PT
70.0u>
SL.ba
PPT
83.2
88.7
10 6a.
4965
SAIy
United
07.5ca
45.0
85.623
81.32x
67.1aa
ST PMER
S7.kg
06.1aa
05.Aca
45-
60-
Accunacy (R2
Table Results of domain generalization. For MetaPT we calculate the average performance only across do- main A, B, D, E, K (without R).
8
4.2 Experimental Details
6
Number Samples Label
Sumples Label
Baselines. Our experiments are built on a smaller- scale model, T5-base (Raffel al., 2020), and then on larger-scale instruction-tuned model, Flan-T5-XL (Chung et al., 2022). For both two backbone models, we use the following baselines: (1) prompt tuning methods with the same num- ber of tunable parameters as SUPMER: vanilla prompt tuning (PT Lester et al., 2021), PPT (Gu et al., 2022), Unified-PPT (Gu et al., 2022), and MetaPT (Huang al., 2022). (2) methods with more tunable parameters: Prefix-Tuning (Li and Liang, 2021), P-tuning-v2 (Liu et al., 2022), full- model tuning (FT). Furthermore, Given that FLAN- T5-XL was also designed with few-shot inference in mind, we additionally compare with two base- line methods on FLAN-T5-XLL i.e., zero-shot in- ference and few-shot inference, which directly employ Flan-T5-XL for downstream evaluation. We list the details of baselines in Appendix B.
Figure 4: The performance on SST-5 and SUBJ when different numbers of training samples are available.
ularization parameters and the soft prompts are the only tunable parameters. We give more details of training hyper-parameters in Appendix C.
4.3 Main Result
Table and Table 2 show the main results of few- shot learning and domain generalization. From the results, we have the following observations.
First, in few-shot learning, SUPMER achieves better performance than all baselines on 10 of 12 datasets, whether using T5-base or Flan-T5-XL the backbone. And the average accuracy of SUP- MER over all datasets reaches 71.3% on TS-base, significantly outperforming other baselines (e.g., improving the performance by +1.3 points com- pared to FT). Notably, when utilizing the larger Flan-T5-XL as the backbone, SUPMER demon- strates more substantial performance gains (e.g., improving the average performance by +2.5 points compared to FT), which indicates that our approach unlocks greater capabilities for stronger models that have undergone instruction-tuning with a higher number of parameters.
Implementation Details. We solve all down- stream tasks in a text-to-text format and run each experiment with different random seeds. For all prompt tuning methods, we follow Lester et al. (2021) to design soft prompts composed of 100 soft tokens, with tunable parameters far less than full-model tuning. For our SUPMER, following PPT (Gu et al., 2022) sample 10GB data from OpenWebTTe (Gokaslan al., 2019), a large- scale unlabeled corpus, to construct self-supervised meta-training tasks. The meta-training stage only requires a one-time execution. In downstream prompt-tuning, we freeze the meta-gradient reg-
Specifically, SUPMER consistently outperforms all other prompt tuning methods with the same number of tunable parameters across all datasets. This indicates that our method offers soft prompts with better few-shot generalization ability. And it is noteworthy to highlight that SUPMER utilizes ex-

---

Methods
SST-2
SST.5
MR
SUBJ
TREC
RTB:
QNI.I
MRPC
QQP
SUPMER (only labeled)
87.5
47.0
83.8
80.9
75.4
79.6
67.9
56.6
59.0
54.6
69.2
69.5
70.0
SUPMER (only unlubeled)
87.3
46.7
84.0
80.3
79.6
57.3
71.3
70.5
3 PPT (labeled unlabeled)
84.7
45.0
82.4
67.2
77.4
64.3
55.3
61.6
53.9
68.9
67.7
68.0
MetaPT (labeled + unlabeled)
86.1
46.3
83.7
73.8
80.1
57.4
60.0
54.3
69.9
69.9
5 SUPMER (labeled+ unlabeled)
89.1
48.2
85.7
79.3
58.8
63.7
55.3
70.5
71.5
72.5
Table 3:
Results of few-shot learning on T5-base, considering different data and methods for prompt initialization.
actly the same unlabelled data as PPT and Unified- PPT for soft prompt initialization. Yet it consider- ably outperforms these two baselines, demonstrat- ing that the performance improvement is primarily attributable our methodology rather than the meta- meta-training data itself. Additionally, SUPMER outperforms baseline methods with more tunable parameters (e.g., full-model tuning) on the majority of datasets, achieving superior performance with fewer parameters.
Methods
Few shat Learning
DG
only sp only me only ss
66.7
85.7
66
67
86.7
wro ta curriculum
68.6
85.3
87.0
weo mgr
60.4
86.1
SUPMER
71.3
88.0
Table 4: Results of ablation study to illustrate the effect of individual components. We report the average accu- racy over all 12 datasets in few-shot learning and all 6 domains in domain generalization (DG).
Second, SUPMER is superior to all baselines in almost all commann-ennralliaaii
setups. For example, compared to MctaPT which meta-trains soft prompts with a supervised sentiment analysis dataset, SUPMER exhibits average gains of 1.1 % on T5-base and 1.4% on Flan-T5-XL., So it can be inferred that SUPMER shows stronger robustness to domain shifts, exhibiting better generalization to unseen tasks or domains.
only accelerates the convergence to the optimal per- formance realizing fast adaptation, but also con- sistently maintains optimal performance across prolonged training periods.
Effect of Sample Size. We also discuss how the performance of SUPMER and other baselines varies when the number of training samples in- creases SST-5 and SUBJ. As shown in Figure 4, with T5-base as the underlying PLM, when the number of training samples per label grows from 4 to 64, SUPMER consistently better than other prompt tuning methods. And the performance gap between these methods is gradually reduced as the number of training data increases.
Third, for both few-shot learning and domain generalization on Flan-T5-XL- SUPMER demon- strates superior performance across almost all datasets and domains in contrast to few-shot in- ference. It provides further evidence that for LMs such T5- with inherent few-shot infer- ence capabilities, our approach can significantly en- hance their abilities in a parameter-officient tuning strategy, without providing any in-context exam- ples during inference.
Self-Supervised v.s. Supervised. To illustrate that self-supervised meta-learning can better gen- eralize to unseen tasks compared to supervised meta-learning, we also collect set of labeled datasets (ensuring no overlap with downstream test- ing datasets) to formulate meta-training tasks for soft prompt initialization and conduct the experi- ments of few-shot learning on base The results are displayed in Table 3 (rows and 2). As our col- lected labeled data contains lots of sentiment anal- ysis datasets (e.g., Yelp5), SUPMER (only labeled) and SUPMER (only unlabeled) reveal proximity in their performance sentiment analysis tasks (i.e., SST-2, SST-5, MR, CR). But in other tasks, using unlabeled data consistently achieves better results than utilizing only labeled data, also with a higher average accuracy over all datasets, which validates
Fourth, SUPMER also results in lower variances on most datasets. Few-shot learning is often notori- ous for its instability. And in our method we keep few-shot prompt tuning more stable.
4.4 Ablation Study
Analysis of Generalization. Figure 3 shows the performance trend for each method after different training steps on datasets CB and MRPC with T5- base model. It illustrates that few-shot prompt tun- ing converges slowly with its performance typically showing an overall decline during the final training steps because they may easily result in overfitting. In comparison, SUPMER achieves faster, stronger, and more enduring shw-shot generalization. not

---

the superiority of self-supervised meta-learning.
avoiding overfitting to some domain-specific corre- lations, thus achieving better performance.
Effect of integrating Labeled Data. To further explore the impact of integrating labeled data and substantiate the efficacy of SUPMER following this integration, we amalgamate the original un- labeled meta-training data with our collected la- beled data mentioned above, with a mixing ratio of labeled to unlabeled as 1:2. The amalgamated data is employed for constructing meta-training tasks to meta-train SUPMER. Moreover, follow- ing PPT (Gu et al., 2022) and MetaPT (Huang al., 2022), We also leverage pre-training and vanilla MAML initialize soft prompts using the same amalgamated data. The experimental results of few-shot learning on T5-base are shown in Ta- ble 3 (rows 3-5). First, we can see that SUPMER (labeled+unlaaeleed) outperforms SUPMER (unla- beled) and SUPMER (labeled) it allows us to harness the high-quality advantages of labeled data while also exploiting the broader semantic concepts encapsulated by unlabeled data. Second, After the integration of labeled data, SUPMER still consis- tently demonstrates significantly superior perfor- mance compared to baseline methods employing the same data for prompt initialization, which fur- ther underscores the effectiveness of SUPMER.
Conclusion
In this paper, we present SUPMER, self- supervised meta-prompa learning framework with meta veda-gradient regularization. With diverse set of well-designed self-supervised meta-training tasks, SUPMER jointly meta-learns universal prompt initialization and an effective gradient regulariza- tion function for efficient few-shot generalization. Extensive experiments on few-shot learning and domain generalization show that SUPMER outper- forms other prompt methods and full-model tuning, achieving ofaeeoo-the--ae performance.
Limitations
Although SUPMER performs superbly a variety of problem scenarios, there still exist some limita- tions in our work: 1) We did not conduct any data filtering or cleaning operations to the meta-training data, which could potentially result in the inclusion of some biased content. 2) Our experiments are solely conducted on English tasks, and also do not involve some kinds of NLP tasks (e.g., language generation Li et al., 2022c) or vision-language tasks (Zhang et al., 2022b; Li et al., 2022b; Zhang et al., 2019; Li et al., 2021).
Effect of Individual Components. We train the following ablation models. 1) only sp mc / ss: we retain sentence-pair classification/ / multi-choice classification single-sentence classification as the only anchor meta-training task format. 2) w/o ta: we entirely remove the task augmentation method. 3) w/o curriculum: we only retain the vanilla task augmentation without the curriculum-basee based idea. 4) w/o mgr: we remove the meta-gradient regu- larization function. All experiments follow the settings in $4.1 and are conducted on T5-base. We report the average accuracy of few-shot learning and domain generalization in Table 4. More de- tailed results are in Appendix D.
To address these limitations, in the future we plan to conduct further cleansing and filtering on the current meta-training data. Besides, we intend to evaluate the few-shot performance of our frame- work the multilingual setting and also broaden the scope of tasks, including retrieval (Pan et al., 2023), language generation (Li ct al., 2022c) and vision-language tasks (Li et al., 2023b; Chen et al., 2023; Li et al., 2022a; Zhang et al., 2022a). Fur- thermore, we hope our work could pave the way for future research better leveraging parameter- efficient methods under few-shot settings.
Acknowledgements
The results of Row 1-3 indicate that considering diversified task formats during meta-training helps efficiently generalize to different tasks as down- stream tasks often contain various task formats. Row 4 and Row 5 highlight that task augmenta- tion plays an essential role in framework, with curriculum-based augmentation further enriching the task distribution and realistically simulating the distribution shift. Moreover, Row 6 validates the superiority of gradiett regularization in
This work has been supported in part by the Zhejiang NSF (LR21F020004), Key Research and Development Projects in Zhejiang Province (No. 2023C01030. 2023C01032), NSFC (No. 62272411), 1), National Key Research and Develop- ment Program of China (2018AAA000190),, Ant Group and Alibaha-Zhejiang University Joint Re- search Institute of Frontier Technologies.

---

References
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Akari Asai, Mohammadreza Salchi, Matthew Pe- ters, and Hannaneh Hajishirzi. 2022. ATTEMPT: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference Empirical Methods in Natu- ral Language Processing, pages 6655 (675,6672, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th Interna- tional Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126 1135. PMLR.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Asso- ciation of Computational Linguistics, pages 440-447, Prague, Czech Republic. Association for Computa- tional Linguistics.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joins Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 38 36-338,,, Online, Association for Computa- tional Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Voss,
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Ste- famie Tellex. 2019, Openwebtext corpus.
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.4401.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2022. PPT: Pre-trained prompt tuning for few-shot learning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8410-8423, Dublin, Ireland. Association for Computational Linguisties.
Nitay Calderon, Eyal Ben-David, Amir Feder, and Roi Reichart. 2022. DoCoGen: Domain counterfactual generation for low resource domain adaptation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volue 1: Long Papers), pages 7727-7746, Dublin, Ireland. As- sociation for Computational Linguistics.
Sepp Hochreiter, A. Steven Younger, and Peter R. Con- well. 2001. Learning to learn using gradient descent. In Artificial Neural Networks -itss--ee ICANN 2001, pages 87-94, Berlin, Heidelherg. Springer Berlin Heidel- berg.
Dong Chen, Kaibang Pan, Guoming Wang, Yueting Zhuang, and Siliang Tang. 2023. Improving vision anomaly detection with the guidance of language modality. arXiv preprint arXiv:2310.0282.
Yutai Hou, Hongyuan Dong. Xinghao Wang, Bohan Li, and Wanxiang Che. 2022. MetaPrompting: Learn- ing to learn better prompts. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 3251-3262, Gyeongju, Republic of Korca. International Committee on Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finn finetuned language models. arXiv preprint arXiv:2210.1 11416.
Minqing Hu and Bing Liu. 2004. Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages
Marie-Catherine De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Investi- gating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung.
Yukun Huang, Kun Qian, and Zhou Yu. 2022. Learn- ing better initialization for soft prompts via meta- learning. arXiv preprint wrXiv:2205.1247/
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre- Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages
Rabech Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzich Sacidi, Veselin Stoyanov, and Majid Yazdani. 2022. Prompt- free and efficient few-shot learning with language

---

models. In Proceedings of the 60th Annual Meer- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 3638-3652, 3652,, Dublin, Ireland. Association for Computational Linguisties.
Unsupervised reinforcement learning of transferable meta-skills for embodied navigation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 12123-12132.
Gregory Koch, Richard Zemel, and Ruslan Salakhut- dinov, 2015. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Work- shop, volume 2.
Juncheng Li, Junlin Xie, Long Qian, Linchao Zhu, Siliang Tang, Fei Wu, Yi Yang. Yueting Zhuang. and Xin Eric Wang. 2022b. Compositional tem- poral grounding with structured variational cross- graph correspondence learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 3032-3041.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameler-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian- Yun Nie. and Ji-Rong Wen. 2022c. Pretrained language mod- cls for text generation: A survey. arXiv preprint arXiv:2201.05273
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame. Julien Plu. Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Maan Majorr, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning:
Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Juint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, 2023. Pre- train, prompt. and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1335.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61-68, Dublin, Ireland. Association for Computational Lin- guistics.
Juncheng Li, Minghe Gao, Longhui Wei, Sil- iang Tang. Wenqiao Zhang, Mengze Li, Wei Ji, Qi Tian, Tat-Seng Chua, and Yueting Zhuang. 2023a. Gradient-regulated meta-prompt learning for gen- eralizable vasion-language models. urXiv preprint arXiv:2303.0657..
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. 2018. A simple neural attentive meta- learner. In International Conference on Learning Representations.
Juncheng Li, Xin He. Longhui Wei. Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang. Qi Tian, and Sil- iang Tang. 2022a. Fine-grained semantically aligned Lanounlanguuae pre-training. Advances in neural information processing systems, 35.7290-7303.
Quang Nguyen. 2015. The airline review dataset.
Juncheng Li. Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat- Seng Chua, Siliang Tang, and Yueting Zhuang. 2023b. Fine-tuning multimodal 1mss to follow zero- shot demonstrative instructions. arXiv preprint arXiv:2388.04152
Alex Nichol, Joshua Achiam, and John Schulman. 2018. On first-order meta-learning algorithms. arXiv preprint
Kaihang Pan, Juncheng Li, Hongye Song, Hao Fei, Wei Ji, Shuo Zhang, Jun Lin, Xiaozhong Liu, and Sil- iang Tang. 2023. Controlretriever: Harnessing the power of instructions for controllable retrieval. arXiv preprint arXiv arXiv:2308. 10025.
Juneheng Li. Siliang Tang, Linchao Zhu, Haochen Shi, Xuanwen Huang, Fei Wu, Yi Yang, and Yueting Zhuang. 2021. Adaptive hierarchical graph reason- ing with semantic coherence for video-and-language inference. In Proceedings of the IEEE/CVF Internu- tional Conference on Computer Vision, pages 1867- 1877.
Bo Pang and Lillian Lee. 2004. A sentimental educa- tion: Sentiment analysis using subjectivity summa- rization hased on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271-278. Barcelona, Spain.
Juncheng Li, Xin Wang, Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang, and William Yang Wang. 2020.

---

Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Anmual Meeting of the Association for Compu- lational Linguistics (ACL' '05), pages 1 115-124, Ann Arbor, Michigan. Association for Computational Lin- guistics.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning, In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 163 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Mohammad Taher Pilehvar and Jose Camacho- Cooladoo 2019. WIC: the coud-in-context dataset for evalu- ating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- fional I Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Tianxiang Sun, Zhengfu He, Qin Zhu, Xipeng Qiu, and Xuanjing Huang. 2022. Multi-task pre-training of modular prompt for few-shot learning. arXiv preprint arXiv:22100 221.0755..
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. 2018. Few-shot image recognition by predicting pa- rameters from activations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR).
Oriol Vinyals. Charles Blundell, Timothy Lillicrap, ko- ray kavukcuoglu, and Daan Wierstra, 2016. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to to-text transformer. Journal of Machine Learning Research, 11614)):1-67.
Ellen M. Voorhees and Dawn M. Tice. 2000. Building question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR '00, page 200-207, New York, NY, USA. Association for Computing Machinery.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Com- puting. Networking, Storage and Analysis, pages 1- 16. IEFE.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers). pages 5039-5059, Dublin, Ireland. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh. Julian Michael. Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates. Inc.
Jeff Rasley, Samyam Rajbhandari., Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 3505-3506.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLPP: Analyzing and Interpreting Neural Networks for NLP, pages 353 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Sachin Ravi and Hugo Larochelle. 2017. Optimization as model for few-shot learning. In International Conference on Learning Representations.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ane natural language processing.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few- shot learners. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.

---

Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Hongyi Zhang, Zhang, Moustapha Cisse. Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In International Conference Learning Representations.
Wenqiao Zhang, Jiannan Guo, Mengze Haochen Shi, Shengyu Zhang, Juncheng Li, Siliang Tang, and Yucting Zhuang. 2022a. Boss: Bottom-up cross- modal semantic composition with hybrid counterfac- tual training for robust content-based image retrieval. arXiv preprint arXiv:2207.04211
Wenqiao Zhang. Haochen Shi, Jiannan Guo, Shengyu Zhang. Qingpeng Cai, Juncheng Li, Sihui Luo, and Yucting Zhuang. 2022b. Magic: Multimodal rela- tional graph adversarial inference for diverse and unpaired text-based image captioning. Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 36, pages
Wenqiao Zhang, Siliang Tang, Yanpeng Cao,, Shiliang Pu, Fei Wu, and Yueting Zhuang. 2019. Frame aug- mented alternating attention network for video ques- tion answering. IEEE Transactions Multimedia, 22(4):1032-100 1041.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems, volume 28. Curran Associates, Inc.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-233
Yftah Ziser and Roi Reichart. 2018. Pivot based lan- guage modeling for improved neural domain adap- tation. In Proceedings the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long Papers), pages 1241-1251, New Orleans, Louisiana. Association for Computa- tional Linguistics.

---

Appendices
and
(U()a)
follows:
60
de
A Additional Information for SUPMER
(U))
8
UCD
(0o
a1
})
06
880
80%-
A.1 Complete Analysis of SUPMER
a (9))
0gu
In this section, we provide a more comprehensive and complete analysis of SUPMER. We will show that during meta-training, the optimization of soft prompt embeddings / and the meta- mneta-gradient regu- larization parameters & tends to maximize the inner product gradients obtained from the support set after regulation and gradients from the query set. Specifically, to update the parameters 0 and o, we should evaluate their gradients at first, denoting them as g and g Considering the original algo- rithm of MAMI., each task consists of a support set
di
agh
08g
(13)
a (9h)
H.
di
agh
(U
0
BCo
(0g
))
06
06
09g
(14)
1
021
86
So based on Eq. (12, 13, 14), we can finally approx- imate the gradients gº and g as:
(01)
and query set. And only one step of gradient de- scent is applied in the inner-loop optimization. To
80g aLI(U())))
make our statement more direct, we denote the loss function based on the support set and the query set
860
0C1 (U))
Co and L1. In SUPMER, ignoring the regular- ized loss, only L1 is directly utilized to optimize Φ while f is optimized in bi-level meta-optimization paradigm. Here we define the following terms re- lated similar Nichol et al. (2018):
a0,
00.
O
oी a H. (36)
aigl
H + O
8g
9
( (o.))) O
30,
0C, (9:)
d. (0,)
(gradient ohtained during SGD)
De
00; 5. (C,(
ac, (U(0))
(gradient at initial point)
06
00e
001 (U))
T 8 (0.)
(Hessian at initial point)
a0,
01 ( 0(a5)
8. an (98)
(gradient descent in the inner-loop)
06
(11)
For each definition {0, 1} and (-) is the meta-gradient regularization operation. to denotes the initial soft prompt embeddings for each step, and 8, denotes the embeddings after the inner-loop optimization. Obviously we have oi = 38. Firstly we perform Taylor series expansion to approxi- mate the SGD gradients of obtained from the query set as follows:
(o( O(a))
a1
0Φ
(15) T th (x)) and 86
indi- cate the optimization direction, which increases the inner product between gradients from the query set and gradients from the support set after transfor- mation. To further consolidate our analysis, we also track the normalized gradient inner product in the first 5000 steps during meta- vetantraining. As shown Figure 5, it is clear that the normalized gradient inner product gradually increases during meta-training.
Thus, 09
2
of (C1()1)
00,
(Ro)
0 (Hfu)
+
(01 0o) + 0(10; 0oll
00.
203
-o(0
On this basis, since there exists distribution shift between the support set and the query set after task augmentation, our method aligns the gradient di- rections across different distributions, which helps enhance model generalization. In other words, the trainable parameters descend in a coordinated man- ner such that the input-output correspondence is as
5. on
H
the ( O(af)
(12)
Then we analysis the gradient descent operation in the inner-loop optimization based on the sup- port set. Define U as the gradient descent and
(CColoo) ). So we can get have ((,) = fu an ( 360

---

Algorithm 1 Mcta- Meta-training Process of SUPMER
Gradient Inter Product
0.8
p(T): Distribbtion over anchor tasks
fo PLM with soft prompt embeddings 0
0.6
3: Vo Meta-gradient regularization
0.4
4: 011, 31, .3 Learning rate
5: TA Task augmentation in Algorithm 2
6: 8 <- --11
2000
4000
Training Steps
Randomly initialize 0,
while not done do
Figure 5: Normalized gradient inner products in the first 5000 steps during meta-training.
Sample batch of task (n) from p(T)
10:
(r [A((=)
close as possible across two distributions with de- viation. Besides, the meta-gradient regularization parameters & also retain some domain invariant in- formation of the meta-training data in the above process. Considering that Φ is fixed in downstream tasks, & can be applied to encourage the alignment between the nomain-speciftt gradients and avoid prompt-tuning overfitting to some domain-specific correlations.
11:
for all ri - {D D4, do
12:
Evaluate VoLDe (fo) with D. Evaluate VoLDy (fo) with D
domain-invarrant
13:
14:
Transform VoLD; (fo) via (.)(.) (6))
Veligg (fe)
15:
= [e (Jail (Veli; (Jo))
16:
8. = 0 - a1 dh V VoLDA (fe))
17:
end for
18:
+ Σ; si Σ
19:
0 0 31 Vo Σn, Lpg (fe)
A.2 Constructing Anchor Meta Tasks
20: - V Er, Lzg (Se)) + Creg)
Given sentence x from unlabeled corpora, we can derive semantically meaningful sentence embed- ding H = (x)) with PLMs, e.g., T5 encoder. And we apply K-means to cluster these unlabeled sentences according to their embeddings:
21: end while
22: return 0,
sentence similarity task, we set two sentences coming from the same cluster as label 0, and those from different clusters as label 1. In this way, the prompt template and verbalizer are designed as:
P. {pc} arg
Σ
Σ
-eel
(16)
(CE)-(ME)
MEC.
where ftc indicates the learned centroid of cluster C. and P indicates the partitions of all sentences. K-means clustering leads more abundant formats and objectives of veta-training tasks. Based on the results of K-means, we design three formats an- chor self-supervised meta-traine veta-training tasks: sentence- pair classification, multi-choice classification, and single-text classification. Here we introduce each of them in detail.
P "si (X) .82
(17)
== (0 yes, - no, 2 maybe)
Multi-choice Classification. Multi-choice clas- sification takes an anchor sentence xo as the query and we should find the correct one in several an- swer candidates. Here we also set two different tasks. The first one aims to select the sentence next to and the second one aims to select the sentence which belongs to the same cluster as 80- In each task we will set four candidates, and only one of them is correct. We design the prompt template and verbalizer follows:
Sentence-pair Classification. Sentence-pair classification takes a pair of sentences (wo, .1) as input, and æo is the anchor sentence. We carry on next sentence prediction task and sentence similarity task in sentence- pair classification with the label list y [0, 2).. For the former one. following Gu al. (2022), we set two sentences next to each other as label 0, those from the same document but not adjacent as label 2, and those from different documents as label 1. And for
P P= "sy? A.81 ... D.5.. Answer: (X)"
(18)
== (0 A,
Single-Sentence Classification. Through K- means clustering, each sentence is associated with a cluster label r; in {0, 1] where Tie = 1 iffe & and yie = 0 if + k. Here represents the cluster

---

which the sentence belongs. We simply use ri the pscudo label for meta-training and construct 4-way classification tasks. As for the designing the verbalizer, we transform the single-sentence classification into the format of multi-choice classi- fication. We insert the centroid of cluster Me into the template and use it to represent the correspond- ing cluster. So that we have:
Algorithm 2 TA Curriculum-basse Task Aug- mentation
I: (Tr)== A batch of anchor tasks
2: P(T) : Distribution over anchor tasks
3: 8 E [-1,1] Avg cos-sim between gradients
a, hyper-parameeerss
8 + (1 +1+
P "#o? ? A. (Mar) .. D. (maa). Answer: (X)
b (m² 1)/(m
V= (0 - A. > 2,2 - C,3 (3++D) D)
(19)
for all ri = (Di: D4. do
On this basis, for each task format, we separate all data into different tasks to construct anchor meta-training tasks with good task distributions. Through K-means, sentences with similar embed- dings are clustered into the same group. So in sentence-pair classification and multi-choice classi- fication, we group samples whose anchor sentence comes from the same cluster into the same meta- training task. And in single-sentence classification, for each meta-training task, we randomly select N clusters N classes and then sample k sentences for each cluster to construct N-way k-shot classi- fication task (N 4). In this way, we completely construct all anchor meta-training tasks.
8:
Sample task T, - ; Di,) from p(T)
9:
Draw À from Beta(a, ba)
// D., = (H ),"). Di Y
// H: the hidden representations of samples
10:
H. -(1 H + +
11:
Y (1- A)Y, XY
DI + -H, Y
12:
13: end for
14: return (T:)
B Dataset & Baseline Details
Few-shot Learning. We conduct experiments of few-shot learning on 6 different downstream En- glish tasks with 12 datasets. Since some of the test sets the datasets are not publicly available, following Karimi Mahabadi et al. (2022), we lever- age the original validation sets of SST-2, CB, RTE, QNLI, WIC, MRPC, and QQP! substitutes for the unavailable test sets. And the validation sets for few-shot learning inal training set, ensuring no overlap with our designated test sets. Besides, we download the datasets of SST-2, SST-5, MR, CR, and SUBJ from Gao et al. (2021). And the rest of the datasets are obtained from the HuggingFace Datasets li- brary (Lhoest et al., 2021). CB, RTE, BoolQ, and Wic are from SuperGL.UE Benchmark (Wang et al., 2019), while QNLI, MRPC, and QQP are from GLUE Benchmark (Wang et al., 2018) with Cre- ative Commons license (CC BY 4.0). We give the statistics of all these datasets in Table 5.
A.3 Additional Loss to Train
Regularization Parameters
In the meta-training stage, we optimize the meta- gradient regularization parameters & via Eq. (7), utilizing the same loss which optimizes the soft prompt embeddings. Here we introduce a regu- larized loss to attach some additional restrictions when updating the meta-gradient regularization pa- rameters. Notably, a higher value of by in Lgq. (10) indicates higher probability of larger distribu- tion deviation between the support set and the query set. Furthermore, in Eq. (5) we also tend in- crease to achieve a more pronounced gradient transformation with a more noticcable distribution deviation. From this perspective, z has a similar monotonicity with by. and they both range between 0 and 1. Thus we further add a regularized loss Grey ||2 1 to constrain the value of 2 and finally modify Eq. (7) into:
sampled from the orig-
Domain Generalization. Similar to Calderon et al. (2022), We evaluate on the sentiment analysis task including 6 different domains: Airlines (A), Books (B), DVDs (D), Electronics (E), Kitchen appliances (K), and Restaurants (R). Each domain has totally 2,000 manually labeled data of binary categories for testing, including 1000 positive and
① - -- AvV Σ Coo,(oo)) ACrea)
(20)
Te-p(T)
A.4 Pseudo-Codes of SUPMER
We show the pseudo-codes for the meta-training process of SUPMER Alg. And the process of curriculum- based task augmentation is described in Alg. 2.
hhttps://quoradar http://uurrraaauuuuuuroouo

---

Dataset
Task
#Train
#Test
K
Hyper-parameter
Value
SST-2
Sentiment analysis
Number of clusters for each task format
250
SST-5
Sentiment analysis
8544
2210
5
Tasks per batch
MR
Sentinent analysis
8062
2000
2
Size support set per task
Sentiment analysis
1774
2000
Size query set per task Optimizer
SUBJ
Subjectivity classification
8000
2000
2
Adam
TREC
Question classification
5452
500
6
Inner loop learning rate Outer loop learning rate Leurning rate for & Scheduler
0.1
0.1
CB
Natural language inference
250
lc-4
RTE
Natural language inference
2490
277
2
Lincar scheduler
QNLI
Question answering
104743
5463
Warm-up up steps
0
WIC
Word sense disambiguation
5428
Max training steps
100,000
Validation steps
2,000
MRPC
Paraphrase detection
3668
408
2
Max sequence length
512
QQP
Paraphrase detection
363846
40430
1,0
2.0
Table 5: Statisties of datasets for few-shot learn-
ing. K is the number of labels. We sample N x K instances from the original training set to construct the few-shot training and validation sets. And #Test shows the size of the test set.
Table Hyper-parameters for SUPMER, denotes the meta-gradient regularization parameters. À is the coefficient of the regularized loss. And m is the curve parameter the curriculum-base task augmentation.
1000 negative. We choose as the source domain and the other five (B, D, E, K, R) constitute the target domains. We sample 16 instances per la- bel from the training set of the source domain for prompt tuning and then evaluate on the test sets of all 6 domains.
shot inference and few-shot inference. For both of them, we directly employ Flan- T5-XL for down- stream evaluation, coupled with carefully designed task instructions for each dataset. Furthermore, in few-shot inference, we also provide an appropriate number of few-shot examples to form demonstra- tion context.
Baselines. We first compare with baseline meth- ods with the same number of parameters SUP- MER. These methods utilize prompt tuning (Lester et al., 2021) to handle downstream tasks, with the key distinction lying in the initialization the soft prompts. Vallina prompt tuning (PT Lester et al., 2021) directly tunes the soft prompts in the down- stream task, which are randomly initialized from a normal distribution. PPT (Gu et al., 2022) pre- trains soft prompts in supe-supe way with formats of pre-training tasks: sentence-pair classi- fication, multiple-choice classification and single- text classification. Unified-PPP (Gu et al., 2022) formulate all these three formats into a unified task form. MetaPT (Huang et al., 2022) using a su- pervised sentiment analysis dataset Yelp5 as the meta-training data and directly leveraging MAML. to initialize soft prompts.
C Training Details
We apply the T5 base model (Raffel et al., 2020) (220M parameters) and Flan-T5-XL model (Chung et al., 2022) (3B parameters) as the underlying PLM, and use the HuggingIFace Pytorch implemen- tation (Wolf et al., 2020). We run experiments with GeForce RTX 3090 24G GPUs. And the meta- training process of SUPMER takes about 140 GPU hours. Next we will describe the details of training hyper-pane parameters in the case of leveraging T5-base as the PLM.
C.1 Training Hyper-parameters for Downstream Tasks
In our experiments, we leverage full-model tun- ing and prompt tuning to solve downstream tasks, including few-shot learning and domain general- ization. In few-shot learning, following some prior work (Schick and Schütze, 2021; Karimi Mahabadi et al., 2022), we set the maximum sequence length of each example to 256 for CR. SUBJ, CB, RTE and WiC, and 128 for other datasets. While in do- main generalization, the maximum sequence length of each example is set to 256.
To further demonstrate the effectiveness of our method, we also consider baseline methods with more tunable parameters, including Prefix- Tuning (Li and Liang, 2021) and P-tuning-v2 (Liu et al., 2022), which add prompts at each layer of PLM. We also compare with full-model tuning (FT) that fine-tunes all parameters of the PLM. Given that FLAN-T5-XI. was also designed with sew-shot inference in mind, we newly compare with two baseline methods on FLAN-T5-XL. zero-
We run each experiment 5 times on the random

---

Methods
SST-2
SS1-5
MR
CR
SUBJ
TREC
CB
RTE
QNLI
WiC
MRPC
QQP
only sp
83.6Ls
42.624
81.7aa
86.0u.
65.82a
71.2a,
64.62.1
57.0g.
58.4a.
53.61
89.9Ls
66.02.0
only me
63.4,.4
44.519
1
88.90.
70.547
66.41.4
65.9g.
64.9..
58.71.7
64.2;
08.8o.s
67.61a
only
81.5
45.0g.
81.5o.
las
73.3
79.1a.s
62.12.
53.9.
56.514
5
67.71.
63.71.7
w/o ta curiculum
S1.T1.
1..
61.9..
87 Bus
73.62a
78 78.8u.
66 66.41.
56.6a.
39.41.
54.324
69.5
70.21.1
90.8o.s
40.82.
$2.3.a
88-40.
74.8s
79.7
71.02.1
50.5c.s
62.61.4
55.41.1
89.7a.s
71.99.2
wdo mgr
85.0
44.5)
62.8o
86.0o.
7
79.5s.
67.1
56.8o.
58.9g.4
64.4g
70.0..
70.369
SUPMER
87.30.s
46.7a.a
84.0p.a
88.33..
79.62.
80.2p.
72.41.
57.31.o
61.71.
64.81
71.3o.s
70.5
Table 7: Detailed results of ablation study for few-shot learning to illustrate the effect of individual Components. In the first three rows we keep only one anchor task format during meta- tataattraiinng.... and sp stands for sentence- pair classification, mc for multi-choice classification, for single-sentence classification. And w/o ta means entirely removing task augmentation, w/o curriculum only retains the vanilla task augmentation without the curriculum- based idea. w/o mgr means removing the meta- gradiett regularization method.
Source
Target
Method
A
B
D>
E
K
R
only sp
83.41.1
82.
83.00.r
88.51.2
88.9p.e
88.1o.7
only me
84.0o.s
82.31.2
81.5u.9
88.51.
89.3u.7
88 88.88.7
only ss
83.6u.7
84.7u.s
84.20.s
88.9u.9
89.7a.9
89.0p.3
w/o ta
83.4o.s
82.01.4
81.71.a
87.8g.s
88 58.2p.6
88 88.6p.c
w/o curriculum
84.0p.s
84.70.a
83.90.a
89.6g.5
90.3p.s
89.71.1
mgr
83.80.4
83.40.5
83.3a.5
88.1a.6
89.2p.s
88.9p.4
SUPMER
85.7o.5
85.3o.a
85.1o.4
90.3o.r
91.1o.5
90.4o.4
Table Detailed results of ablation study for domain generalization to illustrate the effect of individual Components.
seed [10, 20, 30, 40, 50] and report the average ac- curacy as well as the standard deviation. For both full-model tuning and prompt tuning, We imple- ment AdamW as the optimizer. We use a batch size of 32 and train the model for 200 epochs, mean- while evaluating the model every 10 steps. And we report the results for poper-parameters performing the best on the validation set for each task.
validation set.
C.2 Training Hyper-parameters for Prompt Initialization
Pre-training for prompt initialization. Gu al. (2022) proposes two frameworks for unsuper- vised prompt pre-training, named PPT and Uni- fied PPT. PPT designs three formats of unsuper- vised pre-training tasks (sentence- paiteenceeppirr classifica- tion, multiple-choice classification and single-text classification), and Unified-PPP further formulate them into a unified task form. We implement PPT and Unified-PPT following the hyper-parameter provided in Gu et al. (2022) and reset the pre- trained language model to T5-base and Flan-T5- XL. Specifically, for both PPT and Unified-PPT, we sample 10GB of unlabeled data from Open Web- Text to construct pre-training tasks for each task format. And 5% data are split for validation. We apply the "inverse square root" learning rate sched- uler with warm-up steps and set the learning rate as 0. We set the batch size 256 with the max sequence length as 512, and train soft prompts for at most 200,000 steps. We evaluate the perfor- mance on the validation set every 2,000 steps and choose prompts with the lowest validation loss.
Besides, for full-model tuning, all parameters of PLM are fine-tuned without adding soft prompts. We use the learning rate of [lc-5, 2e-5, 3e-5] and choose the obtaining the highest validation performance. Moreover, to fine-tune the Flan-T5- XL. model, we use ZeRO (Rajhhandari et al., 2020) stage-2 provided in DeepSpeed (Rasley ct al., 2020) to reduce GPU memory usage.
For prompt tuning, we freeze PLM parame- ters and only tune soft prompts composed of 100
soft tokens. As a result, the tunable parameters of
prompt tuning are only 77K with T5-base and 205K
with Flan-T5-XX updating around 3000 and 1,5000 times fewer parameters base and F5an-T5-XI, respectively, compared to full-model tuning. And we find that prompt tuning requires a much larger learning rate than full-model tuning. We search for the learning rate in [Ic-1, 1, 3e-1) ]] and also choose the model with the best performance on the

---

Source
Target
Method
AVG
A
B
D
E
K
SUPMER (only laheled)
86.4
84.7
84.8
90.0
90.7
87.3
2 SUPMER (only unlabeted)
85.7
85.3
85.1
90.3
91.1
87.5
3 PPT (labeled + unlabeled)
83.1
79.0
84.4
89.3
90.6
85.3
4 MetaPT (labeled + unlabeled)
86.3
85.3
86.7
90.1
91.4
88.0
5 SUPMER (labeled unlabeled)
86.6
88.6
88.5
92.7
93.7
90.0
Table 9: Results of domain generalization T5-hase, considering different data and methods for prompt initializa- tion. As our collected labeled data includes Yelp5, a sentiment analysis dataset the domain of restaurants, we conduct the experiments of domain generalization only across domains A, B, D, E, K (without R).
Meta-training for prompt initialization. In our SUPMER framework, sample 10GB of unla- beled data from OpenWehTent to construct self- supervised meta-training tasks. We split 5% data to construct tasks for validation. And for each task for- mat, we first set the number of clusters to 250. We sample trata-traning tasks in a batch, and train the prompt embeddings and the meta-gradient regularization parameters & for at most 100,000 steps. We also evaluate the performance validation set every 2,000 steps, choosing and @ with the lowest validation loss for downstream tasks. Table 6 lists all training super-parameters for SUPMER. worth noting that for most hyper- parameters in Table 6, we just set default value by experience without tuning them. we tune the hyper- parameters which are also tuned in other baselines (e.g., learning rate), ensuring all methods have the same number of tunable aperrpprrmeters in our experiments.
T5-base as the underlying PLM. We run each ex- periment 5 times on the random seed [10, 20, 30, 40, 50) and report the average performances well as the standard deviation. The detailed results of few-shot learning and domain generalization are shown Table and Table 8. We can see each component is critical our framework.
Besides, in $4.4, to explore the superiority of self-supervised meta-learning and the impact of integrating additional labeled data for soft prompt initialization, we conduct experiments of few-shot learning T5-base, considering different data and methods for soft prompt initialization. We also carry out experiments of domain generation lever- aging different data with various prompt initial- ization methods, with the results presented in Ta- ble 9. From Table 3 and 9, it is evident that self- supervised meta-learning utilizing unlabeled data exhibits enhanced adaptability to unseen tasks in comparison to its supervised counterparts. And amalgamating both labeled and unlabeled data for the construction of meta-training tasks emerges as a more advantageous strategy. When comes to employing both labeled and unlabeled data for prompt initialization, SUPMER continues to show- case markedly superior results in contrast to base- line methods in the realms of both few-shot learn- ing and domain generalization.
the
Morcover, to illustrate the superiority of self-supervised meta-learning, we also imitate MetaPT(Huang et al., 2022) to initialize soft prompts via supervised meta-learning. MetaPT uses supervised sentiment analysis dataset Yelp5 the meta-training data, which has 650,000 train- ing samples only covering the domain of restau- rants. Following Huang et al. (2022), We group all labeled data into 10 clusters through K-means. And we set the inner loop learning rate to 0.08, the outer loop learning rate to 0.025 with the early stop pa- tience as 6. Other hyper-parametee parameters are consistent with those in SUPMER.
D Full Results of Ablation Study
In this section, first give detailed experimental results of the ablation study illustrate the effect of individual components. We evaluate each abla- tion model over all 12 datasets of few-shot learning and all 6 domains of domain generalization, with


Reviews:
[{"review_id": "jPix4Y17bS", "reviewer": "Reviewer_a2Py", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable comments! We are encouraged to see that our work is recognized as **novel and interesting**. To better explain your concerns, we have added some new experiments: \n1. Comparing with **few-shot inference on Flan-T5-XL**. \n2. Leveraging **T5-v1.1-XL** as backbone model.\n3. Leveraging **GPT-2-Large** as backbone model.\n\n&nbsp;\n\n### **1. Comparing with few-shot inference on Flan-T5-XL**.\nGiven that  FLAN-T5-XL was also designed with few-shot inference in mind, we newly compare with two baseline methods on FLAN-T5-XL: **zero-shot inference** and **few-shot inference**. For both of them, we directly employ Flan-T5-XL for downstream evaluation, coupled with carefully designed task instructions for each dataset. Furthermore, in few-shot inference, we also provide an appropriate number of few-shot examples to form a demonstration context. The experimental results are as follows:\n\n\n**Experimental results of few-shot learning on FlanT5-XL:**\n| **Method & Dataset** | SST-2 | SST-5 | MR | CR | SUBJ | TREC | CB | RTE | QNLI | WiC | MRPC | QQP | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| zero-shot inference | 89.1 | 52.3 | 83.3 | 80.6 | 57.4 | 87.2 | 76.8 | 75.8 | 85.0 | 50.5 | 77.2 | 77.5 | 74.4 |\n| few-shot inference | 93.2 | 53.3 | 88.5 | 87.8 | 58.6 | 91.6 | 83.9 | 79.1 | 86.9 | 64.3 | **79.9** | 81.0 | 79.0 |\n| FT | 92.9 | 53.6 | 89.6 | **91.3** | 88.7 | **93.6** | 77.8 | 76.0 | 84.4 | 63.0 | 77.7 | 82.9 | 81.0 |\n| PT | 88.2 | 45.7 | 85.6 | 88.6 | 81.6 | 85.1 | 65.4 | 61.6 | 69.3 | 54.8 | 71.1 | 70.2 | 72.3 |\n| SUPMER | **95.5** | **55.3** | **91.4** | 90.7 | **90.3** | 93.0 | **87.6** | **81.4** | **88.3** | **65.0** | 78.1 | **85.1** | **83.5** |\n\n**Experimental results of domain generalization on FlanT5-XL:**\n| **Method & Domain** | B | D | E | K | A | R | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | \n| zero-shot inference | 77.8 | 84.6 | 86.2 | 86.8 | 88.6 | 87.8 | 85.3 |\n| few-shot inference | 82.5 | 90.3 | 89.7 | 92.3 | 92.2 | 89.2 | 89.4 |\n| FT | 86.2 | 88.8 | 84.6 | 87.2 | 89.8 | 90.7 | 87.9 |\n| PT | 82.6 | 79.0 | 82.5 | 84.2 | 84.5 | 84.8 | 82.9 |\n| SUPMER | **87.0** | **89.8** | **91.1** | **95.1** | **95.8** | **91.8** |  **91.8** |\n\n> **[1.1]**  Whether the difference between fine-tuning and SUPMER is due to overfitting on FT?\n\nFirst, we can see that few-shot inference surpasses FT on some datasets. This validates that for FLAN-T5-XL, employing few-shot examples as the demonstration context during inference, rather than utilizing them for parameter tuning, can also showcase impressive capabilities. Thanks for your constructive suggestion, and ***it is indeed necessary to consider few-shot inference as a baseline method*** for comparative analysis.\n\nFurthermore, comparing the results of zero-shot inference and FT, we can see that after fine-tuning model parameters based on few-shot examples, **FT consistently outperforms zero-shot inference across almost all datasets, including QQP and TREC** (FT also achieves better average performance than few-shot inference in the experiments of few-shot learning). During inference, both zero-shot inference and FT only utilize test samples without the incorporation of any additional few-shot in-context examples. The superior performance achieved by FT compared to zero-shot inference, thereby underscores ***the positive impact of FT on enhancing model performance, without encountering overfitting*** which may lead to a decline in performance.\n\n> **[1.2]**  Comparing SUPMER with few-shot inference\n\nFor both few-shot learning and domain generalization, **SUPMER demonstrates superior performance across almost all datasets and domains in contrast to few-shot inference**. Specifically, SUPMER exhibits average gains of **4.5** points in few-shot learning and **2.4** points in domain generalization, compared to few-shot inference.  These results provide further evidence that ***for LMs such as FLAN-T5XL with inherent few-shot inference capabilities, our approach can significantly enhance their abilities in a parameter-efficient tuning strategy***, without providing any in-context examples during inference. This effectively proves SUPMER's efficacy on FLAN-T5-XL.\n\n&nbsp;\n\n### **2. Leveraging T5-v1.1-XL as backbone model**\n\nAs FLAN-T5-XL was fine-tuned on a collection of datasets with the ability of few-shot inference in mind, we employ **T5-v1.1-XL** as the backbone model to ensure a more fair comparison between FT and SUPMER. T5-v1.1-XL was only pre-trained on the unsupervised dataset C4, so the primary distinction between T5-v1.1-XL and T5-base lies in their model scales.\n\nWe list the experimental results on T5-v1.1-XL in the tables below. We have the following observations:\n\n(1) **When utilizing T5-v1.1-XL as the backbone, SUPMER still significantly outperforms FT**, exhibiting average gains of 3.1 points in few-shot learning and 5.5 points in domain generalization. This underscores the effectiveness of SUPMER. Furthermore, in comparison to the experimental results obtained by employing T5-base as the backbone model (refer to Tables 1 and 2 of the main paper), the performance improvement of SUPMER over other baseline methods is more significant when utilizing the larger T5-v1.1-XL. ***It shows that SUPMER has a good indication of being scale-friendly.***\n\n(2) Upon integrating SUPMER with T5-v1.1-XL, its performance in both few-shot learning and domain generation surpasses the zero-shot performance of Flan-T5-XL. This underscores that, during inference without few-shot examples, *by additionally integrating the ingeniously trained soft prompts from our approach into the LM input,* ***even models that have not undergone instruction-tuning, can still achieve superior performance compared to their instruction-tuned counterparts***.\n\n&nbsp;\n\n**Experimental results of few-shot learning on T5-v1.1-XL:**\n| **Method & Dataset** | SST-2 | SST-5 | MR | CR | SUBJ | TREC | CB | RTE | QNLI | WiC | MRPC | QQP | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| FT | 90.8 | 42.8 | 86.2 | 88.6 | 84.5 | **88.6** | 76.8 | 58.5 | 66.8 | 54.5 | 70.4 | 70.7 | 73.3 |\n| PT | 73.6 | 39.5 | 80.0 | 86.9 | 73.7 | 75.4 | 60.7 | 55.2 | 61.6 | 53.3 | 68.4 | 67.8 | 66.3 |\n| SUPMER | **92.0** | **46.9** | **88.0** | **89.4** | **89.5** | 88.2 | **85.7** | **60.4** | **70.8** | **57.9** | **73.2** | **74.3** | **76.4** |  \n\n**Experimental results of domain generalization on T5-v1.1-XL:**\n| **Method & Domain** | B | D | E | K | A | R | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | \n| FT | 85.8 | 84.2 | 82.6 | 85.2 | 85.7 | 85.5 | 84.8 |\n| PT | 80.0 | 77.5 | 80.1 | 82.8 | 83.4 | 83.4 | 81.2 |\n| SUPMER | **86.7** | **88.1** | **88.7** | **93.2** | **94.1** | **91.0** | **90.3** | \n\n&nbsp;\n\n### **3. Leveraging **GPT-2-Large** as backbone model**\n\nIn addition to the backbone model from the T5 model family, we have also conducted experiments on GPT-2-Large. We compare SUPMER with FT, PT, PPT, and MetaPT. The experimental results are listed in the following tables. We can see that ***SUPMER can still achieve better performance on LMs beyond the T5 family.*** Specifically, SUPMER outperforms all baselines across 10 of all 12 datasets in few-shot learning, and demonstrates superior performance than all baselines across 5 of all 6 domains in domain generalization, while also achieving significantly better average performance.\n\n&nbsp;\n\n**Experimental results of few-shot learning on GPT-2-Large:**\n| **Method & Dataset** | SST-2 | SST-5 | MR | CR | SUBJ | TREC | CB | RTE | QNLI | WiC | MRPC | QQP | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| FT | 84.0 | 41.5 | 81.4 | 83.1 | **85.3** | **79.0** | 69.7 | 60.2 | 62.7 | 54.4 | 70.1 | 71.0 | 70.2 | \n| PT | 73.2 | 32.4 | 75.6 | 79.8 | 65.7 | 64.2 | 62.9 | 57.0 | 59.5 | 53.2 | 67.8 | 66.6 | 63.2 |\n| PPT | 79.7 | 34.1 | 79.3 | 82.3 | 72.3 | 71.6 | 66.1 | 59.3 | 62.2 | 53.5 | 69.4 | 67.0 | 66.4 |\n| MetaPT | 85.1 | 42.8 | 84.7 | 87.0 | 77.4 | 75.6 | 67.9 | 58.5 | 61.4 | 53.9 | 69.2 | 70.2 | 69.5 |\n| SUPMER | **86.2** | **45.2** | **85.3** | **87.8** | 84.6 | 78.4 | **71.4** | **60.8** | **64.2** | **54.9** | **70.6** | **72.8** | **71.9** |\n\n**Experimental results of domain generalization on GPT-2-Large:**\n| **Method & Domain** | B | D | E | K | A | R | *AVG_SUB* | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | \n| FT | 84.6 | 83.5 | 81.7 | 84.0 | 86.6 | 85.8 | 84.1 | 84.4 |\n| PT | 80.7 | 76.9 | 77.2 | 79.0 | 79.8 | 83.1 | 78.7 | 79.5 |\n| PPT | 83.2 | 79.7 | 83.7 | 88.3 | 85.3 | 85.9 | 84.0 | 84.4 |\n| MetaPT | **85.7** | 85.1 | 85.0 | 88.8 | 90.7 | --- | 87.1 | --- |\n| SUPMER | 85.3 | **86.4** | **86.3** | **90.4** | **91.9** | **88.8** | **88.1** | **88.2** |\n\n&nbsp;\n\nOverall, we extend our heartfelt gratitude for your valuable suggestions! The next version will encompass the aforementioned experiments. We will also thoroughly incorporate your suggestions on presentation improvement and make revisions to Figure 2 in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel approach to initialize soft-prompt tuning which also works in a self-supervised paradigm, meaning there is virtually no need for a human-in-the-loop.\n2. Shows that the results stay consistently high as training continues as oppose to other approaches that deteriorate over time.", "reasons_to_reject": "1. While the approach is interesting, the evaluation done is less compelling given the model used in this work, specifically Flan-T5-XL. The model is based on T5-v1.1 models finetuned on a collection of dataset which already includes TREC and QQP. It is therefore still unclear if the difference between finetuning and SUPMER is due to overfitting on FT.\n2. FLAN-T5-XL was also designed with few-shot inference in mind which may have contributed to the higher gains when compared to a model that wasn't such as T5-Base (Although size may very well be a factor). This means there are a few unobserved variables that were taken to account.\n3. To effectively prove SUPMER's efficacy on FLAN-T5-XL, comparing SUPMER with few-shot inference and showing that SUPMER is still better despite FLAN-T5-XL's initial design would have even shown how important this method is. Instead of FLAN-T5-XL, had this work use models such as T5-v1.1 which was only trained on C4, the results may have become more convincing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work introduces an approach that utilizes meta-learning to effectively train a model with soft-prompt tuning. The method proposes a self-supervised framework to initialize the soft-prompts, integrate a regularization that helps with domain-generalization. The method is compared against existing approaches such as full-model finetuning.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Figure 2 is quite difficult to understand without reading section 3 first. The use of numbers to enumerate which steps are done first between the 3 dotted boxes may help bring better clarity.\n2. The arrows don't seem to be perfectly aligned. This should be easily fixed."}}, {"review_id": "yQVRhPvwXf", "reviewer": "Reviewer_33e1", "summary": "", "questions": "I see no major issues with the work. Given other research and works presented on the topic of few-shot learning, it seems that the results shown fall in line with expectations given other similar works. The authors appear to provide a novel method that advances previous works. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our work and the valuable suggestions you've provided! We are encouraged to see that our work is recognized as **novel, backed by rich experiments**. And we will diligently adopt your suggestions on presentation improvement and revise Figure 2 and the section of introduction in the next version. \n\nBased on your suggestions, we will divide Figure 2 into separate figures for better explanation, while also incorporating some essential textual annotations. We will make a clearer exposition of various elements within the figure, such as the task interpolation and the curriculum-based approach. Furthermore, within the introduction section, we will endeavor to provide a more elegant depiction of the base problem and all relevant terms, ensuring a more intuitive presentation for readers.\n\nOnce again, we extend our heartfelt gratitude to you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The author outlines their novel method thoroughly, breaking the problem down into distinct steps. \n\nThe paper provides a rich comparison between their method and other existing methods. They give results for two models of different sizes, across multiple datasets, and with different tasks. Additionally, they outline the limitations of their study and discuss possible future investigations. \n\nThe author provides the results of an ablation study to show that each component of their method is contributing to the overall advantage in the performance of their method. ", "reasons_to_reject": "Figure 2 could be broken down and better explained as separate figures, which could then be combined into what we see in Figure 2. The meaning behind the elements of your figure is not always immediately clear. For example, the meaning of the colored dots in Task Interpolation becomes clear after some analysis of the text and diagram but it also would be better served by an explanation specifically referring to the Curriculum-based Task Augmentation part of the diagram within Figure 2. \n\nThe introduction could be revised to better introduce key concepts before beginning to discuss the motivations of the paper. This is not critical as the authors always define a term before discussing it. However, the introduction could more elegantly describe the base problem and all relevant terms before then moving to discuss the more specific motivations that lead to the methods which are presented in this paper. ", "questions_to_authors": "I see no major issues with the work. Given other research and works presented on the topic of few-shot learning, it seems that the results shown fall in line with expectations given other similar works. The authors appear to provide a novel method that advances previous works. ", "paper_topic_and_main_contributions": "The author presents SUPMER, a self-supervised algorithm that uses a set of anchor tasks within one or more datasets to use meta-gradient regularization to learn meta-prompts which are then used to train on a series of meta-tasks. By incorporating meta-learning, the authors have been able to create a novel method that learns new data over n-shots by learning the task within this meta-embedding. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "l1emthXf14", "reviewer": "Reviewer_XEMq", "summary": "", "questions": "- How exactly are the sentence embeddings computed for the k-means clustering?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable comments! We are encouraged that you think our work presents a **novel method with strong empirical evidence highlighting its superiority**. We will explain your concerns point by point.\n\n&nbsp;\n\n> **[1]** The current experiments are performed on one model family: T5.\n\nTo further demonstrate that SUPMER can also achieve superior performance on LMs beyond the T5 family, we newly leverage **GPT-2-Large** as the backbone model and conduct experiments under the same experimental setup. We compare SUPMER with FT, PT, PPT, and MetaPT. \n\nThe experimental results are shown in the following tables. We can see that **SUPMER outperforms all baselines** across 10 of all 12 datasets in few-shot learning, and **demonstrates superior performance** than all baselines across 5 of all 6 domains in domain generalization. **SUPMER also achieves significantly better average performance** in terms of both few-shot learning and domain generalization. So we can say that **SUPMER still demonstrates its effectiveness on LMs beyond the T5 family**.\n\n&nbsp;\n\n**Experimental results of few-shot learning on GPT-2-Large**:\n| **Method & Dataset** | SST-2 | SST-5 | MR | CR | SUBJ | TREC | CB | RTE | QNLI | WiC | MRPC | QQP | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| FT | 84.0 | 41.5 | 81.4 | 83.1 | **85.3** | **79.0** | 69.7 | 60.2 | 62.7 | 54.4 | 70.1 | 71.0 | 70.2 | \n| PT | 73.2 | 32.4 | 75.6 | 79.8 | 65.7 | 64.2 | 62.9 | 57.0 | 59.5 | 53.2 | 67.8 | 66.6 | 63.2 |\n| PPT | 79.7 | 34.1 | 79.3 | 82.3 | 72.3 | 71.6 | 66.1 | 59.3 | 62.2 | 53.5 | 69.4 | 67.0 | 66.4 |\n| MetaPT | 85.1 | 42.8 | 84.7 | 87.0 | 77.4 | 75.6 | 67.9 | 58.5 | 61.4 | 53.9 | 69.2 | 70.2 | 69.5 |\n| SUPMER | **86.2** | **45.2** | **85.3** | **87.8** | 84.6 | 78.4 | **71.4** | **60.8** | **64.2** | **54.9** | **70.6** | **72.8** | **71.9** |\n\n**Experimental results of domain generalization on GPT-2-Large**:\n| **Method & Domain** | B | D | E | K | A | R | *AVG_SUB* | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | \n| FT | 84.6 | 83.5 | 81.7 | 84.0 | 86.6 | 85.8 | 84.1 | 84.4 |\n| PT | 80.7 | 76.9 | 77.2 | 79.0 | 79.8 | 83.1 | 78.7 | 79.5 |\n| PPT | 83.2 | 79.7 | 83.7 | 88.3 | 85.3 | 85.9 | 84.0 | 84.4 |\n| MetaPT | **85.7** | 85.1 | 85.0 | 88.8 | 90.7 | --- | 87.1 | --- |\n| SUPMER | 85.3 | **86.4** | **86.3** | **90.4** | **91.9** | **88.8** | **88.1** | **88.2** |\n\n&nbsp;\n\n> **[2]** How exactly are the sentence embeddings computed for the k-means clustering?\n\nTo derive the sentence embedding for each sentence from the unlabeled corpora, we leverage the pre-trained T5 encoder for sentence encoding. The hidden representations from the last layer of the T5 encoder are extracted, and we subsequently employ mean pooling to aggregate the token embeddings, thereby generating the sentence embedding. With sentence embeddings, then we can directly apply K-means to cluster the unlabeled sentences. We give more details about this in Appendix A.2 within our paper.\n\n&nbsp;\n\n> **[3]** Typos Grammar Style And Presentation Improvements: line 168 -> space missing\n\nThank you for pointing out the omission! We will add the missing space in line 168 in the next version. Additionally, we will also incorporate the aforementioned experiments using GPT-2-large as the backbone in the next version. Once again, we sincerely appreciate your constructive advice!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Strong empirical evidence highlighting the superiority of the method in few-shot learning and domain generalization experiments.\n- The paper is well written, despite all the complexity and moving parts of the proposed method.\n- The work focuses on the relevant problem of prompting pre-trained language models, an efficient and practical alternative to finetuning.", "reasons_to_reject": "- The experiments are performed on one model family: T5. Particularly, T5-base and Flan-T5-XL, which limits the scope of the work. However, the method seems to distance itself from other baselines with the larger model, which is a good indication of being scale-friendly.", "questions_to_authors": "- How exactly are the sentence embeddings computed for the k-means clustering?", "paper_topic_and_main_contributions": "This work presents a novel method, SUPMER, that incorporates meta-learning concepts to improve the prompting of pre-trained language models. The different method components tackle common problems of existing prompting methods, namely the need for a good soft prompt initialization and overfitting in few-shot settings due to the small number of training examples. The results indicate the superiority of SUPMER across a variety of tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 168 -> space missing"}, "extra_text": {}}, {"review_id": "BgiJJ5pFfu", "reviewer": "Reviewer_gNne", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate you for the constructive and insightful feedback! We are greatly encouraged by the recognition of our work's **intuitive methodology and solid experiments**. To better address the concern of employing labeled data, we have added additional parts of experiments to show (1) the impact of incorporating labeled data, and (2) the effectiveness of SUPMER compared with other labeled-data-integrated prompt initialization methods after the integration of labeled data.\n\n&nbsp;\n\n### **1. The impact of incorporating labeled data**\n\nIn order to explore whether the utilization of labeled data during the meta-training process could enhance the performance of SUPMER, we collect a set of labeled datasets to formulate meta-training tasks for soft prompt initialization. And then we conduct the experiments under three settings: \n(1) **SUPMER *(only labeled)*** utilizes only labeled data for constructing meta-training tasks. \n(2) **SUPMER *(only unlabeled)*** utilizes only unlabeled data for constructing meta-training tasks.\n(3) **SUPMER *(labeled + unlabeled)*** involves a combination of both labeled and unlabeled data for the construction of meta-training tasks, with a mixing ratio of labeled to unlabeled as 1:2.\n\nTo improve the efficiency of our experiments, we employ the T5-base backbone for few-shot learning and domain generalization. From the results in the following tables, we have the following observations:\n\n(1) By comparing *SUPMER (only labeled)* and *SUPMER (only unlabeled)*, we can see that **self-supervised meta-learning with unlabeled data can better generalize to unseen tasks when compared to supervised meta-learning**. This can be attributed to the fact that unlabeled data generally covers broader semantic concepts. Specifically, as our labeled data contains some sentiment analysis datasets, *SUPMER (only labeled)* and *SUPMER (only unlabeled)* reveal proximity in their performance on sentiment analysis tasks (i.e., SST-2, SST-5, MR, CR). But in other tasks, using unlabeled data consistently achieves better results than utilizing only labeled data, also with a higher average accuracy over all datasets/domains.\n\n(2) *SUPMER (labeled+unlabeled)* also significantly outperforms *SUPMER (unlabeled)* and *SUPMER (labeled)* *(**72.5** v.s. 71.3 v.s. 70.0 in few-shot learning *&* **90.0** v.s. 87.5 v.s. 87.3 in domain generalization)*. Thank you for your constructive advice! The results further validate that **combining both labeled and unlabeled data for meta-training task construction is indeed a superior choice.** It allows us to harness the high-quality advantages of labeled data while also exploiting the broader semantic concepts encapsulated by unlabeled data. As a result, the utilization of both labeled and unlabeled data outperforms the use of either labeled or unlabeled data alone. \n\n&nbsp;\n\n**Experimental results of few-shot learning on T5-base**:\n| **Method & Dataset** | SST-2 | SST-5 | MR | CR | SUBJ | TREC | CB | RTE | QNLI | WiC | MRPC | QQP | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| SUPMER *(only labeled)* | 87.5 | 47.0 | 83.8 | 89.9 | 75.4 | 79.6 | 67.9 | 56.6 | 59.0 | 54.6 | 69.2 | 69.5 | 70.0 |\n| SUPMER *(only unlabeled)* | 87.3 | 46.7 | 84.0 | 89.3 | **79.6** | 80.2 | 72.4 | 57.3 | 61.7 | 54.8 | **71.3** | 70.5 | 71.3 | \n| SUPMER *(labeled + unlabeled)* | **89.1** | **48.2** | **85.7** | **90.8** | 79.3 | **83.4** | **73.2** | **58.8** | **63.7** | **55.3** | 70.5 | **71.5** | **72.5** |\n\n**Experimental results of domain generalization on T5-base**:\n| **Method & Domain** | B | D | E | K | A | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | \n| SUPMER *(only labeled)* | 86.4 | 84.7 | 84.8 | 90.0 | 90.7 | 87.3 |\n| SUPMER *(only unlabeled)* | 85.7 | 85.3 | 85.1 | 90.3 | 91.1 | 87.5 |\n| SUPMER *(labeled + unlabeled)* | **86.6** | **88.6** | **88.5** | **92.7** | **93.7** | **90.0** |\n\n&nbsp;\n\n### **2. The effectiveness of SUPMER compared with other labeled-data-integrated prompt initialization methods**\n\n***To further prove SUPMER's efficacy after the integration of labeled data***, We compare SUPMER with other labeled-data-integrated prompt initialization methods. We apply ***the same combined dataset of labeled and unlabeled data to other baseline methods for prompt initialization***. Following PPT [A] and MetaPT [B], we leverage pre-training and vanilla MAML to initialize soft prompts using the combined dataset of labeled and unlabeled data. We denote these baselines as **PPT *(labeled + unlabeled)*** and **MetaPT *(labeled + unlabeled)***, respectively. \n\nThe experimental results are listed in the tables below, from which we can draw the following conclusions. **First**, when combined with the results in Tables 1 and 2 of the main paper, it becomes evident that the utilization of both labeled and unlabeled data for initializing soft prompts leads to improved performance across all baseline methods that focus on prompt initialization. \n\n**Second, after the integration of labeled data, SUPMER still consistently demonstrates significantly superior performance** compared to baseline models that employ the same dataset for prompt initialization. Specifically, SUPMER exhibits an average performance improvement of 4.5 points over PPT and 2.6 points over MetaPT in few-shot learning. In terms of domain generalization, SUPMER elevates the average performance by 4.7 points compared to PPT and by 2.0 points compared to MetaPT. These results further underscore the effectiveness of SUPMER.\n\n&nbsp;\n\n**Experimental results of few-shot learning on T5-base**:\n| **Method & Dataset** | SST-2 | SST-5 | MR | CR | SUBJ | TREC | CB | RTE | QNLI | WiC | MRPC | QQP | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| FT | 83.6 | 41.2 | 81.7 | 88.3 | **80.0** | 79.8 | 71.9 | 56.9 | 62.3 | 54.6 | 70.2 | 69.5 | 70.0 |\n| PT | 71.9 | 37.3 | 73.2 | 84.4 | 61.5 | 65.3 | 58.9 | 53.2 | 55.2 | 53.1 | 66.6 | 63.0 | 62.0 |\n| PPT *(labeled + unlabeled)* | 84.7 | 45.0 | 82.4 | 87.8 | 67.2 | 77.4 | 64.3 | 55.3 | 61.6 | 53.9 | 68.9 | 67.7 | 68.0 |\n| MetaPT *(labeled + unlabeled)* | 86.1 | 46.3 | 83.7 | 89.4 | 73.8 | 80.1 | 67.2 | 57.4 | 60.0 | 54.3 | 70.1 | 69.9 | 69.9 |\n| SUPMER *(labeled + unlabeled)* | **89.1** | **48.2** | **85.7** | **90.8** | 79.3 | **83.4** | **73.2** | **58.8** | **63.7** | **55.3** | **70.5** | **71.5** | **72.5** |\n\n**Experimental results of domain generalization on T5-base**:\n| **Method & Domain** | B | D | E | K | A | *AVG* |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | \n| FT | 84.4 | 83.9 | 81.0 | 84.1 | 85.0 | 83.7 |\n| PT | 79.8 | 75.3 | 76.0 | 79.6 | 79.8 | 78.1 |\n| PPT *(labeled + unlabeled)* | 83.1 | 79.0 | 84.4 | 89.3 | 90.6 | 85.3 |\n| MetaPT *(labeled + unlabeled)* | 86.3 | 85.3 | 86.7 | 90.1 | 91.4 | 88.0 |\n| SUPMER *(labeled + unlabeled)* | **86.6** | **88.6** | **88.5** | **92.7** | **93.7** | **90.0** |\n\n&nbsp;\n\n***Note**: As our collected labeled data includes Yelp5, a sentiment analysis dataset in the domain of restaurants, we conduct the experiments of domain generalization only across domains B, D, E, K, A (without R). And then there is no overlap between our collected labeled data and the downstream testing datasets.*\n\nOverall, we sincerely thank you once again for your valuable suggestions! We will incorporate the aforementioned experiments in the next version.\n\n\n[A] Gu Y, Han X, Liu Z, et al. Ppt: Pre-trained prompt tuning for few-shot learning[J]. arXiv preprint arXiv:2109.04332, 2021.\n\n[B] Huang Y, Qian K, Yu Z. Learning a better initialization for soft prompts via meta-learning[J]. arXiv preprint arXiv:2205.12471, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Intuitive method\n2. Solid experiments\n3. Thorough ablation study that answered a lot of potential questions.", "reasons_to_reject": "I’m just a bit concerned about the setting of using only unlabeled data. I feel it is not a very real-world setting because we’ve already have a lot of NLP tasks with labelled data. Why not using them? If the authors can employ such labelled data using the proposed method, compare with proper baselines and still achieve superior performance, I will update my rating.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method for better soft prompt initialization, SUPMER.  SUPMER leverages self-supervised meta-learning with a diverse set of well-designed meta-training tasks to learn a universal prompt initialization for efficient adaptation using only unlabeled data. Extensive experiments show that SUPMER achieves better performance for different few-shot downstream tasks, and also exhibits a stronger domain generalization ability.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces SUPMER which leverages meta-learning to enhance the prompting of PLMs. SUPMER utilizes a self-supervised framework to initialize soft-prompts and incorporates meta-gradient regularization, enabling domain generalization. Compared to existing methods, SUPMER demonstrates superior performance by learning meta-prompts through anchor tasks and applying them to a series of meta-tasks. SUPMER addresses common challenges in prompting methods, such as the need for effective soft prompt initialization and mitigating overfitting in few-shot settings due to limited training examples.  


The reviewers initially raised many concerns and the authors provided extensive rebuttal responses. All reviewers engaged in post-rebuttal discussions and converged to find the work sound with moderate excitement scores and acceptable reproducibility.
Decision: Accept-Findings