Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning
Authors: XIANGYU PENG
Chen Xing
Prafulla Kumar Choubey
Chien-Sheng Wu
Caiming Xiong
Venue: ICLR 2023 poster (2023)
Keywords: prompt tuning
natural language processing
few-shot learning

Abstract: Prompt tuning approaches, which learn task-specific soft prompts for a downstream task conditioning on frozen pre-trained models, have attracted growing interest due to its parameter efficiency. With large language models and sufficient training data, prompt tuning performs comparably to full-model tuning. However, with limited training samples in few-shot settings, prompt tuning fails to match the performance of full-model fine-tuning. In this work, we focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks with abundant training samples. Recognizing the good generalization capabilities of ensemble methods in low-data regime, we first experiment and show that a simple ensemble of model predictions based on different source prompts, outperforms existing multi-prompt knowledge transfer approaches such as source prompt fusion in the few-shot setting. Motivated by this observation, we further investigate model ensembles and  propose Sample-specific Ensemble of Source Models (SESoM). SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs. Through this way, SESoM inherits the superior generalization of ensemble methods and simultaneously captures the sample-specific competence of each source prompt.  We conduct experiments across a diverse set of eight NLP tasks using models of different scales (T5-\{base, large, XL\}) and find that SESoM consistently outperforms the existing models of the same as well as larger parametric scale by a large margin.

Published as a conference paper at ICLR 2023
MODEL ENSEMBLE INSTEAD OF PROMPT FUSION: A SAMPLE-SPECIFIC KNOWLEDGE TRANSFER METHOD FOR FEW-SHOT PROMPT TUNING
Xiangyu Pengl Chen Xing Prafulla Kumar Choubey+ Chien-Sheng Wu Caiming Xiong
Georgia Institute of Technology
SSlesforce Research
xpeng62egatech.eduu {cxing, pchoubey, wu. json, www.oongeeaeeleforrrreccoooooeee
ABSTRACT
Prompt tuning approaches, which learn task-specific soft prompts for a down- stream task conditioning frozen pre-trained models, have attracted growing interest due to its parameter efficiency. With large language models and suf- ficient training data, prompt tuning performs comparably full-model tuning. However, with limited training samples in few-shot settings, prompt tuning fails to match the performance of full-model fine-tuning. In this work, we focus on improving the few-shot performance of prompt tuning by transferring knowl- edge from soft prompts of source tasks. Recognizing the good generalization capabilities of ensemble methods in low-data regime, we first experiment and show that a simple ensemble model predictions based on different source prompts, outperforms existing as source prompt fusion the few-shot setting. Motivated by this observation, we further investigate model ensembles and propose Sample-specific Ensemble of Source Models (SESoM). SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs. Through this way, SESoM inherits the superior generalization of model ensemble approaches and simultaneously captures the sample-specifs competence of each source prompt. We conduct experiments across a diverse set of eight NLP tasks using models of different scales (T5 (T5-(base, large, XL)) and find that SESoM con- sistently outperforms the existing models of the same as well as larger parametric scale by a large margin.
knowledge transfer approaches such
INTRODUCTION
Recent few years have witnessed the great success of large pre-trained language models (PLM) (Kenton & Toutanova 2019 Liu et al. 2019 Radford et al. 2019 Raffel et al.) 2020 Brown al. 2020). The size of pre- trained models which can easily go to billions of parameters (Bronn et al. 2020 Raffel al. 2020), however, hinder their real-world deployments and applications. The huge size of pre-trained language models can make model fine-tuning for downstream NLP tasks computationally expensive and memory-inefficient. To alleviate this problem, many parameter- efficient fine-tuning methods are proposed (Li Liang] [021]] Houlsby et al.) 2019 [Zhang et al., 2021 Lester et al. 202T; Liu al., 2021b). Among them, prompt tuning (Lester al.[]202] is one of the most widely adopted methods. Given downstream task, prompt tuning methods keep the entire pre-trained model frozen. Only the newly added task-specific soft prompts are updated on the training data from a target task, conditioning on the original pre-trained model. Compared to traditional fine-tuning methods that update the entire pre-trained model, prompt tuning consumes significantly less memory and less training time per iteration (Table 10 in Gu et al. (2022)).
Despite prompt tuning's advantages in practice and its continuously improved performances on vari- ous NLP tasks (Liu et al.) 2021 a Vu et al. 2022). its performances in few-shot settings where labeled training data is limited, still have large space for improvements (Gu et al., 2022). In low-data sce- narios, of the most widely applied approaches to alleviate data shortage of the target task, is to seek help from source tasks where labeled training data is abundant. Although such knowledge transfer approaches from multiple source tasks are analyzed full-model training in other domains

---

Published as a conference paper at ICLR 2023
CCenn 2021 Li et al., 2020 Sasso al. 2022 Lee et al., 2019). relevant methods for few-shot prompt-tuning still remain under explored. Therefore, in this work, we seek to find an effective strategy to use trained soft prompts from multiple source tasks to benefit few-shot prompt tuning on a new target task.
With soft prompts trained from several source tasks and full training data from target task, there are a few existing approaches one could adopt. Vu et al. (2022) finds the most suitable source soft prompt to initialize the soft prompt of the target task. Alternatively, Asai et al. (2022) directly fuses all source soft prompts together with a target task-specific prompt. Although both source soft prompt based initialization and fusion improve performance with enough training data for a target task, we empirically find them not as effective under few-shot settings. Another tempting alternative we could employ to use source prompts is model ensemble, which is known to provide good generalization and low variance (Hansen & Salamon, 1990). For instance, Dvornik et al. (019)) and Liu et al. (020) show that simple ensemble methods outperform complicated approaches in few-shot settings in the computer vision domain. Therefore, for few-shot prompt tuning, we wonder whether an ensemble of model outputs given different source prompts achieve better performance compared to existing approaches employing source prompts. If so, what is the most effective model ensemble strategy for the knowledge transfer from multiple source prompts?
To answer these questions, conduct empirical analysis and find that simple uniform logit- averaging ensemble of model predictions based on different source prompts, can already outperform existing multi-source knowledge transfer approaches for few-shot prompt tuning. Motivated by this observation, we further look into ensemble approaches and propose our solution, a sample-specific ensemble of source models (SESoM). Source models refer to the trained soft prompts of source lasks, together with the pre-trained language model that the source soft prompts are trained with. As the name suggests, SESoM learns from the few-shot target samples to adaptively decide how much each source task should contribute given different target samples. Specifically, our method trains an attention-style network to generate weights to ensemble the outputs of different source models, in order to make the prediction given each target sample. Through this way, our model is able to capture the sample-specific preferences to ensemble different source models given the few-shot labelled target data. Therefore, compared to existing knowledge transfer approaches for prompt tuning that provide a fixed knowledge transfer strategy for all target samples, SESoM is more effective due its sample-specift strategy.
We conduct experiments across six source tasks and eight target tasks on three model scales, T5- Base, T5-Large and T5-XI. Experimental results show that SESoM outperforms existing methods, such as source prompt fusion approaches and other model ensemble methods, by a large margins in every scenario tested. Moreover, we also find that SESoM can consistently achieve better per- formance compared to existing methods when the number of few-shot labeled target data increases. Even in full-data settings, SESoM outperforms existing methods although not as significantly as in few-shot settings. Finally, we find that SESoM can achieve better performance when the number of source tasks increases, even when the newly added tasks are less preferable in general for the target task. Our case study also shows that SESoM can generate different ensemble weights for different samples of one target task. The generated weights are also aligned with the sample-speciffe performance of different source models.
2 RELATED WORK
Knowledge transfer approaches in the context of prompt tuning. Since the emergence of prompt tuning methods, much recent research has focused on improving the performance of prompt-tuning methods on full-data fine-tuning, Some of them focus on transferring knowledge from other tasks which are similar to the target task, to facilitate prompt tuning of the target task. Among them, SPoT (Vu et al. (022) first learns a prompt on one or more source tasks and then uses it to initialize the prompt for target task. SPoT significantly boosts the performance of prompt-tuning across many tasks. Similarly, PPT (Gu et al. 2022) pre-trains the soft prompt of the target task with data formulated similarly with target data. These two methods provide a fixed knowledge transfer strategy for all target samples, given that they both provide initialization before few-shot prompt tuning of the target task. Different from them, our method provides sample-specific knowledge transfer from source models to each target samples, leading to better performance on few-shot fine-
2

---

Published as a conference paper at ICLR 2023
tuning. More recently, Asai et al. (022) proposes sample-specific prompt fusion method for full-data prompt tuning. In such method, the soft prompts of source tasks are fused together to construct a prompt for each target sample, instead of ensembling the source models' outputs given each target sample in SESoM.
Ensemble learning has been a popular approach to obtain low-variance and generalizable model
(Hansen & Salamon 1990). The basic ensemble techniques include voting (Hansen & Salamon,
1990), bagging (Breiman 1994), boosting (Schapire [990;; Freund & Schapire| [997) and stack-
ing (Wolpert 1992), which have been applied across many NLP problems such as model debiasing
(Elazar & Goldberg 2018 Stacey et al. 2020), cross- cooss-lingual transfer learning (Wang et al. 2021),
calibrating sequence classification and generation models (Reich et al., 22000, domain adaptation
(Kim et al. 2017). Within the paradigm of prompt-based learning, [Schick & Schütze (2021a) ex-
plores unweighted average of logits corresponding to different human-selected verbalizers for zero- shot classification. [ester et al. (202T) uses majority voting over logits from five prompts trained on the same task. They all employ identical ensemble strategy for all test samples, which could be sub-optimal. [Schick Schütze (2221a) also found that the oracle that selects the best verbal- izer for every test sample significantly outperforms the wwweiighed-averagiinn this gap by taking over the role of oracle and train a separate attention module to generate sample- specific importance weight for every source model. The sample-specific weights help measure each source model's competence in making prediction for a sample, and enable better utilization of available prompts.
model. SESoM fills
3 METHOD
3.1 PRELIMINARIES
In SESoM, given the training data for source tasks S1,..,, and pre-trained language model, we first train a soft prompt Pj (j E [1, T)) for each source task by running prompt tuning (Lester al., (02I). Following typical T5-setting, we restructure all downstream tasks to text-to-text generation format, where each label of a training sample is represented by a verbalizer (Schick & Schütze 2021c) and, optionally, a task-specific template (Schick & Schütze 2021e b. Mishra al.
2021) We represent an instance in source or target task as (X. V). where X is a sequence of token embeddings (X = [Xi,..., xi] E R 1 is the length of the input token sequence and is the embedding size of PLM), and is classification label. Then, we map the class label y to its corresponding verbalizer or the verbalizer-template sequence, and call it y Each soft prompt P, [Pi,... .... Pm] E x of is also sequence of embeddings, where is the number of soft prompt embeddings for the task.
Prompt tuming prepends a randomly initialized task-specific soft prompt P, to the input embedding sequence X, resulting in the final input [Py: X]. Then, [P,: ; X] is fed into the pre-trained model to make the prediction. The target task is modeled Pro [P X)) where the parameters of the pre-trained model (0) is frozen during the tuning and only the soft prompt P, is updated in order to maximize Pro (Y) JP;; X]). We show the process of prompt tuning in Fig. (a).
3.2 SAMPLE-SPECIFIC ENSEMBLE OF SOURCE MODELS
Given collection of source prompts PP:...;;Py Pr] from source tasks [S1; Sr] trained using prompt-tuning, and the pre-trained language model 0 that these soft prompts are conditioned on, SESoM aims to use their knowledge on a new target task under few-shot settings. We call the prompt from a source task together with the PLM as a source model, represented as [P;; 0]. In SESoM, we first train each source model [P manner. This enforces the source models to generate target task's verbalizers given the target input sample.
y; 0] with the few-shot target data in a prompt-tuning
Then taking a labeled instance (X, y) from the few-shot target task Tiarget, SESoM first feeds [P;; X] into the corresponding source model [P;; and obtain the pre-softmax logit Ir,j- It then uses the Lz, and X to compute sample-specific attention weight representing the competence of the
We detail all the verbalizers used in our experiments in Appx. B.5
3

---

Published as a conference paper at ICLR 2023
Pro-trained Model
Softmax
Y
Prestrince Model
X
X
P
Pre-trained Mode!
Pre-trained Model
Softmax
X
Pr
P2
1..
Pre-tmined Model
Soltmax
Y
Module
Softmax
Pr
1g.1 la.
O2
(a) Training of source prompts.
(b) Training of attention module G.
Figure 1: Overview of SESoM. Components with updated during training while those with are frozen. (a) describes the training of source prompts in SESoM. Given the input token em- bedding sequence X of different source tasks and a pre-trained model, prompt tuning trains task specific source prompts [P: i..;; Pr] for source tasks [Sii..:; ...: Sr] by prepending them to X as inputs and feeding to the frozen pre-trained model. (b) describes how SESoM trains the attention module G. G takes the logits [I,.1i l.rr from all T source models as input and uses x as the key to calcu- late the attention weights a, of source logits. Then source logits are weighted averaged accordingly to construct la for the final prediction.
source model Ji;; 0] for the given input X. We train an attention-style network G ($ 3.2.1 to com- pute these sample-spp -peecci weights for every source model. Finally, takes the weighted average of logits (L, [Lg, ...; 12,7] 7T where is the vocabulary size of the pre-trained model) from all source models using their attention weights to make its prediction for X.
3.2. ATTENTION MODULE G.
The goal of the attention module G is learn generate weights ensemble the source models' pre-softmax logits based on their competence representation of target sample X. Other candidate inputs of g which might benefit this sample- specific weight generation process are either source prompts pre-softmax logits of source models. We empirically find using X and source prompts as input for G are less effective compared to taking X and source model logits as G inputs under few-shot settings. We hypothesize that it is because prompts are not self-sufficient as their competency is not solely determined by them. They effective only when used along with the PLM for which it tuned. With abundant training samples, the prompt-based attention module may learn to identify source prompts that are relevant for an input sample from the target task. But with few training samples, it is unable to learn to accurately weight predictions from different source models, required to capture their effectiveness for every sample. Logits of source models, the other hand, are a function of both the prompt and the PLM and are also capable of modeling the uncertainty in predictions from different models. Consequently, we propose to use the source model logits Le along with the input X in our attention module.
target sample X. One input of G is therefore the
We first perform max-pon max-pool over the token embedding sequence & E Rd as the representation of X, Then given the representation x of the target input X, we apply a feed-forward network to project non-linearly to a new representational space,
[xi; x1]
Rxddd to obtain
h = W
u.a ?W'".
(1)
where 7(-) is a non-linear activation function, War E dRdx and Wa, E TR' xa' are trainable weights. Then we apply Layer Norm (Ba et al. 2016) to get h E Rdd the final projected representation target input X. Similarly, we project the pre-softmax logit b E Ro of each source model given X into the same space,
A (WT) Led),
where Wa Rexd and W., Rdd are trainable weights. Then, given h, and the projected representations of all source logits (hr compute the attention score an for the source
4

---

Published as a conference paper at ICLR 2023
logit 1 by
(h ha)
(3)
Rm eOh,, hb))
SESoM produces the final output logit 1. E _ by computing a linear combination of [i, L, given the computed input-logit attention,
1. = - 1e,1: LeT) =
az.to
az
(4)
f=1
Then following the prompt tuning approach in Section 3.1 we freeze all the parameters 0 in pre- trained language model and source soft prompts (Pii... P.D. We also show the forward pass of SESoM in Fig. (b). Attention module G is trained by minimizing its cross-entropy loss between softmax(1,) and the label Y. During the few-shot training, we update the attention module 9 with the few-shot labeled target samples. Through this way, G is trained to capture the sample-specific preference of different source models. At inference, we also use G to calculate the sample specific ensemble weight of all source logits, and calculate the weighted average of them as the final logit to make the prediction.
Parameter efficiency. As mentioned in Lester et al. (202) prompt tuning methods are natural fit for model ensemble approaches from the perspective of parameter efficiency, Unlike other models for which ensemble of T models leads to T times different prompt-tuning models only leads to T times more soft prompts. Because the pre-trained model that soft prompts trained to condition on, is identical for all models to be ensembled. Therefore, although SESoM is model ensemble approach, the additional model parameters intro- duced by the ensemble are only the soft prompts of 6 source tasks, i.c., T m d parameters (0.6M in our experiment). SESoM also trains one attention module which includes four projection layers and two layer norms. requires x d' + d' d' + d; + d' + 4d' parameters (~~ 0.9M in our experiment). Therefore, the total number of additional trainable model parameters in SESoM is only less than 0.5% of pre-trained T5-basy model.
model parameters, ensemble of T
EXPERIMENTS
In this section, we first describe our experimental setup. then our main empirical results compared to baselines. Finally we conduct various empirical analysis and case study to show the effectiveness of different components SESoM and reasonings behind our design choices.
4.1 EXPERIMENTAL SETUP
Few-shot prompt tuning setups and prper-parameters of SESoM. For main experiments, we follow existing work (Gu et al. 2022) to set the number of few-shot training and validation samples for every target task as 32. We evaluate all models with the full original validation set of every target task. All presented results are an average of 20 runs with different random seeds. For SESoM, randomly initialize attention module & based Section 3.2 Details of other hyper- parameters of SESoM, such as the embedding sizes of G, the optimizer and learning rate during training. etc, can be found in Appendix B.3 We use pre-trained T5- (5-(bbae, XL) (Raffel al. 2020) as language model 0.
(Rafel
Source and target tasks. Following existing work(Asai et al. 2022). we use six tasks that have been considered generally helpful to other NL.P tasks as our source tasks. They are MNLI (Williams)
al. 2018), QNLI (Demszky et al. 2018 8), QQP /Wang et al. 2018). SST2 (Socher et al.] 2013).
SQuAD RRajpurkar et al. 2016), and ReCoRD (Zhang al. 2018). Given each pre-trained model tested large we train soft prompts for these source tasks separately using prompt
tuning. The syper-parametert for training the soft prompts can be found in Appendix B.1
We then evaluate our model on the following GLUE (Wang et al., 2018) and SuperGLUE Waag et al., 2019) tasks: WNLI (Wang et al., 2018). MRPC (Dolan et al. 2005), BoolQ (Clark et al. 2019). J MultiRC (Khashabi et 2018), RTE (Giampiccolo et al. 2007). ). WiC (Pilehvar & Camacho-
Collados 2019) WSC (Levesque al. 2012), and CB (De Marneffe et al. 2019). They cover
5

---

Published as a conference paper at ICLR 2023
Table 1: Few-shot performance of all methods. All the scores are the average of 20 runs with different random seeds, with standard errors included in subscripts. The best scores are in bold. "Uni Ensemble" is the acronym for Uniform Ensemble. "MV Ensemble" stands for Majority-vote Ensemble. "FW Ensemble" stands for Fixed-weight Ensemble.
Model
Method
WNLI
MRPC
RTE
MultiRC
BoolQ
WiC
WSC
CB
Avg.
SPoT-t
54.44
53.32
51.15
50.61
56.87
54.51
17.98
43.
$1.9210
PPT
47.89
68.97
67.51
60.32
51.81
57.25
63.46
50.00
58.401.03
ATTEMPT
52.95
75.35
52.63
59.52
54.28
54.48
43.89
43.74
6..61
TS-Bas 1s-B
Uni Ensemble
56.34
75.00
57.40
65.92
70.12
62.69
45.19
57.14
61. 61.22s.87
MV Ensemble
50.24
74.65
59.87
71.39
73.97
71.59
49.71
65.82
64. 84..662.1
FW Ensemble
50.07
77.57
65.03
66.32
72.28
63.14
51.39
43.74
62.284.58
SESoM
56.34
84.34
66.66
66.91
74.80
63.68
53.56
74.02
27.542.16
SPoT-
54.93
72.46
65.14
64.83
53.66
56.26
48.76
53.70
58, 141.223
PPT
43.66
79.41
75.81
65.50
67.68
53.61
61.54
80.36
65.95
ATTEMPT
42.25
67.96
62.63
70.11
68.42
54.33
60.04
69.64
61.92v.a7
T5-Large
Uni Ensemble
56.34
86.76
67.14
74.36
78.83
64.42
60.42
58.78
38.383.33
MV Ensemble
49.58
84.20
58.19
73.60
78.58
72.67
53.09
56.60
65. 22.81
FW Ensemble
56.34
83.74
77.97
73.63
81.28
61.44
56.73
62.85
69. 249.
SESoM
56.34
87.84
82.80
74.40
81.21
67.06
63.08
84.73
74.69
SPoTt
56.34
77.65
75.16
51.07
60.97
59.53
50.58
70.89
62. 144.52
PPT
50.07
76.22
75.83
67.57
69.89
58.62
59.73
78.92
67. 10..80
ATTEMPT
49.37
62.46
84.24
71.26
60.39
42.50
45.38
80.18
61.976.65
T5-XI.
Uni Ensemble
56.34
75.73
89.16
76.56
81.07
58.77
63.40
78.57
72.455.61
MV Ensemble
41.29
73.46
84.24
75.30
77.48
64.43
49.49
82.83
68.572.87
Ensemble
56.27
85.62
89.58
76.20
82.83
66.79
57.55
73.48
73.542.41
SESoM
56.34
88.21
89.71
76.90
82.50
68.56
63.41
84.15
76.221.44
wide range of NLP tasks including question answering, sentiment analysis and textual entailment. Details of source and target tasks can be found Appendix A
Baselines. We compare SESoM not only with existing knowledge transfer approaches in the con- text of prompt tuning, but also with multiple existing ensemble strategies. The knowledge transfer methods for prompt tuning are,
SPoT-t (Vu et al., 2022): SPoT-- originally first trains soft prompt with the full training set of the target task for a few epochs, and then retrieve the source soft prompt which shares the high- est cosine similarity with this target prompt to re-initialize the target prompt. Then, it re-trains the target prompt starting from this new initialization.
To adapt SPoT-t to few-shot settings, we keep everything unchanged except that we train the target prompt with the few-shot labelled data, instead with the full target training set.
PPT et al.) 2022): PPT pre-traine trains the target prompt with additional data by pre- processeeing the additional data to the format of the target task. For fair comparison with other approaches, we use the training data of all 6 source tasks pre-train the target soft prompt. We then update the target soft prompt with the few-shot target data as in prompt tuning.
ATTEMPT(Asa.
al. 2022): ATTEMPT is sample-specific knowledge fusion method for full- data prompt tuning. In this method, the soft prompts of source tasks are fused together to construct a new prompt for each target sample, unlike SESoM's model-level ensemble of source models. To apply ATTEMPT in few-shot settings, we keep everything unchanged for ATTEMPT except that we train their model with the few-shot labelled data, instead of with the full target training set.
The ensemble baselines are,
the simplest ensemble method of the source models. Same as SESoM, we first fine-tune each source model with the target few-shot labelled data. Then we take the average pre-softmax logits of all source model's output given a target sample to make the prediction. from SESoM is that Enseobl simply averages the source logits for all target samples instead conducting a sample-specific weighted average. every individual source model to
Uniform Ensemble:
The difference of Uniform-Ensemtne
Majority-Vott Vote Ensemble: in this ensemble baseline, make predictions first. Then we take the prediction that have the most votes as the final prediction. The rest setting of Majority-Vore Vote Ensmeble is the same Uniform Ensemble.

---

Published as a conference paper at ICLR 2023
Table 2: SESoM with top source models. All results are the average of 20 runs with different random seeds. Standard errors are included in subscripts. "# s." stands for the number of source models used in SESoM.
Model
#
WNLI
MRPC
RTE
MultiRC
BoolQ
WIC
WSC
CB
Avgard
54.44
53.32
54.15
50.61
56.87
54.51
47.98
43.48
51 51.9210.61
T5-Base
3
56.97
72.59
62.49
61.59
67.48
61.43
48.08
64.11
61.847.48
5
56,41
81.91
64.01
66.08
72.96
61.54
49.04
69.46
65. 15.188.40
54.93
72.46
65.14
64.83
53.66
56.26
48.76
53.70
58. 58.7214.23
Ta-Lar
3
56.34
86.27
72.17
73.55
76.36
62.24
56.30
68.04
68.91a.93
5
56.34
87.43
77.60
74.09
80.28
66.05
60.19
80.98
72.873.41
Fixed-weiss weight Ensemble: Instead simply averaging logits of source models as Uniform- Ensemble, Fixed-weight Ensemble takes weighted average of the source logits to make pre- dictions. The voting weight of each source model is fixed for all target samples. The voting weight is calculated according to the Fl score of the source models given the few-shot target data. The better a source model performs given the labelled few-shot data, larger voting weight it would have for all target samples.
More details about the training of all baseline methods above can be found in Appendix B.2
4.2
MAIN RESULTS
The few-shot performance of SESoM and other methods are shown in Table 1 First, on every pre- trained model (T5-(base, large, SESoM outperforms all baselines on the average score by large margin. SESoM also outperforms baselines on a definite majority of single target tasks on pre- trained models with different sizes. Specifically, compared to approaches that provide a fixed knowl- edge transfer strategy for all target samples (SPoT-r, PPT, Uni Ensemble, FW Ensemble), SESOM's superior performance indicates the effectiveness of its sample-specific strategy. Compared to ex- isting approaches that conduct sample-specift soft prompt fusion (ATTEMPT), SESoM's superior performance indicates the effectiveness of its model ensemble strategy in few-shot settings. More- over, if we compare SESoM on T5-Base with non-ensemble baselines (SPoT-r, PPT, ATTEMPT) T5-Large, we can see that SESoM on T5-Base even outperforms these baselines with a larger pre- trained model. Similarly, SESoM T5-Large also outperforms non-ensemble baselines on T5-XL. This indicates that SESoM further improves the parameter efficiency of prompt tuning in few-shot settings.
Second, if we compare the performances of ensemble methods (Uni Ensemble, FW Ensemble and our proposed method SESoM) versus the rest methods, ensemble methods have superior perfor- mance against the rest methods in the few-shot setting. The bigger the pre-trained model is, more apparent the performance advantages of ensemble methods are in the few-shot setting. It is likely due to the widely observed strengths of ensemble methods that provide better generalization and lower variance during inference, as illustrated in Section D as our motivation. Finally, compared with all methods, ATTEMPT seems to have the most unsatisfactory results in few-shot settings. AT- TEMPT uses the fhw-shot target data to train a network to output weights to fuse source soft prompts for each target sample. Although it achieves great performances in full-data fine-tuning, few-shot training is not an easy scenario for it due to the overfitting caused by limited training data. While our proposed SESoM takes the logits of source models input and outputs weights for model ensemble during few-shot training, which takes advantage of the superior generalization and robustness of model ensemble approaches.
4.3 EMPIRICAL ANALYSIS
Would more "ordinary" source tasks help SESoM? In our main experiments, we use 6 source tasks as mentioned earlier, regardless of their actual transferability given the target task. However, if a source task is too different from the target task, is possible that this source task wouldn' 't effect the target task positively in general. In order to investigate how SESoM performs when less preferable source tasks are added, we conduct experiments as follows. First, given each target task, we pick top top and top most generally helpful source tasks for this target task, following SPoT-t (Vul
7

---

Published as a conference paper at ICLR 2023
Table Results on target tasks with different number of few-shot training labels. All the scores are the average of 5 runs, with standard errors included in subscripts.
Data Size
Method
WNLI
MRPC
RTE
MultiRC
BoolQ
WIC
WSC
CB
Ayg
PPT
48.51
68.53
69.22
60.54
62.04
53.59
62.00
54.29
1..841.83
ATTEMPT
52.68
76.76
55.88
58.87
62.35
57.74
48.46
50.36
57.894.00
64
Ensemble
53.80
77.87
65.68
64.78
72.65
62.66
50.96
58.21
63.334.40
SESoM
$7.69
84.56
67.33
69.18
76.09
63.71
60.92
77.14
69.581.42
PPT
50.14
68.68
66.75
62.27
62.16
51.25
61.81
75.83
62.362.03
ATTEMPT
54.37
77.60
60.29
58.31
66.47
53.89
51.35
69.29
61.445.20
128
Ensemble
53.55
79.80
67.36
59.78
62.65
56.08
55.58
74.29
63. 63..644.49
SESoM
58.10
84.61
68.09
68.87
75.93
63.73
61.54
82.86
70.461.06
PPT
50.70
87.50
80.87
71.39
79.57
66.61
62.78
78.57
72.24o.o1
ATTEMPT
55.56
88.14
75.31
70.86
79.37
68.50
59.23
75.36
71.541.69
Full
Ensemble
54.93
89 $9.46
79.06
68.59
75.82
66.46
63.46
76.79
71.82u.84
SESoM
57.75
89.92
79.42
72.14
80.79
67.5
63.50
86.43
74.69
Table 4: Results of different design choices pre-trained T5-Base model. All the scores are the average of runs, with standard errors included in subseripts. "Ensemble acc SP stands for Ensemble acc. source prompts. "PLG" stands for Pseudo Label Generation. Training details can be found in Appx. B.4
Method
WNLI
MRPC
RTE
MultiRC
BoolQ
WiC
WSC
CB
Avg
Ensemble acc SP
52.96
79.69
64.15
63.92
74.46
60.97
52.34
71.25
64.964.22
PLG
50.07
77.57
65.03
65.32
72.98
63.14
51.39
43.74
61. 15a.37
SESoM
56.34
84.34
66.66
66.91
74.80
63.68
53.56
74.02
67.54..16
et al. 2022). Then apply SESoM with the top 1, top 3 and top 5 source tasks separately, and compare with SESoM's main results with 6 default source tasks. Results are shown in Table From Table 2 we can see that SESoM can achieve better performances when the number of source tasks increases, although the newly added source tasks might be less preferable to the target task in general. It indicates that for the target samples on which the generally less favourable source tasks can actually perform well, SESoM is able to set proper sample-specifee weights and lean more on these source models for such specific target samples, which implies the effectiveness of SESoM sample-specific strategy.
The effect of the number of shots. In main experiments, the number of few-shot labeled data we use for each target task is 32 following existing work. We wonder how SESoM would perform when we increase the number of shots compared other methods. Therefore we have tried to set the number of shots as 64, and 128 and full. Results are presented in Table We can find that SESoM can consistently achieve better performance compared to existing methods when the number of few-shot labeled target data increases. It suggests that SESoM can be applied to wide range of few-shot settings. Even in full-data settings, SESoM outperforms existing methods although not as significantly in few-shot settings.
Verification of other design choices. Before the current SESoM, we have also tried some related design choices that outperform existing baselines to some extent, but turned out to be less effective than SESOM. For example, we have tried different inputs for attention module G. Instead of using source logits as our attention module's 's other input given input sample X in SESoM, we have tried using the source prompts as inputs, i.e., replacing 1, in Eq.2 with P,. Other components of this method stay the same as SESoM. We refer to this method as "Ensemble acc. source prompts". We have also tried directly generating pseudo labels from source models to train the target soft prompt. Specifically, we first prompt tune source prompts on the target data sets with the few-shot target training data. Then we use these prompt-tuned source prompts with the pre-trained language model, to generate pseudo labels for the entire target datasets. The final pseudo label of each target sample is decided by majority voting of the 6 source models. These pseudo labelled samples are used to train the soft prompt of the target task before few-shot prompt tuning with the few-shot target training data. We refer to this method as "Pseudo Label Generation', Results of these methods are shown in Table We can see that they achieve relatively similar performances with FW Ensemble method in Table T while all pederrm SESOM.
8

---

Published as a conference paper at ICLR 2023
4.4 CASE STUDY
In this section, we conduct case study to verify the effectiveness of SESoM's sample-specifie model ensemble strategy. First, we would like to see whether it is true that given one target task, different samples of this target task require different source models. Because it is the basic assumption under which we propose SESoM's sample-specine from target task MRPC. Under each sample, in the row of "preds from individual source", we show each individual source model's prediction given this sample. We can see that for Example #1, QQP and QNLI makes the correct prediction while for Example #2, MNLI, SST2 and QNLI makes the correct prediction. This samplc-specific performance of different source models on target-task samples not rare for all target tasks. More examples of other target tasks can be found in Appx. Moreover, the majority or at least half of source models make wrong predictions for each example. indicates that universal model ensemble or majority vote model ensemble approaches would fail on these examples.
ensemble strategy. In Table 5, we show two samples
Second, we would like to see whether SESoM's sample-specific contribution weights of different source models generated by its attention module G, are aligned with the performances of different source models. Table 5 under each example, we show the contribution weights from SESoM for ensembling each source models in the row of "weights from SESoM". We can see that SESoM generally generates higher weights for source models that make the correct predictions on each example and the weight distribution is different for different examples. More examples of other target tasks can be found in Appx. E. It indicates that SESoM can generate sample-specific weights for model ensemble according to the actual sample-specific performance of different source models.
Table 5: Case study from MRPC target task. "Label" (yellow cells) is ground truth label and "Pred" (yellow cells) is the predicted label obtained by SESoM with the weights (orange cells) shown in the table. "Preds from individual source" (pink cells) shows predictions of each source model.
Example # 1
[MRPC]
s_1:
Unable to find home for him, judge told mental health authorities they needed to find supervised housing and treatment for DeVries somewhere in California. The judge had told state Department Mental Health to find supervised housing and treatment for Deies somewhere in California.
S.2:
Pred
(Equivalent)
Source Models
Label
(Equivalent)
MNLI
SST2
QNLI
QQP
SQuAD
ReCoRD
Preds from individual source
OX
0X
17
TV
0X
0X
Weights from SESoM
0.0120
0.0708
0.1346
0.5857
0.1266
0.0702
Example #
[MRPC]
s_1:
This integrates with Rational PurifyPlus and allows developers to work in supported versions of Java, Visual C # and Visual Basic.NET.
S.2:
IBM said the Rational products were also integrated with Rational PurifyPlus, which allows developers to work in Java, Visual # and VisualBasic.Ner
Pred
(Equivalent)
Source Models
Label
(Equivalent)
MNLI
SST2
QNLI
QQP
SQuAD
ReCoRD
Preds from individual source
17
17
1V
0X
0
Weights from SESoM
0.2381
0.2055
0.2486
0.2240
0.0219
0.0620
5 CLOSING REMARKS
In this paper we explored the potentials of prompt tuning methods in few-shot settings. We have found in our exploration that by properly transferring knowledge from trained soft prompts of source tasks, prompt tuning's few-shot performance can be significantly improved. Specifically, our pro- posed method, sample-specific ensemble of source models (SESoM) outperforms existing methods by a large margin in every tested few-shot scenario. SESoM adjusts the contribution of each source model for each target sample separately when ensembling source model outputs. Our empirical analysis suggests that the two key components of SESoM, the model-level ensemble instead of prompt-level fusion, and the sample-specific strategy for model ensemble, are both critical to its perior performance. SESoM also consistently achieves superior performance given larger numbers of labelled data and larger numbers of source tasks.

---

Published as a conference paper at ICLR 2023
REFERENCES
Akari Asai, Mohammadreza Salchi, Matthew E Peters, and Hannanch Hajishirzi. Attentional mix- tures of soft prompt tuning for parameter-effiient multi-task knowledge sharing. arXiv preprint, 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450 2016.
Leo Breiman. Bagging predictors, 1994.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yifei Chen. A transfer learning model with multi-source domains for biomedical event trigger extraction. BMC genomics, 22(1):1-18, 2021. 2021.
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Karie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. proceedings of Sinn und Bedeutung. volume 23, pp. 107-124, 2019.
Dorottya Demszky, Kelvin Guu, and Percy Liang. Transforming question answering datasets into natural language inference datasets. arXiv preprint arXiv:1809.0292. 2018.
Bill Dolan, Chris Brockett, and Chris Quirk. Microsoft research paraphrase corpus. Retrieved March, 29(2008):63,, 2005.
Nikita Dvoriik, Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for sew- classification. Proceedings of the IEEE/CVF international conference on computer vision, pp. 3723-3731, 2019.
Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 11-21, Brussels, Belgium, Oetober-November 2018. Association for Computational Linguistics. doi: 18.18653/v (01/53/v/DD1.... 1002.
Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119-139, 1997. ISSN 0022-0000. doi: https://ooi.org/101000/66666
1997.1504. URL /httt::ttt. wwwww sci sciencedi rect. com/coo/sssooooooo /ttp:///cccccaaaaaaa.caaa///////
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9, 2007.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. PPT: Pre-trained prompt tuning for few- shot learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8410-8423, Dublin, Ireland, May 2022. Association for Computational Linguistics.
L.K. Hansen and Salamon. Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10):993-100, 1990. doi: 10.1109/34.8871.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
10

---

Published as a conference paper at ICLR 2023
Jacob Devlin Ming-Wai Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLI, pp. 4171- 4186, 2019.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Ypadhyay, and Dan Roth. Look- ing beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 252-262, 2018.
Young-Bund Bum Kim, Karl Stratos, and Dongchan Kim. Domain attention with an ensemble of experts. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 643-653, Vancouver, Canada, July 2017. Association for Com- putational Linguistics. doi: 10. 1/111//611111177100000 URL https: ant ogy.oooooooooo. P P17-1060
Joshua Lee, Prasanna Sattigeri, and Gregory Wornell. Learning new tricks from old dogs: Multi- source transfer learning from pre-trained networks. Advances in neural information processing systems, 32, 2019.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3045-3059 Online and Punta Cana, Dominican Republic, November 2021. Associ- ation for Computational Linguistics.
Hector Levesque, Emest Davis, and Leora Morgenstemn. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning, 2012.
Junnan Li, Ziwei Xu, Wong Yongkang, Qi Zhao, and Mohan Kankanhalli. Gradmix: Multi-source transfer across domains and tasks. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pp. 3029-3027, 2020.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Join! Conference on Natural Language Processing (Volume 1: Long Papers), pp. 0992-4597, 2021.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 1rXiv:2110.07600 2021a.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103.1l 10385, 2021b.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. An ensemble of epoch-wise empirical bayes for few- shot learning. In European Conference on Computer Vision, pp. 404 421. Springer, 2020.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 11692, 2019.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to gptk's language. arXiv preprint arXiv:2109.008300 2021.
Mohammad Taher Pilehvar and Jose Camacho-Collaan
WiC: the word-in-conterr dataset for eval- uating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Pupers), pp. 1267-1273, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):, 2019.
11

---

Published as a conference paper at ICLR 2023
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21((40):1-67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguisties.
Steven Reich, David Mueller, and Nicholas Andrews. Ensemble Distillation for Structured Pre- diction: Calibrated, Accurate, Fast Choose Three In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5583-5595, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. URL
org.ooggoooog2000 emn p-annlppmannm 450
Remo Sasso, Matthia Sabatelli, and Marco A Wiering. Multi-source transfer learning for deep model-based reinforcement learning. arXiv preprint arXiv:2205 144100 2022.
Robert E. Schapire, The strength of weak learnability. Mach. Learn. 5(2):197-277, jul 1990. ISSN 0885-6125, doi: 10.1023/A:10226448007661 URL /ttp
1022648800760
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255 269, Online, April 2021a. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269, 2021b.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 2339-2352, 2021c.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. 163 1631-1642, 2013.
Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, and Tim Rocktäschel. Avoid- ing the Hypothesis-O -naih Bias in Natural Language Inference via Ensemble Adversarial Training- In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 8281 12221, Online, November 2020. Association for Computational Linguis- tics. doi: 10. 10.18653/v 000655//112220 1ttp://///22eeeem http:///0...mmmmmmmm.mmoo URL http:pp:://.lllaalllllg...../// org/2020
emn wammlppmmannn ma mwi.aiinn..
665
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pp. 5039- 5059, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Lin- guistics.
Alex Wang, Yada Pruksachatkun. Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpost language understanding systems. Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2023
Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, and Graham Neubig. Efficient test time adapter sembling for low-resource language varieties. In Findings of the Association for Computational Linguistics: EMNLP 2021. pp. 730-737, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.1666/1122021.nnniinn findinngs-emll..63.. URL
htttps://acllatthooooo.... www.
ndi ngg- 1p. 63
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
David Wolpert. Stacked generalization. Neural Nenworks, 5(2):241-259, 1992. ISSN 0893-6080. doi: http://boooggoo00000000000000000
URL https: www.
Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng. Zhen Bi, Chuanqi Tan, Fei Huang, and
Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. In International Conference on Learning Representations, 2021.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv: XX1:11810.12885, 2018.
13

---

Published as a conference paper at ICLR 2023
A DATASETS
A.1 SOURCE TASKS
Following existing work(Asai al, 2022 use six tasks that have been considered generally helpful to other NLP tasks as our source tasks. They are MNLI (Williams et al. 2018), QNLI
(Demszky et al. 2018), QQP (Wang et 2018), SST2 (Socher et al. 2013), SQuAD (Rajpurkar
al. 2016), and ReCoRD (Zhang et al. 2018 1018,, Details are shown in T
Table 6: The tasks included in source models. NLI is natural language inference, and QA is question answering.
Dataset
Train
Category
Task
Domain
Metric
MNLI
393k
GLUE
NLI
misc.
accuracy
SST-2
67k
GLUE
Sentiment analysis
Movie Reviews
accuracy
QQP
364k
GLUE
Paraphrase
Social QA questions
accuracy & Fl accuracy F1 EM F1 & EM
QNLI
105k
GLUE
QA/NLI
Wikipedia
SQuAD
MRQA 2019
QA
Wikipedia
ReCoRD
101k
SuperGLUE
QA
News (CNN, Daily Mail)
A.2
TARGET TASKS
We evaluate our model on the following GLUE (Wang et 2018) and SuperGLUE (Wang al.,
2019) tasks: WNLI (Wang et al. 2018), MRPC (Dolan et al. (005), BoolQ (Clark et al. 2019), Mul- tRCC (Kaashabi 2018)) RTE (Giampiccolo al. (007). WiC (Pilehvar & Camacho-Collados, 2019) WSC (Levesque et al. 2012). and CB (De Marneffe et al. (019). Details are shown in Table
Table 7: The tasks included in source models. NLI is natural language inference, and QA is question answering.
Dataset
Train
Category
Task
Domain
Metric
WNLI
634
GLUE
coreference/NLJ
Fiction books
accuracy
MRPC
3.7k
GLUE
Paraphrase
News
accuracy & Fl
RTE
2.5k
SuperGLUE
NLI
News, Wikipedia
accuracy
MultiRC
5.1k
SuperGLUE
QA
various
Fl & EM accuracy
BoolQ
9.4k
SuperGLUE
QA
Google queries, Wikipedia WordNet, VerbNet, Wiktionary fiction books
WIC
6k
SuperGLUE
WSD
accuracy
WSC
554
SuperGLUE
coreference
accuracy
CB
250
SuperGLUE
NLI
various
accuracy & Fl
14

---

Published as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
B.1 SOURCE PROMPTS TRAINING
We follow Asai et al. 2022 to train the source prompts, Pi:...: Psl with corresponding source al. 2021) respectively on MNLI [Williams et al., 2018). QNLI (Demszky et al. 2018). QQP (Wagg et al. 2018). SST2 (Socher al. 2013). SQuAD Raipurkar et al.] 2016). and ReCoRD Zhang al., 2018). Details can be found in Table 6 Soft tokens size (m is 100 for all pre-trained models and embedding dimensions are
task data. These source prompls are prompt-tuned
(d) are 768. 1024 and 1024 respectively for T5-(base, large, 3b). The rest
shown in Table
Table Parameters of baseline and SESoM training. "Batch size" is different for T5-base, T5-large and T5-XL.
Parameters
Source Prompt
SPoT_
PPT
ATTEMPT
SESoM
max source length learning rate train epoch
512
256
256
256
256
3e-1
3e-1
3e-1
3e-1
3e-1
10
30
30
20
20
optimizer
Adam W
A 32/16/4
AdamW 32/16/4
Adam 32/16/8
Adama
batch size
32/16/4
32/16/4
B.2 BASELINES
B.2.1
SPOT-TARGET (SPOT-T)
We first train a randomly initialized soft prompt on target task, which is referred as target prompt, using the target few-shot training data. Then calculate the cosine similarity between the target prompt P. and the source prompts, [P;....; Pi; Po.. The source prompt P.. which shares the highest similarity with Pt, is used to initialize the soft prompt training. We then prompt-tune this initialized soft prompt P. the same target few-shot training data. Finally, the source model [Pr; is tested with verbalizer (Appx. B.5) on the original validation set of target task. Hyper-paramette details can be found in Table
B.2.2 PPT
For fair comparison, we replicate Gu et al. 2022) only with pre-training a soft prompt on MNLI, QNLI, QQP. SST2, SQuAD, and ReCoRD which used for training source prompts. Following Gu et al. (2022), we firstly unify six source tasks to single format-multiple mutipp---hiil classification. For example, CB is formatted as,
#premise.
#hypothesis. s. A. yes. B. maybe. C. no. The correct one is
Classification text labels of CB is A(no), B(maybe) and C(yes). We then prompt-tune randomly initialized soft prompt ppt on these unified datasets. We use the same hyperparametes as training SPoTH. Then for each target, Ppm ppt tuned on the same Dtrain and Daev get target-specific soft prompt P pptt. Finally, the source model [P ppti 0 is tested with verbalizer (Appx. B.5 applied on the same Drest- See training details in Table 8
B.2.3 ATTEMPT
Following Asai et al. 2002 Pr:...: Pal (Appx. B.1 are used as source prompts. ATTEMPT first initialize attention module between source prompts and inputs. ATTEMPT interpolates the source prompts and newly initialized target-task-specinn prompt P ttrgett given attention scores generated by the attention module to produce target soft prompt P,, then [P; (] is used to produce predictions. We use the same training parameters used in Asai et al. (2022). which is shown in Table g
15

---

Published as a conference paper at ICLR 2023
B.3 SESOM
The hyper-parameters of G are shown in Table
Table 9:
Dimensions of g SESoM used in Table
Model
Method
WNLI
MRPC
RTE
MultiRC
BoolQ
WiC
WSC
CB
d,
32
32
100
100
64
100
64
128
32
32
100
100
64
100
100
128
TS-base
32
128
100
100
100
100
128
I's dropout %
50
0
d'
100
64
64
32
100
100
32
64
100
64
256
64
64
32
64
TS-large
100
64
512
100
100
256
G's dropout %
50
0
50
0
50
d'
100
32
32
100
64
64
64
64
100
64
128
64
TS-XI.
100
64
64
100
256
512
64
512
G's dropout %
50
0
50
0
0
0
50
0
B.4
ABLATION STUDY
Before SESoM, we also tried some related methods that turned be less effective.
Ensemble acc SP. Firstly, tried consider use aoput-prompt attention, where max-pool em- bedding input & is used as key and source prompts [Pii..;; Pa] are values to calculate attention score to computing linear combination of pre-softmax logits. The architecture design of the attention module is the same with SESoM. The dimensions of the attention module after tuning shown in Table 10
Table 10: Hyper-parameters of & of "Ensemble acc SP" used in Table
Model
Method
WNLI
MRPC
RTE
MultiRC
BoolQ
WIC
WSC
CB
32
32
128
128
64
100
100
32
32
128
128
64
100
100
T5-base
d'
32
64
128
128
128
100
64
100
G's dropout
50
0
0
0
0
Majority vote. Secondly, we consider whether predictions from each source model [P ..., [Ps: f8] can provide enough information for final prediction. We get the pre-softmax logits L, 1 1; ... 62,6 then conduct verbalizer mapping to obtain L, [Le, ...; le, Then for
each input, we obtain 6 predicted labels 4,...., Is from Lg- The majority vote l,.,, Ig is used as final prediction. For tie condition, uniformly sample of them prediction.
B.5 VERBALIZER DETAILS
Source prompts are trained on six source datasets in text-to-text format. Different source datasets have different verbalizers. For example, CB (De Marneffe et al., 2019) is classified as "entailment", "neutral" and "contradiction"": and MRPC (Dolan et al. 2005) is classified as "equivalent" and "not equivalent". We show the verbalizers used for target task in Table IT For rare cases in which the source model after few-shot prompt tuning is not able transfer all the verbalizers from source dataset to target dataset, for all baselines and our method, we conduct a verbalizer mapping for pre-softmax logits.
Based on original verbalizers in Table II we develop mapping dictionary My to map text to the token for target dataset T. For example, map "True" to "1". and "False" to "0" for the prediction of source models with source prompts trained on Squad(Rajpurkar et al. 2016) and Record(Zhang et al. 2018). Formally, for the ith source model, the pre-softmax logit
[ : [g,] obtained
16

---

Published as a conference paper at ICLR 2023
from source model [P;: f0] target dataset T is transformed to i, by Algorithm where is the vocabulary size of tokens. More specifically, all the logits in each pre-softmax logit of 1, will be swapped to the same position. For example, in source model of ReCoRD, the pre-softmax logit of token 209 in I is replaced with the maximum of the pre-sofimax logit of token 209 and 10998 in I,, because Mr Mr[10998] = 209, where 10998 is the first token representing "True" and 209 is the first token representing "1".
Table 11: Mapping dictionary details.
Label
Target
#
Text
0
not ent.entailment
WNLI
1
entailment
0
not_equivalent
MRPC
equivalent
0
entailment
RTE
1
not.entailment
0
False
BoolQ
1
True
0
False
MultiCC
1
True
0
False
WiC
True
0
False
WSC
True
0
entailment
CB
1
contradiction
2
neutral
Algorithm 1: Mapping Algorithm
Data: MT.
Result: 1, 1, +- 1;
for k E MT do
L + min(k, (M))
i, M, < max(Ie, Mm)):
end
B.6 INFERENCE TIME
Compared to non-ensemble baselines the pre-trained language model, SESoM indeed in- creases inference time due to the multiple forward passes of source models. In order to achieve a fair comparison, we compare SESoM to prompt fusion model (ATTEMPT(Asai et al. 20022). which uses the same number of source tasks as SESoM. As a samplc-specific knowledge fusion method, ATTEMPT fuses N soft prompts of source tasks together construct a new prompt for each target sample and foward it together with the target sample to the pre-trained language model, which is one forward pass per sample for inference. While SESoM runs N forward passes for each sample to obtain the source logits.
Although SESoM requires more forward passes given one target sample, requires a smaller pre- trained language model (shorter inference time per forward pass) to achieve the same for even better performance. SESoM with a small pre-trained model (shorter inference time per forward pass) can outperform non-ensemble baseline with much larger pre-trained model (longer inference time per forward pass). For example, SESoM with T5- outperforms ATTEMPT with T5-XL on few-shot classification (please Table 1). The inference time of SESoM with T5-base is 31.38s while the inference of ATTEMPT with T5-XL takes 18s, which shows that the inference time of SESoM with T5-base is shorter. The inference time is the average of 20 runs with different random seeds
17

---

Published as a conference paper at ICLR 2023
all 8 target tasks. It indicates that SESoM can achieve better performance with shorter inference time.
18

---

Published as a conference paper at ICLR 2023
C ANALYSIS ON ATTENTION
Figure 2 shows the average attention weights of g between source and target tasks on Drest. The av- erage weights are produced by the SESoM model with T5-base and pyper--arameters are shown in Appx. B.3 g has higher attention score between more sumilar target and source tasks. gives sig- nificantly higher attentions to QQP for MRPC and WSC or MNLJ for RTE and WNL.I MRPC and QQP are both tasks to determine whether pair of questions/senttenes are semantically equivalent. WSC is coreference resolution task to determine the correct referent the pronoun from the pro- vided noun, which shares similarities with QQP as well. Besides, WNLI, RTE,and MNLI are all tex- tual entailment challenges, which share similarities between each other.
00.
MRPC
0.4
MutRRC
BoolQ
0.2
WSC
0.1
CB
S0uAD ReColt
We also find the attention weights and Fl scores are highly positive correlated on MRPC, RTE, Mul- tiRC, WSC and CB tasks. Pearson product-momen correlation coefficients between attention weights and scores are shown as follows,
Figure 2: Average attention weights of G.
Target Task Corr. coeff.
WNLI -0.29
MRPC
RTE
MultiRC
BoolQ
WiC
WSC
CB 0.45
0.42
0.44
0.46
0.03
0.75
0.63
High Fl scores of source model on target task also indicate underlying task similarities. Fixed- Weight ensemble model takes advantage of these scores to get a weighted average of source pre-max logits to make predictions. But showed in Table and Section 4.2 these weights are not enough to obatin the best performance on target tasks.
19

---

Published as a conference paper at ICLR 2023
D MORE RESULTS
D.1 EFFECT OF THE NUMBER OF SHOTS
Besides our main experiments (Table B also reported results on target tasks with training
samples in Table 12 The performance of SESoM is better than all the baselines with 32 training samples, which are shown in Table
Table 12: Results on target tasks with samples of few-shot training labels. All the scores are average of 10 runs, with standard errors included in subscripts.
Data Size
Method
WNLI
MRPC
RTE
MultiRC
BoolQ
WIC
WSC
CB
Avg
FW Ensemble
53.73
71.10
58.70
63.17
72.44
63.17
51.44
61.52
61.90s.97
SESoM
58.31
78.01
63.65
66.91
71.27
62.73
54.97
66.61
65.31s.14
D.2 EFFECT OF PROMPT-TUNING SOURCE MODELS
In main experiments, we train each source model with the few-shot target data in a prompt-tuning manner. We wonder how SESoM would perform when we skip this step. Therefore, we tried to perform SESoM on non-tuned source models. Results are presented in Table 13 We can find that with only 8 and 32 samples few-shot training samples, SESoM with prompt-tuned source models cannot achieve significantly better performance than SESoM with non-tuned source models. However, when the number of few-shot labeled target data increase, SESoM with prompt-tuned source models achieve better performance, because the source models themselves have a better performance on target datasets.
Table 13: Results on target tasks with different numbers of few-shot training samples. "SESoM w/ PT" represents SESoM with prompt-tuned source models: "SESoM w/o PT" is SESoM with non- tuned source models. All the scores of data size of 8, 32 and 128 are the average of 10, 10 and 5 runs, with standard errors included in subscripts.
Data Size
Method
WNLI
MRPC
RTE
MultiRC
BoolQ
WiC
WSC
CB
Avg
SESoM w/o PT
58.24
76.34
63.16
66.45
69.50
63.08
54.97
67.41
64.89a.4a
SESoM w. PT
58.31
78.01
63.65
66.91
71.27
62.73
54.97
66.61
65.31a.14
SESoM w/o PT
55.21
82.57
65.20
63.53
74.89
62.74
53.37
75.18
66.59..s1
32
SESoM PT
58.31
84.22
66.86
66.72
72.87
64.20
53.56
72.86
67.452.41
SESoM w/o PT
58.24
83.61
65.89
63.89
75.74
63.52
55.29
74.29
67.572.67
128
SESoM PT
58.10
84.61
68.09
68.87
75.93
63.73
61.54
82.86
70.461.08
D.3 EFFECT OF ENSEMBLE
We wonder whether our ensemble method is better than choosing single most appropriate source model's prediction as final prediction. We implement hard variant, where after attention weights are computed, we set the argmax position to and all remaining entries to Os. Results are presented in Table [4] It shows that our ensemble attention is useful compared to selecting the single most appropriate source model.
Table 14: Results on target tasks with 32 samples of few-shot training labels with different ensemble. All the scores are the average 20 runs, with standard errors included in subscripts.
Method
WNLI
MRPC
RTE
MultiRC
BoolQ
WiC
WSC
CB
Avg
Hard Variant
48.52
73.00
54.49
53.58
60.53
56.17
49.18
56.52
56,50 66
SESoM
56.34
84.34
66.66
66.91
74.80
63.68
53.56
74.02
67.544.1u
20

---

Published as a conference paper at ICLR 2023
E CASE STUDY
Table 15: Case study from WSC and MultiRC target task. "Label" (yellow cells) is ground truth
label and "Pred" (yellow cells) is the predicted lahel obtained by SESoM with the weights (orange cells) shown in the table. "Preds from individual source" (pink cells) shows predictions of each source model PPi, 0) ..., Po,c 0] obtained by the pre-softmax logits [i.,i;...
Example #
[SCJ]
text: Alice tried frantically to stop her daughter from barking at the party, leaving us wonder why she was hehaving so strangely.
1: Alice
2: She
Pred
0 (False)
Source Models
Label
(False)
MNLI
SST2
QNLI
QQP
SQUAD
ReCoRD
Preds from individual source
XX
0/
XX
0
X
Weights from SESoM
0.0480
0.3348
0.0912
0.3270
0.0984
0.1005
Example #
[SC]
text: Well satisfied with his purchases and feeling very elegant indeed, Babar goes to the photographer to have his picture taken.
s_1: photographer
8.2: his
Pred
0 (False)
Source Models
Label
0 (False)
MNLI
SST2
QNLI
QQP
SQUAD
ReCoRD
Preds from individual source
X
X
X
07
1X
X
Weights from SESoM
0.0732
0.0285
0.0972
0,5973
0.1048
0.0991
Example #
[MultiC]]
question: Sarah introduces him to three other guests. Name them.
paragraph: The bar was manned by an expensive humanoid robot. It turned toward Sarah's 's wave and acknowledged her with nod, moments later setting fluted glass of sparkling liquid in front of her. I marveled the robot's smoothness and coordination. Clearly, was high-end model. Sarah transferred the glass to my free hand and pulled me away from the bar for more introductions, with Alexis trailing after us. spent the evening listening, mostly. Listening and stuffing my face with all the bits of fine food provided. No one minded: Sarah's 's inner circle was content to fill circle of couches with plenty of chatter. Ray, plump man who grey where he wasn't bald. Zheng, short and dark lean, with a very intense gaze. He made me little uncomfortable.
Kishori, petite, her hair strung out a series of braids that reached nearly to her waist.
answer: Ray, Zheng, and Kishori
Pred
(True)
Source Models
Label
(True)
MNLI
SST2
QNLI
QQP
SQUAD
ReCoRD 1/
Preds from individual source
1
0x
x
x
1/
Weights from SESoM
0.2094
0.1028
0.99
0,1019
0,2377
0.2383
Example # 4
[ulttiRC
quest lon: What were Zheng's 's traits?
paragraph: Same with Example # 3.
answer: Humanoid
Pred
(False)
Source Models
Label
(False)
MNLI
SST2
QNLI
QQP
SQUAD
ReCoRD
Preds from individual source
0 0
0/
1
1
1
X
Weights from SESoM
0.2347
0.1932
0.1891
0.1145
0.1386
0.1300
We show more examples of attention weights on samples from different target task in Table L5 Table and
and Table 17 They show the superior power of SESoM to give higher attention to those pre-softmax logits, which prefers correct label.
21

---

Published as a conference paper at ICLR 2023
Table 16: Case study from BoolQ and WiC target task. "Label" (yellow cells) is ground truth label and "Pred" (yellow cells) is the predicted label obtained by SESoM with the weights (orange cells) shown in the table. "Preds from individual source" (pink cells) shows predictions of each source model [P p.:: O), ..... P 0) obtained by the pre-softmax logits ....:.......))
Example #
[BooQQ]
quest i on: Did the stock market crash T929 caused the great depression?
passage: Wall Street Crash of 1929 1199-Thaa The Wall Street Crash of 1929, also known as Black Tuesday,
the Great Crash, or the Stock Market Crash of 1929, began October 24, 1929.
and the most devastating stock market crash in the history of the United States, when taking into consideration the full extent and duration of after effects. The crash.
which followed the London Stock Exchange's ' crash of September, signalled the beginning of the y2ar Great Depression that affected all Western industrialized countries.
Pred 1 (True)
Source Models
Label
(True)
MNLI
SST2
QNLI
QQP
SQUAD ReCoRD
Preds from individual source
1
OX
0
Weights from SESoM
0.0194
0.0036
0.0073
0.0627
0.4285
0.4785
Example # 6
[BoolQ]
question: Do house and cuddy get back together in scason 8?
passage: Lisa Cuddy The relationship between House and Cuddy is known by the portmanteau term, Huddy. Cuddy has what USA Today's 's Peter Johnson terms cat-and-mouse relationship with House. Edelstein has described as "a really complicated, adult relationship", explaining: "These are people who have very full lives and lots of responsibilities that perhaps conflict with their feelings for each other." The actress "would love for them to have (romantic) relationship. because could be complicated the rest of their relationship". however, she is unsure how it would affect the dynamics of the show. Jacobs commented at the end of the show's 's third scason: "I can't 't see them pairing them in a permanent fashion. But they are close; they have gone through a lot together. Might there be moment of wcakness in which the two might explore their chemistry? Maybe." Questioned at the end of the fourth season on whether Cuddy and House.
Pred 0 (False)
Source Models
Label 0 (False)
MNLI
SST2
ONLI
QQP
SQUAD
ReCoRD
Preds from individual source
X
0/
IX
0/
0
0/
Weights from SESoM
0.0100
0.3151
0.0176
0.0878
0.3977
0.1717
Example #
[WiC]
s s_1: An emerging professional class.
s_2: Apologizing for losing your temper, even though you were badly provoked, showed real class.
Pred
0 (False)
Source Models
Label O (False)
MNLI
SST2
QNLI
QQP
SQUAD
ReCoRD
Preds from individual source
X
1x
1X
0/
0/
0/
Weights from SESoM
1.6376e
2.4063e 4
2.1660e
0.2839
0.3430
0.3723
Example #8
[WiC]
s-1: He could not touch the meaning the poem.
s.2: Helen Keller felt the physical world by touching people and objects around her.
Pred 0 (False)
Source Models
Label 0 (False)
MNLI
SST2
QNLI
QQP
SQUAD ReCoRD
Preds from indívidual source
X
0/
1
0/
0/
0/
Weights from SESoM
0.0375
0.0904
0.0121
0.1120
0.4110
0.3371
22

---

Published as a conference paper at ICLR 2023
Table 17: Case study from CB and RTE target task. "Label" (yellow cells) is ground truth label and "Pred" (yellow cells) is the predicted label obtained by SESoM with the weights (orange cells) shown in the table. "Preds from individual source" (pink cells) shows predictions of each source model Pr 0), ... (P..3 0) obtained by the pre-softmax logits (b..i;....b.7
Example # 9
[CB]
premise:
Who knows? The point is, do go with it or not? Do assume there is a shipment? aypothesis: there shipment.
Pred
(Neutral)
Source Models
Label
(Neutral)
MNLI
SST2
QNLI
QQP
SQuAD
ReCoRD
Preds from individual source
0X
0X
1X
0X
2
2
Weights from SESoM
0.0310
0.0260
1.8677
9.433e-5
0.4084
0.5342
Example # 10
[CB]
premise:
But what may not know is just what makes somebody sucker. What makes people blurt out their credit-card numbers to caller they've 'e never heard of? Do they really believe that the number is just for verification and is simply formality on the road to being a grand-prize winner? The number is just for verification and is simply a formality on the road to being
hypothesis:
grand prize winner.
Pred
(Contradiction)
Source Models
Label
(Contradiction)
MNLI
SST2
QNLI
QQP
SQuAD
ReCoRD
Preds from individual source
OX
1
OX
1
Weights from SESoM
0.7075
2.1385e-3
0. 1998
1.2705e-4
0.0234
0.0668
Example # 11
[RTE]
premise:
Dana Reeve, the widow of the actor Christopher Reeve, has died of lung cancer age 44, according to the Christopher Reeve Foundation. Christopher Reeve had accident.
hypothes i:s: s:
Pred (not_ enntailment)
Source Models
Label (not_cntte )ot__enna)))))
MNLI
SST2
QNLI
QQP
SQUAD
ReCoRD
Preds from individual source
1
OX
I
OX
1
Weights from SESoM
0.5337
0.0092
0.2753
0.0560
0.0825
0.0434
Exampie # 12
[RTE]
premi se: Hands Across Divide was formed in March 2001, and one of immediate aims was to press for more freedom of contact and communication right away between the two parts of Cyprus, and for early progress towards solution to "the Cyprus problem"".
aypothesis: Cyprus divided into two parts in March 2001.
Pred
(not_entailment)
Source Models
Label (not entailment)
MNLI
SST2
QNLI
QQP
SQUAD
ReCoRD
Preds from individual source
0
OX
17
0X
0
IV
Weights from SESoM
0.0698
0.1879
0.3535
0.1364
0.0259
0.2265
23


Reviews:
[{"review_id": "ZDH1ym-hjCw", "reviewer": "Reviewer_ATJm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper prose SESoM can adjust the contribution of each model when ensembling the model outputs. SESoM uses an attention module to calculate the attention of each model’s output. The proposed SESoM can achieve SOTA performance in both full-data settings and few-shot settings.", "summary_of_the_review": "The paper prose SESoM can adjust the contribution of each model when ensembling the model outputs.  The experiments are comprehensive. The idea is not novel. ", "strength_and_weaknesses": "The main strengths: \n1. The paper is well-written and the idea is well-motivated. \n \n2. The experiments are quite comprehensive. The empirical analysis section answers several interesting questions. \n \n3. The empirical study that compares the prediction ensemble with the prompt ensemble is quite interesting and can inspire many related fields. \n \nThe main weaknesses: \n1. The idea that combining the output of several models using the attention strategy is not novel in deep learning. \n\n2. I can not understand why sample-specific rather than task-specific preference is important for prompt tuning. What if a similar sample exists in a quite different source task? Will the attention model pays more attention to this similar sample resulting in a negative impact on the target task performance since the source task and target task are quite different? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X1xxTwH_GcP", "reviewer": "Reviewer_eQBR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach for transferring knowledge from source tasks to a target task that has only a few examples available. Specifically, they use source models based on a pre-trained (and frozen) language model, where each source model consists of a trained prefix (obtained via prefix tuning on that source task). Their approach is the following: first, they finetune the prefix of each source task using data from the target task. Then, keeping all parameters frozen (both the language model as well as the per-source learned prefixes), they train an attention model which, given (the tokens of) a (max-pooled) input sequence representing a single example from the target domain, and given the logits produced by each source model for the target task, figures out how to ensemble those logits on a per-example basis, so as to do best on that target example. After having trained the attention model on the few-shot data from the target task, it is deployed to make predictions for other examples from the target task. They show empirically that their method outperforms previous approaches for transferring knowledge that weren’t explicitly designed for the few-shot setting. They also demonstrate that they outperform ensembling baselines, that their approach can work well for larger numbers of shots too, and they verify different empirical decisions (like ensembling the logits instead of interpolating the prompts themselves to create a new prompt for the target domain).", "summary_of_the_review": "Overall, I found the paper well-written, the problem explored is interesting and relevant and the proposed method seems to outperform previous work on a variety of datasets. The empirical analysis is interesting but a couple other design choices / ablations should also be explored (see above). I’m not an expert in NLP so it’s hard for me to judge if the previous work considered covers all the relevant related work and whether the datasets used are the most relevant to study. However, I recommend weak acceptance for the above reasons. ", "strength_and_weaknesses": "Strengths\n========\n[+] The paper is clearly written and easy to understand\n\n[+] The proposed method outperforms the previous approaches and baselines considered on a variety of datasets\n\n[+] The authors empirically verify different design choices and baselines and perform interesting analyses (like the effect of using the top-k most similar source domains for each target task)\n\nWeaknesses\n===========\n[-] Some useful ablations / exploration of alternative design choices still missing (see below)\n\n[-] The proposed method requires several forward passes through the language model (one per source task, that uses a different prefix) in order to perform a single prediction, thus adding computational overhead at test time compared to other methods. Would be good to add some discussion on this.\n\n[-] While the paper is clearly written for the most part, I would have liked to see a detailed description of the tasks used, both as source tasks and the target tasks considered. I couldn’t find details of them in the Appendix either, though please let me know if I missed it. This is important as understanding the tasks considered is useful for interpreting the results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GhtlD8dK8A", "reviewer": "Reviewer_HJCk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes model ensemble for few-shot prompt tuning in knowledge transfer tasks. The paper use an attention module to do the sample-specific ensemble for different tasks' soft prompt and apply the new ensemble of logits to the new task. Numerical experiments are presented to show the effectiveness of the idea. ", "summary_of_the_review": "Based on my lack of expertise, I think this paper is interesting. I would like to weakly accept this paper while maintaining a low confidence.", "strength_and_weaknesses": "Strength:\n\n1. The idea to use ensemble is straightforward while effective. It makes the idea simple and easy to implement.\n2. The experiment results are good and improve the previous SOTA on most of the tasks considered in this paper.\n3. The paper is well written and easy to understand.\n\nWeakness: I am not an expert in knowledge transfer and prompt tuning, thus I do not know the explicit weakness of this paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6MFjlH-bp1", "reviewer": "Reviewer_bhqg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to improve the effectiveness of the prompting method.  The prompt tuning approaches have attracted growing interest due to its parameter efficiency. However the limited training samples in few-shot settings, prompt tuning fails to match the performance of full-model fine-tuning.  The authors design an ensemble method to deal with few shots and improve the effectiveness of transfer learning.  The empirical study shows promising results. \n", "summary_of_the_review": "Prompt tuning is a hot topic since 2021. There have been several “hacks” showing promising results,  But realistically, there were so many unreported failure cases.  It is good to see a paper using principled approaches to systematically improve performance.  Since I have worked on this subject with several interns, I know the challenges and appreciate the improvements that this work could achieve.\n\nMy question is how the source models are obtained, and given a novel task, how can you identify the most useful source models from which to transfer information.  If source models are not available, what is your fall back scheme?", "strength_and_weaknesses": "Strengths:\n1. A method developed based on well though through motivations.\n2. Empirical study shows better results.\n\nWeaknesses:\n1. To ensure this method work, the source models are expected to be robust.  In other words, some robust source models must be trained to be borrow.  But it may not be true that these source models are always available and comparable.\n2. Can the source models trained on top of other pre-trained models?  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster