Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning
Authors: Hao Zhao
Jie Fu
Zhaofeng He
Venue: EMNLP 2023 Main (2023)
Keywords: Multi-task learning
Fine-tuning
Sample-efficiency
Prototype learning

Abstract: Parameter-efficient fine-tuning (PEFT) has shown its effectiveness in adapting the pre-trained language models to downstream tasks while only updating a small number of parameters. Despite the success, most existing methods independently adapt to each task without considering knowledge transfer between tasks and are limited to low-data regimes. To overcome this issue, we propose Prototype-based HyperAdapter (PHA), a novel framework built on the adapter-tuning and hypernetwork. It introduces an instance-dense retriever and a prototypical hypernetwork to generate the conditional modules in a sample-efficient manner. This leads to comparable performance improvements against existing PEFT methods on multi-task learning and few-shot transfer learning. More importantly, when the available data size gets smaller, our method outperforms other strong baselines by a large margin. Based on our extensive empirical experiments across various datasets, we demonstrate that PHA strikes a better trade-off between trainable parameters, accuracy on stream tasks, and sample efficiency. Our code is publicly available at https://github.com/Bumble666/PHA

Prototype-based HyperAdapter for Sample-Effcieent Multi-task Tuning
Zhao Jie Fu Zhaofeng He'
Beijing University of Posts and Telecommunications 'Hong Kong University of Science and Technology
(haozhao, waaadha,,zaofengggeee jebppee..eeebbp..cooo jiefuEust.hk
Abstract
Multi-task Tuning
New Task Generalization
Parameter-efficient fine-tuning (PEFT) has shown its effectiveness in adapting pre- trained language models downstream tasks while only updating small number of parame- ters. Despite the success, most existing meth- ods independently adapt each task without considering knowledge transfer between tasks and limited to low-data regimes. To over- come this issue, we propose Prototype-based HyperAdapter (PHA), novel framework built the adapter-tuning and hypernetwork. It introduces an instance-dense retriever and prototypical hypernetwork to generate the con- ditional modules in sample-efficient ner. This leads to comparable performance improvements against existing PEFT methods on multi-task learning and few-shot transfer learning. More importantly, when the avail- able data size gets smaller, our method outper- forms other strong baselines by large margin. Based on our extensive empirical experiments across various datasets, demonstrate that PHA strikes a better trade-off between trainable parameters, accuracy on stream tasks, and sam- ple efficiency. Our code is publicly available at https://g/h thub. com Com//umble666/PHg
prototype
similarity score
retrieval
retrieval vector
Embedding Space
Memory Space
02
HyperNet
Instance-dense
Instance-dense Retriever
Retriever
Adapter
Frozen Encoder
Frozen Decoder
Frozen Encoder
input
output
unknow input
Figure 1: Prototype- based HyperAdapter generates task- specific adapter with shared hypernetwork and pro- totypes calculated by task-specific instances. During generalization, the relevant task prototype is retrieved by the Instance-dense Retriever, and the new adapter layers are generated by hypernetwork.
1 Introduction
per task and adapts PLMs downstream tasks. Nonetheless, these methods require learning differ- ent modules to adapt to diverse tasks, and the cost of the parameter increases proportionally with the number of tasks. On the other hand, training task- specific modules separately fails to reap benefits from other relative tasks.
Fine-tuning pre-trained language model (PLM) yields extraordinary potential for simultancous adaptation to multiple downstream tasks in multi- task setting. However, fine-tuning all the parame- ters of models induces substantial storage and de- ployment costs, especially as pre-trained model sizes are growing rapidly. To address this is- sue, several works (Houlsby et al., 2019; Lester et al., 2021; Karimi Mahabadi et al., 2021a; Hu et al., 2022; Ding et al., 2022; Gui and Xiao, 2023; Zeng al., 2023; Liao al., 2023; Xie and Lukasiewicz, 2023) have developed parameter- efficient fine-tuning which trains compact modules
Recent work (Karimi Mahabadi et al., 2021b: Ivison and Peters, 2022) has proposed training a hypernetwork to generate the parameters of these modules to achieve better trade-off between pa- rameter efficiency and adaption for downstream tasks. These methods encourage the multi-task learning model to capture the shared information by leveraging task-shared hypernetwork while climi- nating negative task interference by generating con-
"Equal technical contribution, co-first authors. Conresponding authors.

---

ditioned modules individually. Despite these meth- ods' success in multi-task learning. there are still some issues:(1) hypernetwork based methods gen- erally optimize the specific embedding and shared hypernetwork together by end-to-end training with- out any regularization. Task-specific information is inseparably intertwined, which suppresses the effi- ciency of the hypernetwork, especially in resource- limited settings. (2) these existing approaches gen- eralize to new tasks that require task-specific prior knowledge or knowledge from frozen pre-trained models.
the specific embeddings are optimized directly dur- ing tuning, our method efficiently learns specific embedding with side information, which helps to optimize the embedding space in low-data regimes. During the adaptation of new tasks, since we main- tain the previous task-level semantic features as prototypes that align with the instances in the em- bedding space, we match the corresponding pro- totype for the current new task by calculating the distance between the new instances and the previ- ous prototypes.
We evaluate PHA on 13 NLP datasets across diverse tasks. Extensive experiments show the ef- fectiveness of PHA, especially in low-data regimes. Meanwhile, PHA is able to achieve a few-shot do- main adaption with 4-32 shots. For example, PHA outperforms the strong multitask adapter transfer baseline by 1.0% with lower trainable parameters on the GLUE benchmark. In low resource regimes where only 100 samples per task from the GLUE benchmark is available, PHA outperforms adapter- tuning by 8.0%. Our analysis shows that PHA efficiently captures specific information and shared information while reducing negative transfer. We also present detailed analysis to demonstrate that the instances from different tasks be identified by corresponding prototypes and used for new task adaption.
These works (Karimi Mahabadi et al., 2021b; Pfeiffer et al., 2023) indicate that the task-shared hypernetwork serves cross-task information captor, while specific embedding should encapsu- late task-level semantic features in order extract pertinent information from the hypernetorkk for generating corresponding module parameters. Em- pirically, task-level features are typically implicitly represented by related instance features. A natural idea to encourage embedding generation is to calcu- late the central points (prototypes) of task task-specific instance features.
In this paper, we introduce the Prototype-baat based HyperAdapter(PHA),, a novel framework built on adapter-tuning that achieves both multi-task learn- ing and generalization to new tasks in a sample- efficient manner. As depicted in Figure 1, PHA consists of two main components, Instance-dense Retriever and Prototypical HyperNetworks. The first part aims train a retriever discriminate the instances from different tasks in embedding space. For the second part, we aim to estimate the task-specific prototypes with the instance-level fea- tures and keep the prototypes as embeddings to be trained with the hypernetwork.
2 Background
HyperNetworks. A hypernetwork(Ha et al., 2017) can generate parameters to be used by networks or modules. Specifically, the hypernetwork, denoted as lww leverages an embedding I to generate the module parameters Φ
Φ = h (I)
(1)
Specifically, we project the encoded instance fea- tures into embedding space using the retriever. To avoid instances interference in embedding space, we train the retriever with the InfoNCE estimator (Oord et al., 2018). As a result, it clusters intra- task instances and increases the distances between inter-task instances. The projected features here can be deemed as instance-level semantic features that are used to estimate leskl embeddings. In- spired by PCL (Li al., 2021), we estimate the task-specific embedding using the contrastive pro- totypical loss, which encourages the prototypes to become the center points of instance-levé fea- tures. Compared with the existing method, where
Adapter. Adapter(Houlsby al., 2019) is com- monly used in pframeter-efficient tuning that aims to apply PLM to downstream tasks. Specifically, adapter-basee tuning inserts trainable task- specific modules into transformer layers, while keeping the PLM fixed. The adapter A'(z) for layer i is defined as:
A'(x) D'(ReLU((UU(())) +
(2)
where D Rdd and U E Roxd are down/up- projection matrices. IRd refers to the input. d is the hidden dimension of PLM, is the bottleneck size satisfying b < d.

---

He et al. (2022a) propose using the parallel adapter, which is different from the traditional se- quential insertion method. They demonstrate the more efficient parameter-efin efficieent fine-tuning that the parallel adapter offers.
LIR
(4)
where L is the learning objective for task i, f()) is the cosine similarity function. D, is set of positive samples of 2; and S(i) denotes set of negative samples for 24.
3 Method
Problem Setup. We are following a general multi- task learning problem. Given a pre-trained lan- guage model MM with parameters 0 and a set of target tasks {D} {Di, D2....... ....... Dr}, where is the total number of tasks and (D:) {af, vf") ) represents the training data of the i-th task with N; samples. The main objective of our study is to fine-tune Ma for downstream tasks {D} using a multi-task learning setup and to ensure that it is capable of generalizing to new tasks.
The instance-dense retriever aggregates instance- level information from the same task and enables flexible reuse of knowledge used for few-shot trans- fer learning.
3.2 Prototypical HyperNetworks
Simply using the learned task embedding to en- capsulate task-specific information biases the task- shared hypernetwork to overfit the training data distribution, which means that inadequate sample efficiency and mixed knowledge are more suscep- tible to changes in distribution during cross-task transferring.
Method Overview. The key idea of our approach is to direetly learn prototype embeddings for each task from training instances which is acquired by using a task-shared encoder, then generating task- specific adapter layers by retrieving prototype em- bedding and feeding it into a hypernetworks. As shown in Figure 1, the encoded instances are pro- jected into retrieval vectors by the instance-dense retriever for prototype learning. These prototypes that represent the task-specific information enable hypernetwork to update efficiently. This potentially allows more sample-efficient fine-tuning and few- shot transfer learning.
To overcome this issue, we propose to implicitly exploit the instance-leva information to instruct the task embedding instead of end-to-end training. To be more specific, we found the contrastive formula- tion is an efficient strategy for learning robust and sample-efficies Hypernetworks. We first initial- ize a set of embedding { where {ka} E Id is trainable vector learn the specific informa- tion of the i-th task. The learning objective for an embedding is
3.1 Instance-dense Retriever
C = Σ
N log Σ
exp [(: ki)
exp
We define an instance-dense retriever for better generalization. Let 1ı E I& denote the last layer's mean-pooled hidden state of the training sample, which belongs to the i-th task. To align the embed- ding feature with the training sample in the latent space, an instance-dense retriever G(.) is applied to construct the retrieval vector 2; G(hx). where G() is an MLP consisting of two feed-forward layers and a ReLU non- non-linearity. Furthermore, need the retriever have the ability which explic- itly encourage alignment between instances from the same task, as well as push away instances from different tasks.
exp f(zi kmm)
RED
Ko VV)
(5)
1
L
C
(6)
where V(i) is a set of negative embedding. The objective forces each embedding to make use of the relative relationships between samples across tasks and avoid iimpll-inefficceen knowledge transfer. To generate specific parameters for different transformer layers and reduce the number of train- able parameters, we introduce a learnable layer em- bedding denoted as cn, following a similar recipe as in Hyperformer (Karimi Mahahadi et al., 202 m denotes the m-th layer of transformer model.
To efficiently learn the discriminative retriever, we introduce the following loss function LIR based on the InfoNCE:
Let (-) denote the HyperNetwork which gen- erates the weight matrices D and U for task conditional adapter ATT:
exp (z, 2z)
Σ RCD
Σ log Σ exp S mm) 3jED,
N₂
=mE)(1)
(3)
(D U = (C(kss em)), (m)))
(7)

---

where C(-) is project network to concatenate the task embedding and layer embedding into a mixed embedding ll
of adapters for new tasks and optimize with the cross-entropy loss.
Our method enables efficient generalization to new tasks with limited training examples, owing to the retrieved adaption embedding containing knowledge similar to that required for the new task.
Inspired by Pfeiffer et al. (2021) and He et al. (2022a), we only insert these conditional param- eters (Adapter) into the Feed-Forward Networks
(FFNN sub-layer in parallel:
Experiments
- (F)( + A(LN(x)),
(8)
4.1 Datasets
where (.) represents the LayerNorm layer. This enables efficient decoupling of knowledge from different tasks to task prototypes and adapts the changeable data distribution during transfer learn- ing.
Following prior works on multi-task learning for natural language understanding (NLU) tasks, we consider 8 datasets from GLUE (Wang et al., 2019b) benchmark and 4 datasets from Super- GLUE (Wang et al., 2019a) benchmark to eval- uate the performance of our models. This bench- marks is collection of text classification tasks including CoLA (Warstadt et al., 2019) for sen- tence acceptability, SST-2 (Socher et al., 2013) for sentiment analysis, MNLI (Williams et al., 2018), QNLI (Demszky et al., 2018), RTE (Giampiccolo et al., 2007), CB (De Marneffe et al., 2019) for nat- ural language inference, STS-B (Cer et al., 2017) for sentence similarity, MRPC (Dolan and Brockett, 2005), QQP (Wang et al., 2019c) for paraphrasing similarity, WSC (Levesque al., 2012) for corefer- ence resolution, BoolQ (Clark et al., 2019) for ques- tion answering and WiC (Pilehvar and Camacho- Collados, 2019) for word sense disambiguation. addition, we also introduced an additional dataset: SeiTail (Khot et al., 2018) 11) for few-shot adaption.
3.3 Multi-task Tuning and New Task Generalization
PHA achieves sample-efficient multi-task learning and few-shot adaption with different training meth- ods.
Multi-task Tuning. We follow general multi-task learning setup, where the task identity included, and the different datasets are concatenated together. To achieve efficient fine-tuning, the encoder with task-shared adapters is used for encoding the train- ing sample, and we estimate the embedding cor- responding to the retrieval vector given the task identity via the loss function in Equation 5. The decoder is attached by the specific adapters con- ditioned on contextual information. The whole model is trained a sequence-to-sequeee setting with the following objective function:
4.2 Baselines
To evaluate the effectiveness of our proposed method, we conduct an analysis against several established methods that serve as strong baselines for multi-task learning: Adapter (Houlsby et al., 2019) and Shared-Adapter, we train adapters on single task group of tasks and place them into transformer layers. Hyperformer (Karimi Ma- habadi et al., 202 and Hyperdecoder (Ivison and Peters, 2022) that use task-conditioned or sample-conditioned Hypernetworks to generate adapters and place them into transformer layers. In addition, We compare our method with Fully fine-tuning(FT), and Shared-FTT share the model across different tasks. We also compare the state-of- prompt transfer methods: Prompt tuning(PT) (Lester et al., 2021), prompt tuning prepends tun- able embeddings to the input layer, and the em- beddings are initialized with each task respectively. SPoT (Vu et al., 2022) and ATTEMPT (Asai et al.,
CTotal LPLM + XLIIR + LPTo).
where LPLM Ei= L, denotes the cross-entropy loss for all training tasks and À scalar balancing factor.
PHA allows the specific embedding to efficiently capture contextual information which helps the hy- pernetwork to generate the parameters of adapter layers in sample-efficient adaption.
New Task Generalization. For few-shot adaption, we retrieve the adaption embedding ki by calculat- ing the similarity scores of retrieval vector z and learned embeddings (k) after multi-task training:
R arg max [(k/] 2)-
(10)
During training, we feed the adaption embedding to hypernetworks that generate the weight matrices

---

GLUE
SuperCLUE
Tinable
Method
CoLA
SST-2
STS-B
MRPC
QQP
MNLI
QNLI
Avg
BoolQ
CB
WSC
Avg
Params
220M
61.8
94.6
89.7
90.2
91.6
86.8
93.0
71.9
84.9
81.1
70.2
85.7
59.6
74.2
Shared-FT'
28M
54.9
92.5
88.8
90.2
91.1
85.7
92.0
75.4
83.8
78.5
69.5
85.2
66.7
75.0
Adupter!
1.M
64.0
90.7
85.3
90.2
86.5
71.9
84.5
85.7
75.7
Shared-Adapter!
1.8M
61.5
93.0
89.9
90.2
90.5
86.3
93.2
70.3
84.4
78.4
67.3
85.2
64.7
73.9
PT"
76,8k
10.6
90.9
89.5
68.1
89.7
81.3
92.8
54.7
72.2
61.7
48.9
67.9
51,9
57.6
Hyperformeres!
638K
59.0
93.7
90.3
88.6
89.9
85.0
93.3
77.5
84.7
75.8
68.9
81,5
52.9
69.8
HyperDecoder!
1.8M
55.9
94.0
90.5
87.7
90.5
86.0
93.4
71.7
83.7
77.8
66
92.6
66.7
75.8
PHA (ours)
616K
60.6
94.0
88.9
89.2
90,9
86.3
93.4
80.4
85.5
80.7
64.8
96.3
62.7
76.1
Table I: Overall comparison on Multi-task Adaption. b5-base is used as the PLM backbone of methods. We also report Tunable Params, which represents the number of parameters that need to be fine-tuned for each task. The best result on each block in bold. For GLUE results, i denotes results reported from (Karimi Mahabadi et al., 10). j denotes results reported from (Ivison and Peters, 2022). denotes results reported from (Asai et al., 2022).
2022), MPT (Wang al., 2023) adapts the target tasks with the shared prompts obtained by distilling knowledge from the source tasks.
3e-4 with linear decay and 500 warm-up steps. We set the balancing factor 0.1 in Eq. and keep fixed for all our experiments. All experiments run for times with different seeds and we report the average for each result. The detailed configura- tions per method on diverse datasets are shown in Appendix A.
4.3 Experiments Details
Following the setting of Karimi Mahabadi et al. (2021b), when an original testing set is unavail- able, the validation set is utilized as the testing set. In situations where the dataset contains less than 100k records, the validation set is divided into two sets: validation and testing. Conversely, larger datasets utilize 1000 training set samples selected for validation, with the original valida- tion set used for testing. For the multi-task adap- tation experiment, we performed multi-task learn- ing on 8 datasets from GLUE and 4 datasets from SuperGLUE. For the low-data adaption experi- ment, we separately sample each individual task in GLUE with different proportions and quantities (100, 500, 1000, 2000, 4000, 1%, 3%, 5%). As for the evaluation strategy, we use Pearson Correlation for STS-B and accuracy for other tasks as metrics. We save checkpoint every 1000 steps for all mod- els and report the average performance of all tasks on a single checkpoint. In the fho-shot adaption experiment, we randomly sample & = 4, 16, 32 in- stances from the training set while the entire test set is used for testing. We mainly use T5-Base (220M) model (Raffel et al., 2020) as the pre-trained lan- guage model. In addition, we also use T5-Small (60M) and T5-Lar (770M) to explore the effect of model size PHA performance in Section 4.4.5, Unless specified, we train for 65k steps using the A (Loshchilov and Hutter, 2019) optimizer and set the batch size as 128 for all experiments. During training, the initial learning rate is set to
4.4 Results and Analysis
4.4.1 Multi-task Adaptation
Table shows the evaluation results on GLUE and SuperGLUE. The results indicate that PHA out- performs all comparative methods regarding per- formance improvement while maintaining param- eter efficiency. Note that do not compare with SPOT, ATTEMPT, and MPT since they require pre- training prompts to save the knowledge from source tasks and transfer them to target tasks. Extending these methods to the same setting where only pre- trained models are available is beyond our scope. Therefore, under the experimental setup of multi- task learning, our method cannot achieve fair comparison with them. It is worth mentioning that MPT, ATTEMPT, and our method both use the same two-step training method in the sho-shot transfer setting (Section 4.4.3). Specifically, our PHA approach achieves a performance increase of + +1.0% over Adapter while only using 3x fewer trainable parameters. When we compared with the state-of-art adapter-based multi-task methods in- cluding recent Hyperformer++ and Hyperdecoder that use a hypernetwork to generate conditioned parameters similar to our approach, PHA achieves 0.8% and 1.8% point accuracy improvements re- spectively on GLUE while utilizing the same or lower number of trainable parameters. This demon-

---

k-shot
FT!
Adapter'
PTT
SPoT
HFt
ATP
HD
MPT
PHA(our)
CB
57.7
51.1
53.5
714
60.7
82.1
69.1
73.6
76.5
4
SeiTail
79.6
79.5
57.7
69.6
82.0
80.2
75.4
80.2
82.5
BoolQ
50.5
53.4
61.6
50.5
48.0
61.8
54.4
62.2
68.2
CB
77.0
74.8
63.5
64.3
76.3
78.5
75.3
78.6
79.6
SeiTail
80.0
86.5
79.5
85.4
87.3
87.7
BoolQ
56.5
51.4
61.9
50.6
50.2
60.0
64.6
63.3
71.3
CB
80.0
67.8
64.3
81.4
85.7
79.6
82.1
82.7
32
SeiTail
81.9
85.0
60.2
71.9
85.8
80.2
85.1
86.3
88.6
BoolQ
58.4
54.5
61.7
61.2
58.3
65.33
688.3
68.9
72.0
Result for few-shot transfer (k: = 4, 16, 32). We report the accuracy for all tasks over 10 random seeds. The best result each block bold. All of the models are trained GLUE tasks with T5-Base backbone. denotes results reported from (Asai et al., 2022). denotes results reported from (Wang et al., 2023).
Table 2:
14) By quaniity
(6) By scale
As shown in Figure PHA oulperforms other baselines consistently across different configura- tions. We obverse that Fine-tuning performs the worst in cases of limited available data. A poten- tial reason is that tuming all parameters of the base model makes it susceptible to keep the model in the state of over- wver-fitting, especially in the low-data regimes. As for other experimental methods, we observe that when the number of available sam- ples is relatively smaller, the existing state-of-the- art multi-task methods with hypernetworks close to the performance of Adapter until the number of available samples increases. This observation indicates that hypernetworks struggle to generate optimal parameters when using randomly initial- ized task embeddings instances contextual information under low-data regimes. Moreover, to better simulate the training task distribution in the GLUE benchmark, we randomly sample each individual task in GLUE for different proportions. Figure 2(b) shows the comparison between PHA and udapter-based methods. Similar to the results in Figure 2(a), our proposed method outperforms others by large margin. We also conduct experi- ments on 4 datasets (BoolQ, WiC, CB, WSC) from SuperGLUE, as detailed in Appendix C.
Accuarcy
Apsueres
40
XD
R1A
samplesttss
samplestask
Figure 2: Results on GLUE for sample efficiency exper- iments, which sample the various number or proportions of training samples per task. AD, HF and HD denote Adapter, HyperFormer and HyperDecoder.
strates the potential of our method to reduce the negative interference between tasks better. In ad- dition, we observe that FT performs the best in all experimental methods, while it requires 220M trainable parameters on a single task. We find that our PHA is as competitive as that of the FT(85.5 vs. 84.9) and reduce the trainable parameters from 100% to 0.28%. We also analyze the effectiveness of PHA in parameter efficiency, as detailed in Ap- pendix B.
4.4.2 Low-data Adaptation
our framework, we posit that the task proto- types, which are estimated by instance-level fea- tures, can better represent task information and improve the performance of hypernetworks a sample-efficient manner during multi-task adapta- tion. To verify that our proposed method general- izes better when there are only limited available re- sources, conduct low-data adaption experiments the GLUE benchmark. Following Karimi Ma- habadi al. (2021b), we train all models(Fine- tuning, Adapter, Hyperformer++, and Hyperde- coder) for 15k steps. More details are in Sec- tion 4.3.
The superior performance PHA over all com- peting methods indicates that our proposed task prototype efficiently captures contextual informa- tion to enable hypernetwork to generate the param- eters of adapter layers.
4,4.3 Few-shot Adaptation
We explore how proposed method performs when adapting new tasks with sample-efficient. Specifically, following Wang al. (2023), we con- duct few-shot experiments on BoolQ, CB, SciTail and compare PHA with the strong baselines, includ- ing Fine-tuning, Adapter, Prompt tuning, SPoT, Hy-

---

(a] w/o retriever
retriever
Prototype
Retriever
GLUE Score
x
84.0
10-
84.3
5-
X
84.7
85.5
20
Table 3: Ablation study on GLUE.
-10
0
10
-20
20
MB3C
114
001.
007
SNJ
(c) TG Largo
Figure 4: t-SNE visualizations for samples w/ (right) and w/o (left) retriever.
60
85
90
75
80
Avg
70
75
information hinders the positive transfer across tasks under the limitation of hypernetwork capacity. while our design of task prototypes allows hyper- network to capture shared-task information well, which is vital for enabling positive transfer across tasks. Table (row 3) removes the retriever. The task prototypes estimated by the originally en- coded instances. This results in intertwined task prototypes due to the relative dispersion of instance information in the embedding space. The decrease in performance suggests that adding the instance- dense retriever enables the prototype to encode task-specific knowledge better. Furthermore, we provide visualization of the encoded instances from the GLUE benchmark to compare the effect of adding and removing the instance-dense retriever, as shown in Figure While samples belonging the same task tend be located near each other in the latent space, samples from different classes (e.g., STS-B, MRPC, RTE) still interleave with each other. After the retriever is added, instances from the same task are tightly clustered, while dif- ferent tasks are widely separated.
66
70
75
60
1%
3%
5%
100%5
85
156
100%
1%
100% simples/lask
samples/tiok
Sompleedlens laakk
Figure 3: The comparison of PEFTs (Adapter, Hyper- former++, Hyperdecoder, and our proposed PHA) and FT on GLUE with three scale pre-trained models under low-data regimes and full-data regimes.
performer, ATTEMPT, HyperDecoder, and MPT. The results in Table are obtained by training an 8-task adaptation for GLUE and fine-tuned with few-shot samples from BoolQ, CB, and SciTail. More details are in Section 4.3.
Table summarizes the results on the few-shot adaption setting. Among Adapter-based transfer methods, PHA brings around 3% 20% absolute improvements over the Adapter across different settings. While Hyperformer achieves better gener- ation for new tasks, requires us have precise understanding of target tasks. PHA significantly improves the performance of Hyperformer with- out requiring task-specific prior knowledge. In addition, our proposed method significantly outper- forms other prompt-based transfer methods in most settings.
4.4.5 Impact of Model Scale
To verify that method is applicable differ- ent pre-trained model sizes, we also experiment T5 with sizes from Small (60M) to Large (770M) on GLUE datasets, while reporting the average result of PHA as well as fully Fine-tuning (FT), Adapter (AD), Hyperformer++ (HF) and Hyper- decoder(HD). As shown in Figure 3, under three scales of pre-trained model, we find that PHA achieves superior and competitive performances in low-data and full-data regimes, respectively. This indicates that our proposed prototypical strategy is still able to achieve the best sample efficiency when the size of large-scale transformer models increases.
Our results demonstrate that approach effec- tively generalizes new tasks despite the limited availability of training samples.
4.4.4 Ablation Study
We perform an ablation study on the GLUE bench- mark to evaluate the effectiveness of the proposed modules. The prototype design and the instance- dense retriever removed independently for this purpose. As shown in Table (row 2), when remove the prototype design and use the retrieval instances to train, we observe the performance has a significant drop. This shows that the instance-level

---

3st GlUF
RN New
shared information between the different tasks and train unified model simultancously solve mul- tiple tasks. the context of NLP, this typi- cally achieved by sharing certain layers across all tasks while using task-specific layers for specific tasks (Liu et al., 2019). With the popularization of large language models (LLMs), Raffel et al. (2020) explores the training of LLMs on various tasks which are transformed into a unified format, and some works (Aghajanyan et al., 2021; Aribandi et al., 2022; Sanh et al., 2022; Wei et al., 2022) indicate that the LLMs can be better generalized to new tasks through large-scale multitusking training. More recent work (Pfeiffer et al., 2021: Vu et al.. 2022; Asai et al., 2022; Wang et al., 2023) focuses on multi-task transfer with parameter-efficien fine- tuning the increasing LM size. Though the ef- fectiveness of multitask learning is improved. they need to finetuning the LLMs twice on souree tasks, which are carefully selected, and multiple target tasks. This limits the applicability of the methods. Differently, our proposed method only requires a pre-trained model to achieve multi-task learning and transfer to new task.
5372
Tex pwcpetngge
5758
6TEA
COlA
05F
111
NBRC
COLA
DDP
Bca0 Targer tack
Task PStO0ce
Figure 5: Visualization the similarity scores, which are calculated by the dense retriever, between the in- stances and task prototype in GLUE (a) and between pre-trained prototypes and the task agsostgi instances in three datasets (b). Darker colors indicate higher scores.
4.4.6 Effecc of Retriever for Generalization.
The retriever plays important role in adapting to new tasks. To explore how the retriever works af- ter training under multi-task setting, we consider the similarity scores, described in Eq. 10, to mea- sure retrieval results. Specifically, we randomly sample each individual task in GLUE and calcu- late the similarity scores through the trained task prototypes and retrieval vectors transferred by the trained retriever. Figure 5(a) shows visualiza- tion of similarity scores. We find that the retriever precisely retrieved the task identity of the corre- sponding task instance. This suggests that task prototypes and instance vector has aligned em- bedding space to enable more efficient capture of single-task common features.
Several works (Jin et al., 2020; Karimi Mahabadi et al., 2021b; Ivison and Peters, 2022; He et al., 2022b) introduce hypernetworks (Ha al., 2017) to share the cross-lask information by generating the parameters of adapter layers (Houlsby et al., 2019) from specific embeddings during multi-task learning. Our work is motivated by Ivison and Peters (2022), but proposes to use task-level infor- mation represented by prototypes to optimize the embedding distribution of hypernetworks, which reduces negative transfer between different tasks and improves the performance of adaption across tasks, especially in low-resource regimes.
We also demonstrate that the retriever has the ability to match the corresponding task prototype to target tasks that require generalization. Figure 5(b) illustrates that the similarity score is relatively high for related tasks such as CB and MNLI, SciTail, and STSB, all of which belong to the NLI task family. As for QNLI and BoolQ, since the task pro- totype trained on GLUE does not include Boolean Question Answering (QA) tasks, the retriever has matched QNLI prototype, which is in the same do- main as BoolQ. Therefore, our proposed method can naturally generalize to new tasks when the re- lated task prototype and the hypernetwork contain- ing cross-task knowledge are both available.
Prototype Learning. Prototype learning is widely used to improve the representation ability of net- works for few-shot learning. Some works (Gao et al., 2019; Caron et al., 2020; Ding et al., 2021; Li et al., 2021) 1) indicate that the prototype is forced to learn some common features of samples within the class by prototype estimation. Cui et al. (2022) propose to construct verbalizer for prompt-based few-shot tuning by estimating prototypes with con- trastive learning. This differs from our method, which uses a prototypical strategy to explore the specific information for corresponding tasks.
5 Related Work
Multi-task Learning and Transfer. Multi-task (MTL) aims to take advantage of the

---

6 Conclusion
with pre-finetuning In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 5799-5811, I, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics.
We introduce Prototype-basee based Hyperrd (PHA), a novel framework built on adapter-tuning. PHA achieves both multi-task adaption and adap- tion to new task in sample-efficient manner. It generates parameters of adapter layers conditioned on task-specific prototypes which are calculated by the corresponding instance-level features. In addition, the specific prototype is retrieved and transferred to new tasks to be further tuned. The resulting method significantly outperforms previ- ous SOTA on full/low-date multi-task task adaption and few-shot adaption.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Hon- glei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2022. Ext5: Towards extreme multi-task scaling for transfer learning. In ICLR.
Akari Asai, Mohammadreza Salehi, Matthew Pe- ters, and Hannaneh Hajishirzi. 2022. ATTEMPT: Parameter-efficieni multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages
Limitations
Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics
Our work has demonstrated strong experimental results and sample efficiency in multitasking adap- tion. However, there are several limitations: Firstly, in few-shot adaption, the method we proposed, which tunes the base model on 8 NLP tasks, can generalize to new target tasks efficiently. But tun- ing on more large-scale tasks may result in better generalization improvements. Secondly, as shown in Figure 5, a new task may be related to multi- ple task prototypes, rather than a single one. In our method, we only select the most relevant pro- totypes, which may ignore the transfer of some weakly related knowledge. In addition, we use adapters in this work, but our method could pos- sibly also benefit from other Barameter-efficient approaches (Lester et al., 2021; Mahabadi et al., 2021; Li and Liang, 2021; Hu et al., 2022; Liu et al., 2022).
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised learning of visual features by contrast- ing cluster assignments. In Advances in Neural Infor- mation Processing Systems, volume 33, pages 9912- 9924. Curran Associates, Inc.
Damiel Cer, Mona Diab, Eneko Agirre, lnigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop Semantic Evaluation (SemEval-2011), pages 1-14, Vancouver, Canada. Association for Computational Linguisties.
Christopher Clark, Kenton Lee, Ming-Wai Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACL.
Ganqu Cui, Shengding Hu, Ning Ding, Longtao Huang. and Zhiyuan Liu. 2022. Prototypical verbalizer for prompt- based few-shot tuning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 7014-7024, Dublin, Ireland. Association for Compu- tational Linguistics.
Acknowledgements
This work is supported by National Key R&D Pro- gram of China (Grant No. 2022YFF1102400), Ma- jor Science and Technology Innovation Program of Hangzhou (Grant No. 2022A12D0154), Natural Science Foundation of China (Grant No. 62176025
Marie-Catherte De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Investi- gating projection in naturally occurring discourse. In Simn und Bedeutung 23.
Beijing Nova Program (Grant No. 20220484161), the Fundamental Research Funds for the Central Universities (Grant No. 2023RC72) and Theme-based Research Scheme (I45-205/21-
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into nat- ural language inference datasets. arXiv preprint 1eXx:1800.0029.
N), Research Grants Council of Hong Kong.
Ning Ding, Yujia Qin. Guang Yang, Fuchao Wei, Zong- han Yang. Yusheng Su, Shengding Hu, Yulin Chen. Chi-Min Chan, Weize Chen, et al. 2022. Delta tuning: A comprehensive study of parameter efficient meth- ods for pre-trained language models, arXiv preprint arXiv:2203.0690.
References
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations

---

Ning Ding, Xiaobin Wang, Yao Fu, Guangwei Xu, Rui Wang, Pengjun Xie, Ying Shen, Fei Huang. Hai- Tao Zheng, and Rui Zhang. 2021. Prototypical represen- tation learning for relation extraction. In Interna- lional Conference on Learning Representations.
Hamish Ivison and Matthew Peters. 2022. Hyperde- coders: Instance-specific decoders for multi-task NLP. In Findings of the Association for Computa- rional Linguistics: EMNLP 2022, pages 1715-1730, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrusing (IWP2005).
Tian Jin, Zhun Liu, Shengjia Yan, Alexandre Eichen- berger, and Louis- Douis-Philippe Morency. 2020. Lan- guage to network: Conditional parameter adaptation with natural language descriptions. In Proceedings of the 58th Annual Meeting of the Association for Computational L Linguistics, pages 6994-7007, On- line. Association for Computational Linguistics.
Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun. 2019. Hybrid attention-bast baaed prototy networks for noisy few-shot relation classification. In Proceed- ings of the AAAI conference on artificial intelligence, volume 33, pages 6407-6414.
Rabeeh Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 202 la. Compacter: Efffcientt low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems, volume 34, pages 1022-1035. Curran Associates, Inc.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACI-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1 1-9, Prague. Association for Computational Linguistics.
Rabeeh Karimi Mahabadi. Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021b. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Pupers), pages 565-576, Online. Association for Computational Linguistics.
Anchun Gui and Han Xiao. 2023. HiFi: High- information attention heads hold for parameter- efficient model adaptation. In Proceedings of the 61 Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers). pages 8521-8537, Toronto, Canada. Association for Com- putational Linguistics.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI.
David Ha, Andrew M. Dai, and Quoc V. Le. 2017. Hy- pernetworks. In International Conference on Learn- ing Representations.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficien prompt tuming. In EMNLP.
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig. 2022a. Towards a unified view of parameter-effictt transfer learning. In ICLR.
Hector Levesque, Emnstt Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning.
Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, Yaguang Li, Zhao Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi. 2022b. HyperPrompt: Prompt- based task- task-conditioning of transformers. In Proceedings of the 39th Interna- tional Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8678-8690. PMLR.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. 2021. Prototypical contrastive learning of unsuper- vised representations. In International Conference on Learning Representations.
Xiang Lisa Li and Percy Liang, 2021. Prefix-tuning Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 4582- 4597. Online. Association for Computational Lin- guistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Baohao Liao, Yan Meng, and Christof Monz. 2023. Parameter-efficient tineituning without introducing new latency. In Proceedings of the 61 Annual Meet- ing of the Association for Computational Linguistics (Volume Long Papers), pages 4242 4260, Toronto, Canada. Association for Computational Linguistics.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations.

---

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P- P-tuning: Prompt tuning can be comparable to fine-tuning actoss scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 61 -688 Dublin, Ireland. Association for Computational Lin- guistics.
of transfer learning with a unified text-to-text trans- former. JMLR.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jian- feng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics. pages 4487-4496 Florence, Italy, Association for Computational Linguisties.
Nya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Rabeeh Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. In NeurlPS.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 163 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv: (tfl::887... 1807. 037.03748.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning li- brary. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pages 0039-5059, Dublin, Ireland. Association for Computational Linguistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Superglue: A stickier benchmark for purpros language understand- ing systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguisties: Main Volume. pages 487-503, Online. Association for Computational Lin- guistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding, In ICLR.
Jonas Pfeiffer. Sebastian Ruder, Ivan Vulié, and Edoardo Maria Ponti. 2023. Modular deep learning. arXiv preprint arXiv: 11529.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019c. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding, In ICLR.
Mohammad Taher Pilehvar and Jose Comacho-Collados. 2019. WIC: the word-in-context dataset for evalu- ating sontext-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Roge- rio Feris, Huan Sun, and Yoon Kim. 2023. Multitask prompt tuning enables parameter-efficien transfer learning. In The Eleventh International Conference on Learning Representations.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits

---

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu. Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022. Finetuned language mod- els zero-shot learners. In International Confer- ence on Learning Representations.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume 1 (Long Papers), pages lI 11-2-1 122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
Zhongbin Xie and Thomas Lukasiewicz. 2023, An pirical analysis of parameter-efficine methods for debiasing pre-trained language models. In Proceed- ings of the 61 st Annual Meeting the Association for Compulational Linguisties (Volume I: Long Papers), pages 15730-15745, Toronto, Canada. Association for Computational Linguistics.
Guangtao Zeng, Peiyuan Zhang, and Wei Lu. 2023. One network, many masks: Towards more parameter- efficient transfer learning. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 7564- 7580, Toronto, Canada. Association for Computa- tional Linguistics.

---

A Implementation Details
samples/ task
AD
11F
IID
PIIA
We set the dimension the retrieval vector (2;), the task prototype (k:), and the layer embedding Zm to 128. For the instance-dense retriever G(-) with bot- tleneck architecture, we select the bottleneck size to 128. For the mixed embedding 1 the dimension of ! is selected 32 for multi-task experiments and 64 for other experiments. We use the default hyperparameters by Ivison and Peters (2022) for Adapter and Hyperdecoder. The hyperparameters other baselines are set according to the original papers (Karimi Mahabadi et al., 202 lb; Asai et al., 2022; Wang et al., 2023). Following Karimi Ma- habadì et al. (2021b); Ivison and Peters (2022), the length of the sequence is 128 the encoder and the decoder. We use its HuggingFace (Wolf al..
1%
57.8
50.4
57.7
59.2
3%
61.3
51.3
59.6
67.6
5%
60.3
54.3
59.9
Table 4: Results on datasets (BoolQ. WiC, CB, WSC) from SuperGI.UE. for the various proportions of training samples per task (1 (.,3%,5%) We report the average ac- curacy, AD, HF and HD denote Adapter, HyperFormer and HyperDecoder.
Method
XSUM
Ro-En
En-Ro
Avg
FT
7.2
24.1
20.1
17.1
Small
AD
2.9
25.1
9,7
12.6
PHA
7.6
24.9
19.7
17.4
FT
10.9
25.3
24.2
20.1
Base
8.2
26.5
19.5
2020) Pytorch (Paszke et al., 2019) implementation. All data obtained from huggingface datasets and preprocessed into a "seq2seq" format following Raffel et al. (2020).
PHA
10.8
26.6
24.8
20.7
FT
12.3
24.7
29.7
22.2
Large
12.6
27.1
25.9
21.9
PHA
14.2
27.0
28.6
23.3
B Parameter Efficiency
Table 5: Result for NLU task. We report the ROUGE2 score for XSUM and BLEU score for Ro-En/En-Ro The best result each block bold.
Our proposed method only balances sample efficiency and accuracy on stream tasks but also achieves parameter efficiency. We compare the trainable parameters of PHA with other baselines. We assume that the basic model is a transformer model with an L-layer encoder-decoder structure for T-tasks and d is the model dimension. The tasks- share adapter is attached the encoder layer and adapter is attached to the de- coder layer in our work. The bottleneck dimension Adapter Adapter for the single layer costs 2db + + d. Given bottleneck size d', the instance- dense retriever G(-) with bottleneck architecture to construct the retrieval vector C dR which result in dd' + d² parameters. We have the task proto- type k; E RR for i-th task and layer embedding em Rdd for m-th layer. These have total of rd' Ld' parameters. The project network C(-) is used for concatenating the task prototype k; R and layer embedding Cm E dRd into a mixed em- bedding 1T с Rdd, which result in 2d' d, parame- ters. The hypernetworks costs d, (2db + b + d) pa- rameters. The total cost of PHA is ddl + d² (rd' Ld") + 2d' d, + (2db b + d) + L(2db + b d). There large number of tasks and transformer layers in work settings. For the adapter, its pa- rameter count increases linearly with the task and number layers. For Hyperformer, its parameter count is more than our method due to using two
We used bottleneck structured retriever to reduce trainable parameters compared to Hyperdecoder.
hypernetworks.
C Additional Results
the
Here, we conduct more experiments to demonstrate the sample efficiency of our method on different datasets (BoolQ, WiC, CB, and WSC) with T5- Base pre-trained model. The result are included in Table 4. The performance PHA with T5-Base achieves the best performance in terms of accuracy and sample efficiency in the 4 datasets from Super- GLUE. In addition, verify the effectiveness of our method on NLU tasks, we conducted sample efficiency experiments on three datasets: Xsum, WMT16 Ro-En, and WMT16 En-Ro. Due to lim- ited computation resources, we sampled I 1% of the data and conducted experiments three backbone models of different scales(small, base, and large). The results are shown in Table 5.


Reviews:
[{"review_id": "1MrTJl7sDF", "reviewer": "Reviewer_JcAH", "summary": "", "questions": "1. The first issue with existing hyper-network approaches mentioned in the introduction section (i.e. task-specific knowledge is intertwined with shared hypernetwork) is clear to me. I was wondering if the authors can elaborate on the second issue \"these existing approaches generalize to new tasks that require task-specific prior knowledge or knowledge from frozen pre-trained models.\". Does the suggested approach address this? From what I gather, PHA also depends on task-specific prior knowledge and task embeddings (prototypes).\n\n2. I was wondering if the authors could provide Prompt tuning results for low-data adaptation. PT results are reported for both multi-task setup and few-shot adaptation.\n\n3. I was wondering if authors can explain how they ablate the prototype module. In the ablation study in section 4.4.4, the authors mention \"when we remove the prototype design and use the retrieval instances to train\". Does this imply that for the ablation, they compute task-specific cluster centroids and input these to the hypernetwork? This seems connected to the introduction's suggestion of generating embeddings by calculating central points (prototypes) from task-specific instance features. (i.e. \"A natural idea to encourage embedding generation is to calculate the central points (prototypes) of task-specific instance features\").\n\n4. In few shot setup, how does the method compute a single retrieval vector z? Do they concatenate different shots and then compute a single representation for this concatenated instance?\n\n5. I am curious to know the authors' perspective on employing a variational framework such as VAEs for prototype learning. VAEs bring regularization by modeling a distribution over class samples. Could the approach involve fitting such distributions per task, with the prototype as the distribution's mean?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments. Our responses to your comments and questions are listed as follows.\n\n---------\n\nQ1: The paper does not compare their approach with methods such as ATTEMPT and MPT that directly train prompts on source tasks.\n\nA1: We do not compare our method with ATTEMPT and MPT due to their two-step model training approach in multi-task learning. First, They train pre-training prompts to keep knowledge from source tasks and then train target prompts and source prompts together to facilitate knowledge transfer. In contrast, our method directly fine-tunes the target task using the pre-trained model. From this perspective, ATTEMPT and MPT are more like transfer learning. Therefore, under the experimental setup of multi-task learning, our method cannot achieve a fair comparison with them. Additionally, determining the appropriate sample sizes for each stage of the two-stage training process to ensure a fair comparison is also a controversial matter. It is worth mentioning that MPT, ATTEMPT, and our method both use the same two-step training method in the few-shot transfer setting (section 4.3.3), so we performed a fair comparison, and the results show that our method generalizes to new tasks more effectively compared to MPT and ATTEMPT despite the limited availability of training samples.\n\n-------\n\nQ2: Hypernetwork-based methods require prior knowledge when generalizing to new tasks. Does PHA address this problem?\n\nA2: Yes, our method solves this problem. The problem with existing hypernetwork-based methods is that when existing hypernetwork methods are generalized to new tasks, some prior information about the new tasks is required. For example, in the case of HyperFormer [1], prior knowledge of the domain to which the new task belongs is required to manually match it with the trained task embedding. **Our method does not require any prior information for new tasks,** allowing for efficient generalization through the use of the instance-dense retriever.\n\n[1] Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. ACL2021\n\n-------\n\nQ3: I was wondering if the authors could provide Prompt tuning results for low-data adaptation. PT results are reported for both multi-task setup and few-shot adaptation.\n\nA3: Following your advice, we conducted sample efficiency experiments on PT. The GLUE results are shown below.\n\n|    samples/task    | PT | PHA (ours) |\n|- | :---: | :------:|\n| 1% |  50.3 | **79.6** |\n| 3% |  56.3 | **82.1**|\n| 5% |  58.2 | **82.4**|\n\n------\n\nQ4: I was wondering if authors can explain how they ablate the prototype module.\n\nA4: We have opted to utilize the instance that aligns with the retriever as the embedding input into the hypernetwork. Additionally, we have performed additional ablation experiments on GLUE as per your suggestion. Specifically, we calculate task-specific mean instance embeddings which are then inputted to the hypernetwork. The results are shown below.\n\n|    Embedding Method    | GLUE Score |\n| - | :---: |\n| Mean embedding |  84.7 |\n| Our       |  **85.5** |\n\nThe experimental results demonstrate that our proposed task prototypes outperform the mean instance embedding, enhancing the model's ability to learn similar features among instances within the same task.\n\n-------\n\nQ5: In few-shot setup, how does the method compute a single retrieval vector z? Do they concatenate different shots and then compute a single representation for this concatenated instance?\n\nA5: Not quite. Because the dimensions of the task prototype and the sample are the same, we do not concatenate all few-shot samples. Instead, we directly select all few-shot samples and calculate the average cosine similarity between them and the task prototypes. The adaption embedding is determined by identifying the task prototype with the highest similarity value.\n\n-------\n\nQ6: I am curious to know the authors' perspective on employing a variational framework such as VAEs for prototype learning. VAEs bring regularization by modeling a distribution over class samples. Could the approach involve fitting such distributions per task, with the prototype as the distribution's mean?\n\nA6: Using a variational framework like VAEs for prototype learning is an interesting and creative idea. Your idea involves using the mean of the learned distribution as the prototype. This introduces a probabilistic perspective on prototypes, which can be advantageous in scenarios where data is inherently uncertain or noisy. The variance of the learned distribution could provide a measure of uncertainty around each prototype, potentially helping the model to make more informed decisions. While the idea is intriguing, VAEs might require a larger amount of data to learn meaningful latent representations effectively.\n\n-------\n\nWe also thank the reviewer for their minor comments regarding clarity, typos, etc. We will address each of them in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Strong motivation. Existing hypernetwork approaches train task embeddings and hypernetwork end to end so there is no inductive bias that disentangles the task specific knowledge encoded by the embedding and task-shared knowledge encoded by the hypernetwork.\n\n2. Strong ablation experiments, in particular the ablation on the instance dense retriever module and the t-sne visualization. I was curious about the necessity of a retriever and its associated loss, as one might consider optimizing equation 5 directly using instance embeddings to learn prototypes. Yet, the ablation results demonstrate that an additional loss to align task samples in the retrieval space can enhance performance. The visualization also indicates that without the retriever loss, samples from various tasks can become intertwined.\n\n3. Strong few-shot results which is comparable/better compared to methods such as ATTEMPT and MPT that pre-train source task prompts.", "reasons_to_reject": "The paper does not compare their approach with methods such as ATTEMPT and MPT that directly train prompts on source tasks.", "questions_to_authors": "1. The first issue with existing hyper-network approaches mentioned in the introduction section (i.e. task-specific knowledge is intertwined with shared hypernetwork) is clear to me. I was wondering if the authors can elaborate on the second issue \"these existing approaches generalize to new tasks that require task-specific prior knowledge or knowledge from frozen pre-trained models.\". Does the suggested approach address this? From what I gather, PHA also depends on task-specific prior knowledge and task embeddings (prototypes).\n\n2. I was wondering if the authors could provide Prompt tuning results for low-data adaptation. PT results are reported for both multi-task setup and few-shot adaptation.\n\n3. I was wondering if authors can explain how they ablate the prototype module. In the ablation study in section 4.4.4, the authors mention \"when we remove the prototype design and use the retrieval instances to train\". Does this imply that for the ablation, they compute task-specific cluster centroids and input these to the hypernetwork? This seems connected to the introduction's suggestion of generating embeddings by calculating central points (prototypes) from task-specific instance features. (i.e. \"A natural idea to encourage embedding generation is to calculate the central points (prototypes) of task-specific instance features\").\n\n4. In few shot setup, how does the method compute a single retrieval vector z? Do they concatenate different shots and then compute a single representation for this concatenated instance?\n\n5. I am curious to know the authors' perspective on employing a variational framework such as VAEs for prototype learning. VAEs bring regularization by modeling a distribution over class samples. Could the approach involve fitting such distributions per task, with the prototype as the distribution's mean?", "paper_topic_and_main_contributions": "The paper proposes PHA as a novel parameter efficient multi-task fine-tuning method using hypernetworks and adapter tuning. PHA introduces a latent space to learn prototypes for each task. The prototypes are learnt in this space using instance embeddings from various tasks. A dense retrieval module computes instance embeddings by transforming encoded sample representations (which are encoded using the encoder model) into the latent space.\nThe results indicate PHA's efficiency compared to full fine-tuning and outperform existing methods based on hyper-networks, prompt tuning methods that do not train task-specific prompts, and adapter tuning methods.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The word \"Adaption\" in titles of section 4.4.2 and 4.4.3 should change to \"Adaptation\".\n\n"}, "extra_text": {}}, {"review_id": "en746Mx32h", "reviewer": "Reviewer_rff6", "summary": "", "questions": "Please respond to the concerns listed in Weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments. Our responses to your comments and questions are listed as follows.\n\n--------\n\nQ1: For Table 1, the improvement on the SuperGLUE benchmark looks marginal. The improvement on GLUE is larger, even though its absolute number is higher than SuperGLUE. Do the authors have an explanation of why it is the case?\n\nA1: Thank you for raising this question. Firstly, the marginal improvement observed in SuperGLUE is constrained by the fine-tuning architecture, which solely adjusts task condition parameters. This finding aligns with the results analyzed in Prompt Tuning [1] and Hyperprompt [2].\n\nSecondly, it is important to emphasize that our method prioritizes sample efficiency, an aspect that has frequently been neglected in previous related studies. We performed experiments to assess sample efficiency on SuperGLUE. The results are shown below.\n\n\n|    samples/task    | AD | HF | HD | PHA (ours) |\n| - | :---: | :------:| :------:| :----: |\n| 1% |  57.8 | 50.4 | 57.7 | **59.2** |\n| 3% |  61.3 | 51.3 | 59.6 | **67.6** |\n| 5% |  60.3 | 54.3 | 59.9 |**69.3**|\n\nOur method can maintain sample efficiency even on very difficult tasks such as SuperGLUE, which proves the effectiveness of our method.\n\n[1] The power of scale for parameter-efficient prompt tuning. EMNLP2021\n\n[2] Hyperprompt: Prompt-based task-conditioning of transformers. ICML2022\n\n----------------\n\nQ2: The motivation for introducing prototype for target tasks need further explanation. For example, whether it will work if we just replace the k_i with the mean instance embeddings of the i-th task? The relation between task embeddings and instance embeddings needs further explanation.\n\nA2: The motivation behind introducing task prototypes is to encourage the model to capture similar features among instances belonging to the same task. In order to validate the superiority of the task prototype compared to mean embedding, we performed ablation experiments on GLUE. The results are shown below.\n\n| Embedding Method | GLUE Score |\n| - | :---: |\n|Instance embedding+Mean embedding |  85.0 |\n|Task prototype|84.7|\n|Instance embedding+Task prototype (ours)|  **85.5** |\n\nCompared to directly taking the mean embedding vectors, the task prototypes lead to better performance and stability, with an improvement from 85.0 to 85.5.\n\nIn addition, removing the instance embedding design decreases the GLUE score from 85.5 to 84.7. The main reason might be the entanglement of samples from different tasks, as depicted in the t-SNE visualization. Under the influence of these entangled samples, task prototypes are unable to accurately capture task-specific information, resulting in negative transfer between different tasks during multi-task learning. **This indicates that the presence of instance embedding is necessary for better computation of task prototypes.**\n\n-------\n\nQ3: The experimental details for each setting need to be clarified. I can possibly guess that the results in Table 1 are obtained by training an 8-task adaptation for GLUE and a 4-task adaptation for SuperGLUE, and the results in Table 2 are obtained by training an 8-task adaptation for GLUE and fine-tuned with few-shot samples from BoolQ, CB, SciTail. But the explanation for these settings is obscure in the paper.\n\nA3: We performed multi-task learning on 8 datasets from GLUE and 4 datasets from SuperGLUE, yielding the results presented in Table 1. Further details are provided in line 380 of the paper and section 4.3. The details of the few-shot experiment are provided in line 463 of the paper and section 4.3. We sincerely thank you for the constructive comments which improved our paper's readability. While unable to modify the manuscript at present due to regulations, we will include a section in the appendix of the final version to offer additional insights into the experimental setup.\n\n--------\n\nQ4: According to Figure 1, the method proposed in this paper is applicable to encoder-decoder-like LMs. The performance of the approach is unpredictable when applied to language models with other architecture, e.g. decoder-only LMs.\n\nA4: Not all language model architectures have encoders. The utilization of an external, independent encoder has the potential to expand the applicability of our method to other architectures. We appreciate your insight into this valuable future direction, which involves the generalization of our framework to encompass broader architectures. However, this would require a significant systems-level effort, which we leave as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. This paper presents a rather concise approach to tackle the problem of multi-task adaptation for LMs.\n\nB. The PHA method described in this paper achieves clear improvements with fewer tunable parameters compared to previous methods.\n", "reasons_to_reject": "Weaknesses:\nA. According to Figure 1, the method proposed in this paper is applicable to encoder-decoder-like LMs. The performance of the approach is unpredictable when applied to language models with other architecture, e.g. decoder-only LMs.\n\nB. For Table 1, the improvement on the SuperGLUE benchmark looks marginal. The improvement on GLUE is larger, even though its absolute number is higher than SuperGLUE. Do the authors have an explanation of why it is the case?\n\nC. The motivation for introducing prototype for target tasks need further explanation. For example, whether it will work if we just replace the $k_i$ with the mean instance embeddings of the i-th task? The relation between task embeddings and instance embeddings needs further explanation.\n\nD. The experimental details for each setting need to be clarified. I can possibly guess that the results in Table 1 are obtained by training an 8-task adaptation for GLUE and a 4-task adaptation for SuperGLUE, and the results in Table 2 are obtained by training an 8-task adaptation for GLUE and fine-tuned with few-shot samples from BoolQ, CB, SciTail. But the explanation for these settings is obscure in the paper.", "questions_to_authors": "Please respond to the concerns listed in Weaknesses.", "paper_topic_and_main_contributions": "This paper presents a Prototype-based multi-task adaptation framework for tuning LM on downstream tasks, especially in the low-data regime. The instance-dense retriever facilitates matching between sample embeddings and prototype embeddings, and the HyperNetwork provides a fluent way of generating task-conditional adapters. Experiments on standard benchmarks show consistent improvements.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0UB2hQ6nUs", "reviewer": "Reviewer_Lu8U", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments. Our responses to your comments and questions are listed as follows.\n\n----\nnovelty — We view the relative simplicity of our method as a strength, as we believe simple but effective methods have the best chance of being widely adopted. In the field of NLP efficient methods, previous work often focused on parameter efficiency while overlooking sample efficiency. To our knowledge, no work has explored hypernetwork-based methods for improving the sample efficiency of language models. We introduced a new perspective for efficient learning of NLP: sample efficiency and proposed a new method to address multi-task learning in a sample-efficient manner. We believe our method and code, data, and checkpoints will be directly useful to the NLP community.\n\n-----\n\nQ1: 3.3 is more like a description of how to apply the proposed methods in multi-task settings.\n\nA1: Thanks very much for the valuable suggestions. While unable to modify the manuscript at present due to regulations, we will improve the organization in the final version to make it clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to follow. The proposed methods are well-motivated and are shown to be the most efficient ones compared to other methods.\n\nThe experimental studies are comprehensive and the proposed methods are evaluated on important benchmark datasets.", "reasons_to_reject": "The novelties of the proposed methods are not strong. Most used techniques have been proposed in previous works like INFONCE and HyperNetworks. Also, 3.3 is more like a description of how to apply the proposed methods in multi-task settings.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposed a prototype-based hyperAdapter for sample-efficient multi-task learning. The authors proposed a novel framework PHA based on adapter-tuning and hypernetwork. An instance-dense retriever and a prototypical hypernetwork are used to generate conditional modules in a sample-efficient way. The extensive performances demonstrate the effectiveness of the proposed methods. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oh0DXzbUwN", "reviewer": "Reviewer_9tvA", "summary": "", "questions": "Q: As shown in Figure 3, when the model size is small, there is a significant performance gap between the small number of samples (e.g. 1%, 3%,  5%) and 100% samples, which seems to challenge the authors' claim (i.e. PHA is a sample-efficient method) in this paper. In other words, the effectiveness of PHA is based on a strong backbone model, which reduces the applicability of the proposed method.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments. Our responses to your comments and questions are listed as follows.\n\n-----\n\nnovelty — We view the relative simplicity of our method as a strength, as we believe simple but effective methods have the best chance of being widely adopted. In the field of NLP efficient methods, previous work often focused on parameter efficiency while overlooking sample efficiency. To our knowledge, no work has explored hypernetwork-based methods for improving the sample efficiency of language models. We introduced a new perspective for efficient learning of NLP: sample efficiency and proposed a new method to address multi-task learning in a sample-efficient manner. We believe our method and code, data, and checkpoints will be directly useful to the NLP community.\n\n------\n\n\nQ1: As shown in Figure 3, when the model size is small, there is a significant performance gap between the small number of samples (e.g. 1%, 3%, 5%) and 100% samples, which seems to challenge the authors' claim (i.e. PHA is a sample-efficient method) in this paper. In other words, the effectiveness of PHA is based on a strong backbone model, which reduces the applicability of the proposed method.\n\n------\n\nA1: The performance difference between low-data and full-data adaption stems from the model itself and is not specific to our method. Small-scale pre-trained models contain limited knowledge due to their capacity limitations. With a small number of training samples, the model struggles to rapidly adapt by leveraging existing knowledge. That is why all methods we compared methods exhibit substantial performance differences when using small models.\n**Moreover, it is crucial to highlight that our method outperforms previous state-of-the-art multi-task learning methods, even under challenging low-resource conditions with limited model scale.** This precisely demonstrates the effectiveness and applicability of our method.\n\n------\n\nQ2: The experiment was insufficient, for example, the authors should test the significance of their experimental results. In addition, the current experiment only involves the NLU task, and testing on the NLG task would be useful, based on different backbone models.\n\nA2: Thanks a lot for this valuable suggestion! In light of your suggestion, we have applied our method to NLG tasks. Specifically, We conducted sample efficiency experiments on three datasets: Xsum, WMT16 Ro-En, and WMT16 En-Ro. Due to limited computation resources, we sampled 1% of the data and conducted experiments on three backbone models of different scales(small, base, and large). The results are shown below.\n\n|    Method(Small)    | XSUM(ROUGE2) | Ro-En(BLEU) | En-Ro(BLEU) | Avg |\n| - | :---: | :------:| :------:| :----: |\n| Finetuing |  7.2 | 24.1 | 20.1 | 17.1 |\n| Adapter |  2.9 | 25.1 | 9.7 | 12.6 |\n| PHA (ours) |  7.6 | 24.9 | 19.7 |**17.4**|\n\n|    Method(Base)    | XSUM(ROUGE2) | Ro-En(BLEU) | En-Ro(BLEU) | Avg |\n| -| :---: | :------:| :------:| :----: |\n| Finetuning |  10.9 | 25.3 | 24.2 | 20.1 |\n| Adapter |  8.2 | 26.5 | 19.5 | 18.1 |\n| PHA (ours) |  10.8 | 26.6 | 24.8 |**20.7**|\n\n|    Method(Large)    | XSUM(ROUGE2) | Ro-En(BLEU) | En-Ro(BLEU) | Avg |\n| - | :---: | :------:| :------:| :----: |\n| Finetuning |  12.3 | 24.7 | 29.7 | 22.2 |\n| Adapter |  12.6 | 27.1 | 25.9 | 21.9 |\n| PHA (ours) |  14.2 | 27.0 | 28.6 |**23.3**|\n\n**The results show that our method still achieves better performance on the NLG tasks.** While unable to modify the manuscript at present due to regulations, we will include a section in the appendix of the final version to add the NLG experiment results and analysis. \n\nIn addition, we show the results of the sample efficiency experiment conducted 5 times with different seeds. We will add the spread around the mean, i.e., the standard deviation, to our all experiment results in the revised manuscripts.\n\n|    Method(Base)    | samples/task | 1 | 2 | 3 | 4 | 5 | Mean | Std. |\n| -| :---: | :------:| :------:| :----: | :----: | :----: | :----: | :----: |\n| Finetuning|  1% | 71.9 | 73.3 |73.4|72.9|72.1| 72.9|0.61|\n| Adapter|  1% | 73.3 | 72.5 |72.1|73.0|71.9| 72.5|0.52|\n| PHA (ours) |  1% | 80.5 | 79.8 |79.6| 78.8 |79.1 |79.6|0.59|\n\n|    Method(Base)    | samples/task | 1 | 2 | 3 | 4 | 5 | Mean | Std. |\n| -| :---: | :------:| :------:| :----: | :----: | :----: | :----: | :----: |\n| Finetuning|  3% | 76.0 | 76.5 |75.9|76.8|75.5| 76.1|0.46|\n| Adapter |  3% | 74.9 | 75.1 |75.7|75.1|75.5| 75.3|0.30|\n| PHA (ours)|  3% | 82.2 | 81.9 |82.8|82.1|81.8| 82.1|0.37|\n\n|    Method(Base)    | samples/task | 1 | 2 | 3 | 4 | 5 | Mean | Std. |\n| -| :---: | :------:| :------:| :----: | :----: | :----: | :----: | :----: |\n| Finetuning  |  5% | 75.6 | 75.9 |75.5|76.0|76.1| 75.8|0.23|\n| Adapter  |  5% | 80.7 | 80.1 |80.5|80.2|81.1| 80.5|0.36|\n| PHA (ours) |  5% | 82.4 | 82.3 |82.6|82.4|82.3|82.4|0.11|\n\nThe results summarized in the table indicate that our method has stronger robustness and significant differences compared to previous methods.\n\nWe also thank the reviewer for their minor comments regarding clarity, typos, etc. We will address each of them in the revised manuscript.\n\n------\n\nQ3: Missing References\n\nA3:  Thanks a lot for this valuable suggestion, we will reorganize the related work of PEFT and cite the missed papers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well organized and easy to follow.\n\n2. The paper presents a new approach to addresse an important issue in multi-task learning.\n\n3. The paper conducts extensive experiments to demonstrate the effectiveness of the proposed method.", "reasons_to_reject": "1. The novelty of the methods is limited, for example, the methods (e.g. InfoNCE) used in this paper are relatively mature technologies in other fields. However, considering the different application scenarios, this combination is also acceptable.\n\n2. The experiment was insufficient, for example, the authors should test the significance of their experimental results. In addition, the current experiment only involves the NLU task, and testing on the NLG task would be useful, based on different backbone models.", "questions_to_authors": "Q: As shown in Figure 3, when the model size is small, there is a significant performance gap between the small number of samples (e.g. 1%, 3%,  5%) and 100% samples, which seems to challenge the authors' claim (i.e. PHA is a sample-efficient method) in this paper. In other words, the effectiveness of PHA is based on a strong backbone model, which reduces the applicability of the proposed method.", "paper_topic_and_main_contributions": "This paper proposes a new method to address multi-task learning in a sample-efficient manner. Specifically, based on adapter-tuning and hypernetwork, the authers propose Prototype-based HyperAdapter (PHA) framework to generate parameters of adapter layers conditioned on task-specific prototypes, and the specific prototype is retrieved and transferred to new tasks to be further tuned, which significantly improves the generalization capacity. In addition, extensive empirical experiments demonstrate that PHA outperforms previous methods.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "minor typo: \n1. \"fer-shot\"-> \"few-shot\" in Line621 \n2. \". but\" -> \". But\" in Line628\n3. \"Secondly, As\" -> \"Secondly, as\" in Line630"}, "extra_text": {"Missing_References": "The following recent PEFT papers should be considered for citation:\n\n1. Liao et al., Parameter-Efficient Fine-Tuning without Introducing New Latency. ACL 2023.\n2. Zeng et al., One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning. ACL 2023.\n3. Gui et al., HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation. ACL 2023.\n4. Xie et al., An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models. ACL 2023."}}]

Author rebuttal:

Meta-review: Reviewers agree on the quality of the evaluation and find the work exciting. Authors address most of the concerns/questions (if not all) raised by reviews and provide additional experiments and ablations. I recommend authors to improve the explanation of the evaluation procedure (i.e. how it differs from the setting used by ATTEMPT and MPT + single task fine-tuning) in the camera ready version.
Decision: Accept-Main