Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning
Authors: Haowen Wang
Tao Sun
Congyun Jin
Yingbo Wang
Yibo Fan
Yunqi Xu
Yuliang Du
Cong Fan
Venue: ICLR 2024 poster (2024)
Keywords: Modular skill learning
Multi-task learning
Parameter-Efficient
Fine-Tuning
Primary area: generative models
TL;DR: A novel paradigm of Parameter Efficient Fine-Tuning (PEFT) for multi-task learning, harnessing specialized and shared domain skills.

Abstract: Modular and composable transfer learning is an emerging direction in the field of Parameter Efficient Fine-Tuning, as it enables neural networks to better organize various aspects of knowledge, leading to improved cross-task generalization.
In this paper, we introduce a novel approach Customized Polytropon ($\texttt{C-Poly}$) that combines task-common skills and task-specific skills, while the skill parameters being highly parameterized using low-rank techniques.
Each task is associated with a customizable number of exclusive specialized skills and also benefits from skills shared with peer tasks. A skill assignment matrix is jointly learned. To evaluate our approach, we conducted extensive experiments on the Super-NaturalInstructions and the SuperGLUE benchmarks.
Our findings demonstrate that $\texttt{C-Poly}$ outperforms fully-shared, task-specific, and skill-indistinguishable baselines, significantly enhancing the sample efficiency in multi-task learning scenarios.

Published as a conference paper at ICLR 2024
CUSTOMIZABLE COMBINATION OF PARAMETER- EFFICIENT MODULES FOR MULTI-TASK LEARNING
Haowen Wang; Tao Sun; Congyun Jin, Yingbo Wang Yibo Fan, Yunqi Xu, Yuliang Du, Cong Fan
Ant Group, Shanghai, China
(wanghaowen. whw, suntao. sun, jincongyun. joy, wangyingbo wyb, fanyibo. fyb, xuyunqi.. xyд, duycliang. fancong. fan) (aal://attggopooooo com
ABSTRACT
Modular and composable transfer learning is emerging direction in the field Parameter Efficient Fine- TuninT as it enables neural networks to better orga- nize various aspects of knowledge, leading to improved cross-task generalization. In this paper, we introduce a novel approach Customized Polytropon (C-Poly) that combines task-common skills and task-specific skills, while the skill param- eters being highly parameterized using low-rank techniques. Each task is associ- ated with customizable number of exclusive specialized skills and also benefits from skills shared with peer tasks. A skill assignment matrix is jointly learned. To evaluate our approach, we conducted extensive experiments on the Super- NaturalInstrrutionn and the SuperGLUE benchmarks. Our findings demonstrate that C PPlly outperforms fully-shared, task-specific, and skill-ndditinguishabbl baselines, significantly enhancing the sample efficiency in multi-tass learning sce- narios.
INTRODUCTION
As the number of parameters in Large Language Models (LLMs) continues to grow, training these models efficiently with limited computational resources has become challenge. recent years, there has been a shift towards employing Parameter Effective Fine-Tuning Tuning (PEFT) methods to ad- dress this issue. Examples of such methods include LoRA (Hu al., 2022). AdaLoRA (Zhang al. 2023a), and (IA) (Liu et al.) 2022a). These methods focus fine-tuning the adapter while freezing the pre-trained model, effectively reducing the computational cost. By selectively updating only a portion of the model parameters, PEFT methods enable efficient training and utilization of large foundation models. This line of approaches allows for more effective use of resources while maintaining the performance of the pre-trained model on downstream tasks (Hu et al. 2022). How- ever, despite the popularity and widely adoption of PHFT methods, the learning effectiveness of such methods, especially in multi-task scenarios, is under explored.
LLMs are famous for their extraordinary capabilities on solving multiple tasks in zero-shot or few- shot manners (Brown et 2020). Basic PEFT methods mentioned earlier don't 't take the multitask essence of real-world data into account and rely heavily on the base foundation model's 's capacities on the multitask generalization. Building upon the basic PEFT methods, various training approaches designed for Multi- Task Learning (MTL) have been proposed (Pfeiffer et al.) 20200 Vu et al.) [021]]
Asai et al. 2022 Chronopoulou et al. 2023 Zadouri et al. (023). One simple solution is to perform multitask training by training large model on combination of multiple tasks. This involves training the model on the union of training tasks and subsequently evaluating its performance on different testing tasks (Ye al.] 2021 Liu et al.. 2022a). However, this approach overlooks the relationships between the tasks and is vulnerable to negative transfer, where the gradients associated with different tasks are misaligned (Wang et al. 2020). This misalignment of gradients lead to sub-optimal performance and hinder the effective utilization of learned knowledge across tasks.
"qqual contribution. Corresponding author.

---

Published as a conference paper at ICLR 2024
To enhance sample efficiency, MoLoRA Zadouri al. 2023) has been introduced. MoLoRA successfully applies the Mixture-of-Exper (MoE) architecture to PEFT methods and improves the model'ss generalization capacity across various tasks by jointly learning multiple LoRA instances. Mol.RA views each LoRA as a lightweight expert, following MoE framework, and thus allows for more specialized adaptation different tasks from shared knowledge learnt through parallel instances.
In recent work, Ponti et (2022) developed Polytropon (Poly) tackle the challenges associ- ated with multitask learning. The central idea of Poly is to consider each task-specific adapter as composition of reusable skillset of basic adapters or modules. Specifically. Poly jointly learns an inventory of adapters (for example, LoRA) and a simple routing vector that selects and combines variable-size subset of adapters for each individual task. This approach significantly improves the efficiency of sample sharing and utilization between multiple tasks. Although not mentioned explic- itly in the original work (Ponti et al.) [2022). we argue that each adapter in Poly can be viewed as lightweight expert and then the whole structure follows the same pattern as Multi-gate Mixture- of-Experts (MMoE) (Ma et al. 2018), a well-known MTL framework. To noted, in Ponti et al. (022 also introduced structure called MoE-LoRA, which is simplified version of MoLoRA where the routing is controlled by function of hidden states instead.
Based on Poly, subsequent research titled Multi-Head Routing (MHR) (Caccia et al. 2022) was proposed. MHR extends the concept of parallel thinking to low-rank decomposition and introduces further improvements the basic unit of the adapter. By leveraging low-rank decomposition, MHR enhances the model's ability to generalize across different tasks. The parallel improvements in the adapter's basic units allow for more efficient adaptation different tasks while still benefiting from shared knowledge.
C-POLY
Embedding
Customized Polytropon
Pretrained Weights
W R pdd
Input
Task-Common Part
Saec-Specific Part
Figure 1: Overview of Customized Polytropon (C- (C-Poly) framework
Inspired by Customized Sharing Control (CGC) and Progressive Layered Extraction (PLE) TTng et al. 2020), our research has made additional improvements over Poly. We assume that in MTL, many tasks do share transferable knowledge, while each of them requires discriminative abilities. Based on this assumption, we explicitly divide modular skills into task-common skill modules and task-specific skill modules. We propose Customized Polytropon (C-Poly), where for each task, two components are learnt jointly, task-specific skill module and task-common skill module. This allows each task to be characterized by not only a shared task cosk-common module as in Poly and MHR, but also unique subset of skills, mitigating the effect of negative transferring and leading to improved multi-task performance. Furthermore, C-Poly promotes interpretability by learning an explicit hierarchy of tasks based the skills they select, which provides insights into the relation-
2

---

Published as a conference paper at ICLR 2024
ship between tasks and the important skills needed for each task. Overall, approach enhances the efficiency and effectiveness of modular skill multi-task learning, enabling better performance, interpretabilityy and resource utilization.
In Section 3] we conducted extensive experiments on multiple datasets and model architectures. The results of these experiments have consistently demonstrated our method C-Poly has surpassed the performance of the conventional PEFT methods and achieved state-of-the-an (SOTA) results in different multitasking scenarios.
2
METHODOLOGGY
The proposed unified MTL framework C-Poly is shown in Figure which aims to enhance sample efficiency for each task by leveraging strengths from all other tasks while keeping task- speci-ic abilities. Suppose there are 7 tasks and for each task, task-specific data input x², {1, 2, T).
The MoE-like structure consists of adapter modules (or experts), + (ń), 62. ........ ) and each adapter o; can be regarded as function of the input data æ². The major improvement C-Poly to explicitly categorize adapter modules O into two separate parts:
A task-common skillset: + P { 02, .e,,....,..,.. AA) having adapters.
A task-specific skillset: & {di 02 ..... ΦBB having B adapters for each task t.
In total, there are | A| + x ( A T x adapters. discussion, would set B to keep only one task-specific adapter in the following experiments.
simplified yet generalizable
The combined output of the C-Poly adapter modules for each task input x' can be expressed in Equation D with w; representing the learnable weight of each adapter's output.
B Σ
Σ o wloi((")
wh u
=
Σ wx))(x") + w w'd'(x")
(I)
Task-Common
Task-Spocific
In the task-common part, the set of adapters di are shared across all tasks, while the weights w are exclusive to each task t. In contrast, both the weights wi and adapters of are customized for each individual task t in the task-specific part.
Following the notation above, various MoE-like PEFT structures can be mathematically formulated together Table 1 Both the MoE and MMoE models only consist of the task-common part of Equation T in the conventional MoE approach, tasks are not differentiated, leading to shared pa- rameters across all tasks; the MMoE framework assigns task-specific weights for each individual task, while still maintaining a shared pool of experts or adapters.
Table 1: Comparison between different MoE-like PEFT methods
MoE Structures
PEFT Methods
Task Output
MoLoRA Zadouri al.
2023
Conventional MoE
werc((")
MoE-LoRA Pont et al.
2022
Poly (Pont et al. 2022
MMoE
Ma et al.
2018
i (x*)
MHR
Caccia et al. 2022
1=1
CGC
Tang et al.
2020
Our Method C-Poly
Σ w(o)(x")) w/ w/ (xx")
PLE
Tang et al.
2020
In C-Poly, the weights associated with each adapter can be represented together as one allocation 1. This matrix be further decomposed into distinct
matrix W (Tx(A++) when
3

---

Published as a conference paper at ICLR 2024
components: W
R Txx
and Wa RTXT:
W
WA
WB
w
0
0
W1
A
0
10
0
(3)
w
W2
To optimize the learning process in skill acquisition, we have employed different learning methods for each component of the allocation matrix. Additionally, we have incorporated low-rank approxi- mations to further enhance the parameter efficiency.
2.1 TASK-COMMON SKILLS LEARNING
Task-common skills are universally applicable skills that all tasks can leverage. Previous research has focused on identifying the effectiveness of general skills modules for specific tasks by employing modular concepts at the structured input level inspired by cognitive mechanisms (Bengio) [2077]
Ponti et al. 2022 Caccia al. 2022 Zadouri et al. 2023). This concept has been translated into Softmax for cross-module or top-k selection practical implementation.
Therefore, following et al. (2022) and Caccia et al, (2022), we utilize task-common allocation matrix WA E {0,1) with uniform initialization. This matrix is employed to achieve the soft partitioning of general skills. Each element w in WA is a binary value that indicates whether
particular task activates the adapter module Φi. However, since discrete binary matrices like WA non-differentiable, learning cannot be accomplished through gradient descent. To overcome this limitation, we adopt the Gumbel-sigmoid approach (Maddison et al. 2016 Jang et al. (2016), which allows us obtain set of continuously relaxed Bernoulli distributions. This approach guarantees both randomness and the ability to perform differentiable sampling:
owwju
w
log (1 (1- ((11)
(4)
2.2 TASK-SPECIHIC SKILLS LEARNING
Specialized skills refer to modular skills that acquire the distinctive attributes of each task. In com- plex and interconnected multitasking scenarios, a secsaw phenomenon commonly arises. In multi- tasking learning mode, there is often trade-off between enhancing specific tasks' effectiveness and compromising others' effectiveness (Wang et al., 2020). To address this, we explicitly differenti- ate between shared and exclusive skill modules. This separation allows us to amplify the inherent characteristics of individual tasks.
We initialize the task tasc-spe skill allocation Wn T unit diagonal matrix. Notably, during the actual training process, although only the diagonal of WB are weights we concerned, entries off the diagonal solving the current task, the exclusive specialized skills of other tasks can be leveraged without influencing the parameter values of specialized skills for those tasks.
also subject potential updates. These side-effects indicate that while
2.3 PARAMETER EFFICIENCY
We accomplished an efficient parameterization of skill modules by employing low-rank techniques. Every adapter in our C-Poly experiments is Low-Rank Adapter (LoRA) (Hu et al. (002). LoRA straightforward yet effective structure specifically tailored for Transformer- baaseoomme--bbbedddddd models (Vaswani ctt al., 2017). The idea behind LoRA is relatively straightforward. LoRA decomposes each weight matrix of the linear transformation in Transformers into the multiplication of two low-rank matrices. In other words, the linear projection f Rdd - ddd can be represented as follows, disregarding the bias part:
h hy W AW h1 Wi + W Woonn W Wup)
(5)
4

---

Published as a conference paper at ICLR 2024
Instead of AW E Raxd two much smaller matrices Wdown E and W up d, are obtained through gradient descend optimization. Through the adoption of LoRA, updating each linear layer in the model only requires 2 x r x d parameters in the calculation, opposed to the original d x d parameters. This results in a notable enhancement in parameter efficiency, enabling faster training even with limited computing resources.
R with <
As examined in Hu et al. (2022). LoRA can be applied to various components of Transformers, such as query, key, value, and ferd-forward layers, while the choice of rank does not hold significant importance. This suggests that LoRA exhibits versatility in its applicability. In our experiments, we
patched all query, key, and value layers with C-Poly, a combination of multiple LoRAs.
EMPIRICAL EXPERIMENTS
3.1 EXPERIMENTAL. SETUP
In order to evaluate the efficacy of our proposed unified MTL framework C-Poly, which incorpo- rates both task task-common and task-specific skills, we conducted experiments on two publicly available multitasking benchmarks: SuperGLUE (Wang et al. 2019) and Super Intural-Instructionn (Super NI) (Wang et al. 2022). The SuperGLUE benchmark is widely acceptable benchmark for evaluat- ing general-purpose language understanding. In our experiments, 7 distinct tasks were selected from the benchmark that can be effectively evaluated using the accuracy metric. The Super NI dataset, as a meta-datase (Triantafillou et al. 2019), covers a wide range of 76 distinct task types within the field of natural language processing and comprises over 1,600 diverse NLP tasks. During the experiments, 100 tasks were randomly selected, and for each task, 1000 samples were randomly selected for training and another 100 were selected for evaluation purpose. To ensure comparability. our sampling method follows the identical approach as described in Ponti et al. (2022). To evaluate the effectiveness of the trained model, we employed various metries for all selected tasks, including Exact Match (EM) and Rouge metrics (Lin 2004), including Rouge- Rouge-L, L, and Rouge-LSum.
To verify the universal effectiveness of our multitasking learning approach C-Poly, we chose T5 Version 1.1 LM Adapted (T5) (Raffel et al. 2020), FLAN-T5 (Chung et al. 2022) and GLM (Du
et al. 2021 as the base models.
In our study, we thoroughly compared our proposed approach, C-Poly, and several existing MoE- like PEFT methods. The methods we compared against include LORA, MoE-LoRA, Poly and MHR. The comparison allowed us to analyze and evaluate the performance and effectiveness of our framework relative to these established PEFT methods.
3.2 TRAINING DETAILS
In our experiments, we applied PEFT methods to all query, key, value matrices within every attention layer in the base models. In the case of vanilla LoRA, we set the rank of the low-rank approximation, r == For all MoE-like tuning methods, we utilized in total 4 parallel LoRAs (experts) with r = 2. In C-Poly, we set A = 3 LoRA for task-common skills and B LoRA for task-specific skills. This decision was made to ensure a comparable number of training parameters across all methods.
We trained our model with cross entropy loss for only 1 epoch, and set batch size of 4 on both Super NI and SuperGLUE datasets during training. The Adam W optimizer (Loshchilov & Hutter 2017) was used, with learning rate of 5e -3 We also employed the linear decay strategy (Loshchilov & Hutter 2016) as the learning rate scheduler with a weight decay of 0.01 and a warmup ratio of 0.06. All experiments were conducted on a single NVIDIA Tesla A100 graphics card.
3.3
MAIN RESULTS AND DISCUSSION
3.3.1
ANALYSIS OF BALANCED MULTITASK LEARNING
In Figure 2 we present a comparative analysis of various fine-tuning methods across multiple tasks within the SuperGLUE benchmark. When tuning with full parameters (FT), the overall average accuracy is the lowest among all approaches because of relatively poor performance in the MultiRC sub-task. This phenomenon, known as the seesaw effect, manifestation of the negative transfer
5

---

Published as a conference paper at ICLR 2024
95
90
LoRA
MOE-LoRA
85
Poly
Accuracy
80
MHR
Our Method
75
70
65
60
55
AVERAGE
BoolQ
CB
COPA
MuliRC
RTE
WiC
WSC
SubTasks in SuperGLUE
Figure 2: FLAN-T5-Large with different PEFT methods on SuperGLUE benchmark, compared with Full Fine-tuning (FT). We reported overall averaged (AVERAGE) and task-specific accuracy for all sub-tasks.
problem when tackling multiple tasks concurrently, holds significant importance in the domain of multi-task learning (Pan & Yang 20100) Sun et al. [2017] Tang et al. 2020). Our method C-Poly, on the other hand, demonstrates constant improvement over all sub-tasks thanks to the task-specific skill learning module. The results reveal that C-Poly can effectively mitigate the negative transfer and seesaw effect issues. As shown in Appendix A.4 conducted experiments FLAN-T5-X: (2B) and found that the base model with larger amount of parameters has a stronger fitting ability for multiple tasks. Consequently, the experimental results showed that method can bring stable improvements in reducing negative migration.
Table 2: FLAN-TS-Large and GLM-10B with different adaptation methods on the SuperGLUE benchmark. We report the overall (matched and mismatched) accuracy for BoolQ, CB, COPA, MultiRC, RTE, WiC and WSC. Higher better for all metrics.
Base Model
PEFT Method
AVG
BoolQ
CB
COPA
MultiRC
RTE
WiC
WSC
LoRA
81.85
81.85
85.71
90.00
82.78
85.92
59.56 64.42
MOE-LoRA
82.31
85.14
87.50
91.00
83.46
86.64
57.99 66.35
FLAN-T5-Large
Poly
82.09
85.11
87.50
90.00
82.53
86.28
60.66
76.92
MHR
82.31
85.02
87.50
90.00
82.96
86.28
61.13
75.96
Our Method
83.21
85.69
85.71
90.00
83.35
88.81
67.08
75.00
LoRA
52.05
60.98
46.38
65.70
62.43
57.37
39.15
32.32
MoE-LORA
53.86
63.31
45.02
63.41
64.01
61.22
40.35
39.66
GLM-10B
Poly 56.99
64.65
52.
65.54
65.66
62.15
41.71
47.08
MHR
56.92
64.85
50.79
66.36
65.79
62.75
42.35
45.58
Our Method
62.26
67.31
60.38
70.04
67.90
68.01
48.71
53.42

---

Published as a conference paper at ICLR 2024
3.3.2 EVALUATION ON MORE MODELS AND MORE TASKS
The effectiveness of different PEFT methods evaluated two architectures: T5 and GLM. Table 2 presents the performance of these methods on dataset consisting of tasks from the Su- perGLUE. We display the evaluation results for the sub-tasks individually, as well as their average performance. Table [ compares the performance of different PEFT methods on the SuperNI. The indicators in the table represent the average performance evaluation over 100 tasks.
Table 3: FL-N-TS-Large, Ta-Large and GLMM-0B with different adaptation methods on the 100 randomly selected tasks from SuperNI dataset. We report the average Rouge-1, Rouge-L, and Rouge-LSum for all tasks. Higher is better for all metrics.
Base Model
PEFT Method
Rouge-1
Rouge-I.
Rouge-LSum
LoRA
68.26
67.42
67.42
MoE-LoRA
68.59
67.76
67.75
FLAN-TT--arge
Poly
68.45
67.60
67.58
MHR
68.84
67.77
67.78
Our Method
68.69
67.80
67.82
LoRA
34.16
33.64
33.65
MoE-LoRA
36.82
36.13
36.15
T5-Large
Poly
43.04
42.05
42,09
MHR
44.24
43.32
43.34
Our Method
49.34
48.50
48.51
LoRA
43.16
42.04
42.09
MoE-LoRA
45.97
44.79
44.89
GLM-10B
Poly 47.96
46.80
46.80
MHR
48.53
47.34
47.33
Our Method 49.53
48.45
48.45
The results are two-folded. Firstly, it highlights that MoE-LoRA consistently demonstrates improve- ment over LoRA, attributed to the enhanced parameter flexibility from the MoE structure. We com- pared FLAN--T5-Lrrg T5-Large, and GLMM-0B, and the results showed that it can significantly alleviate the phenomenon of negative migration in both architectures and improve the flexibility of adaptive parameters compared to LORA and Po Poly. y. In the comparison of FLAN-T5-Lare,, although has undergone a large number pre- pre-training with similar instruction samples, our method can still bring some improvement. Moreover, in Appendix A.2 we conducted the ablation experiments on a broader range of models including T5-Large (0.78B), T5-XL (3B), and stronger one T5-XXL (IIB), as well their FLAN-T5 counterparts, and the results showed that methods remained robust and significant as model parameters increased.
Secondly, the results demonstrate that method C-Poly y has achieved optimal performance in both architectures and on the SuperGLUE and SuperNI datasets. The ablation experiments in Ap- pendix A.I on the number of tasks (10-50-100) conducted on GLM-10B and FLAN-T5-Large indi- cate that our method is still significantly effective when the number of tasks increases. The explicit separation of task aask-specific skills and fask-common skills in our design enables the skill modules to capture task-specific differences while sharing abstracted general skill modules effectively. Due to explicit separation, the negative transfer phenomenon has been significantly reduced as in Tang et al. (2020), which can be verified both datasets and becomes more pronounced as the number of learning tasks increases.
3.3.3 PARAMETER EFFICIENCY ANALYSIS
In Figure 3 we assessed PEFT methodologies across varying parameter magnitudes, gauging their efficacy the SuperGLUE benchmark with FLAN-T5-Large
the base model. Furthermore, we compared their performance against that derived from full parameter fine-tuning. It is evident that approach attains better performance among all PEFT methodologies with identical scales of parameters and training epochs. Notably, C-Poly even outperforms other methods with more parameters. Our methods explicitly segregate shared and proprietary skills and effectively ensure

---

Published as a conference paper at ICLR 2024
parameter efficiency in multi-task learning under equivalent training settings. This trend is also observed in the SuperNI dataset evaluation. As a result, we argue that adopting both task-common skills learning and task-specific skills learning paradigm constitutes robust strategy for achiev- ing parameter efficiency in adapting multiple tasks.
83.9
MOI-LeRA
Poly
MIlR Mahhed
Accuney
81.5
80.5
1M
10M
Adaptation Paramezers
Figure 3: Accuracy of PEFT methods and Full Fine-tuning on SuperGLUE benchmark when using Ta-Large as the base model. The X-axis shows the number of trainable parameters.
3.3.4 DEEPER INSIGHTS INTO C-POLY
As outlined in Section 3.3.2 our dual-skill framework, C-Poly, consistently delivers robust en- hancements across various architectures, task scales, and model sizes. Here, we try to explore the intrinsic attributes of C-Poly y that drive these performance gains.
Explicit skill separation enhances knowledge sharing Task-specific skills enable the task- common component to focus on task similarities, while they manage the distinct elements of each task. In Appendix A.5 we conducted deep analysis on the task-common allocation matrix WA- Figures 5 and 7 demonstrate WA of C-Poly in certain layers GLM-10B for SuperNI-10, SuperNI-50, and SuperNI- 100 respectively. After proper normalization, these matrices show clear differences in skill allocation for different tasks. Additionally, we performed task clustering based the skill allocations learned in all layers of the GLMM10B model trained SuperNI-100 with and without the task-specific component (i.e., comparing C-Poly y and Poly). The clustering out- comes are shown as dendrograms Figure and which suggest the enhanced performance of C- C-Poly is due to a more balanced task hierarchy. facilitating more effective knowledge transfer among similar tasks and improved distinction of unrelated ones.
Equilibrium in skill allocation is crucial The introduction of C-Poly prompts the need to balance task-common and task task-specific parameter allocation. We must establish the optimal and values. In Appendix A.3] we explore various (A, B) pairings, maintaining a fixed total parameter count and confirm that -ask-specific skills significantly boost model performance. However, allocating enough parameters to task-common skills is crucial for shared knowledge acquisition. Excessive focus on task-specific parameters can impede the learning process, potentially causing overfitting and hindering the model's ability to recognize similarities across tasks, which may lead to suboptimal performance. A balanced parameter distribution promotes clear task distinction and helps prevent overfitting, preserving the model's generalization capabilities. Identifying the ideal parameter ratio and configuration is critical aspect for C-Poly and may vary depending on the nature of the tasks and datasets involved.
8

---

Published as a conference paper at ICLR 2024
4 RELATED WORKS
4.1 PARAMETER- FINE-TUNISS
Numerous researchers have proposed incorporating adapters within neural networks, strategically placed between existing layers, and enforcing weight constraints on these adapters. LoRA (Hu al., 2022). for instance, advocates for fine-tuning the model by learning low-rank matrix weights and aligning them with sovereignty. Building upon LoRA, (IA)² (Liu et al. 2022a) offers further enhancements by introducing a relatively modest number of novel parameters. the context of model adjustment, Prefix Tuning (Li & Liang 2021), along with Prompt Tuning (Lester et al. 2021) and P- Tuning (Liu et al. 2022b), emerges as a technique that exclusively optimizes a small segment of continuous task-specific vectors, thereby strengthening downstream task optimization.
4.2 MODULAR MULKI-TASK LEARNING
Modular neural network architectures (Jacobs et al. 1991a) offer benefits like positive transfer, compositionality, and parameter efficiency (Zhang et al. 2023b). They consist of modules (which represent skills that can be combined and updated independently), routing function (to select mod- ules per task or example with variants like fixed, learned hard, or learned soft routing (Rosenbaum) al. 2017: Jacobs et al., 1991b, Fernando et al., 2017). and an aggregation function (lo merge outputs of active modules, often as a learnable network). Notably, within the realm of NLP, Fedus et al. 2021 successfully extended the pre-training of large language models to trillions of parame- ters by leveraging the MoE architecture. Previous studies (Rajendran et al.) [0155] (onti et al. 2022] Kingetsu et al. 2021] Kudugunta et al. (00I]) have explored approaches enforce parameter reuse and modularity in multitasking learning. Rajendran et al.] (2015) trained individual modules for each task and subsequently learned how to reuse these modules.
4.3 LANGUAGE MODELS
The Transformer architecture serves as fundamental framework for sequence pair modeling. Building upon this foundation, [Radford Narasimhan (2018) employed a stack of Transform- to effectively model autoregressive languages through the deployment of encoders and decoders. BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019 are classic text modeling methodolo- gies, both relying Transformer units pre-traine mre-trained only architecture model is particularly suited for comprehension-basse tasks benefit from both encoder-decoder and decoder-only architecture models due their auto- regressive nature (Fu et al. 2023 Sarrouti et al. 2022).
vast amounts of textual data. The encoder-
tasks, whereas generative
5 CONCLUSION
In this article, we introduce a novel paradigm for PEFT called Customized Polytropon C-Poly. By explicitly distinguishing between task-common and task-specific skills, method enables efficient multi-task fine-tuning on large language models, even with limited computational resources. Our approach addresses the challenge of resource limitations and allows for efficient training. The sepa- ration of exclusive and general skills effectively mitigates the seesaw problem and negative transfer commonly encountered multitasking learning, leading to superior overall performance, which also offers compelling interpretability. Evaluations on various benchmark demonstrate the effectiveness of the proposed method, which surpasses existing PEFT baselines and achieves state-of-the-ar per- formance. These results highlight the potential and significance of our unified multi-task learning framework C-Poly in the field of parameter- efficceent multi-tast task learning.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Akari Asai, Mohammadreza Salehi, Matthew E Peters, and Hannaneh Hajishirzi. Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6655-6672, 2022.
Yoshua Bengio. The consciousness prior. arXiv preprint arXiv: met.::709000050 2017.
Tom B. Brown, Benjamin Mann, Nick Ryder. Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, fbss/2005.14165 2020. URL https: http:p::/aaxxi..ooo org wwtp:/wwww...////.....22222 abt.aaa.22...66.........
Lucas Caccia, Edoardo Ponti, Lucas Liu, Matheus Pereira, Nicolas Le Roux, and Alessandro Sor- doni. Multi-head adapter routing for data-efficient fine-tuning arXiv preprint arXiv:2211.0388) 2022.
Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:230..07027. 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghami, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetueed language models, 2022.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint (0t.:2/003.03600 2021.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 2017.
Zihao Fu, Wai Lam, Qian Yu, Anthony Man-Cho So. Shengding Hu, Zhiyuan Liu, and Nigel Collier. Decoder-only encoder-decoder? interpreting language model as regularized encoder- encoder-decoder, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference on Learning Representations, 2022. URL. https: ftttp:://openreviiee .et. forum?
id=nZevKeeFyf9
Robert A Jacobs, Michael Jordan, and Andrew G Barto. Task decomposition through competition in a modular connectionist architecture: The what and where vision tasks. Cognitive science, 15 (2):219-250, 1991a.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991b.
10

---

Published as a conference paper at ICLR 2024
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softnax. arXiv preprint arXiv:1611.0144, 2016.
Hiroaki Kingetsu, Kenichi Kobayashi, and Taiji Suzuki. Neural network module decomposition and recomposition, 2021.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh- Minh-Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-cxperts for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3577-3599, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.1886//11222111111 1666:////222...nnnneeeemm////// findingg-emmlp.3004. URL battps://aclantbone wtg.:/gyy..gg/221/
Findinge emn 3334
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. http:///2....mmmmmi..222222 URL https: //acl acl antho ogy. wwwoorg/2222 .ww.021..mmnm... p- ma i n. 243
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. N0.18853/v1/2222 1a06:///12222...aoooo....5535
URL http:///ccaatthooggg.ooooo
acl -one-llog..... 353
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances Neural Information Processing Systems, 35: 15:0999-1965 2022a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du Zhilin Yang. and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 61-68, Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi: 10.08865/w1/2022.acl
URL
https::/aaet antho ogy www.org///o 2022 -cl.-sorrl.hh.......
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608: 1008033,,,.,,83, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1111.05100 2017.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task rela- tionships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '18, pp. 1930-1939, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450355520. doi: 10.1145//2198119322200
URL Bxttps://@cl.org//10. 1145/
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.007/1 2016.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. 12(10311345-135, 2010.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.002 00247, 2020.
11

---

Published as a conference paper at ICLR 2024
Edoardo M Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining modular skills in multitask learning. arXiv preprint arXiv:2202.13914. 2022.
Alec Radford and Karthik Narasimhan.
Improving language understanding by generative pre-
training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (111):485-5555 2020.
Janarthanan Rajendran, Aravind Srinivas, Mitesh M. Khapra, P Prasanna, and Balaraman Ravindran. Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain, 2015.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning, 2017.
Mourad Sarrouti, Carson Tao, and Yoann Mamy Randriamihaja. Comparing encoder-only and encoder-decoder transformers for relation extraction from biomedical texts: An empirical study on ten benchmark datasets. In Proceedings of the 21 Workshop on Biomedical Language Pro- cessing, pp. 376 382, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ 1ttp5/3////2222.oooo... 1.37. URL ptt.ps://aclanthooooggooo org 2022 bionlp-1.
37
Shengyang Sun, Changyou Chen, and Lawrence Carin. Learning structured weight uncertainty in bayesian neural networks. In International Conference on Artificial Intelligence and Statistics, 2017.
Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In Proceedings of the 14th ACM Conference on Recommender Systems. RecSys '20, pp. 269-278, New York. NY, USA, 2020. Association for Computing Machinery. ISBN 478145037583.. 11......111131311111
doi: 10.1145/3383313.
3412236. URL
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Car- les Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. CoRR, abs/ htss/1933.03066, 2019. URL
btrrg/bbe abs 9033..... 03096
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31 Inter- national Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint orXiv:2/10.079000 2021.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mi- rali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Pa- tro, Tanay Dixit, and Xudong Shen. Supper-aturallnsstuutiooss Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5085-5109, Abu Dhabi, United Arab Emirates, December
12

---

Published as a conference paper at ICLR 2024
2022. Association for Computational Linguisties. doi: 10.18653/w122022.mmmlm... 340.. URL attpps///acammhhoooo..og///////2 een1222n.nlp-mmn.. 340
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating
and improving multi-task optimization in massively multilingual models. arXiv preprint arXiv:2010.05874, 2020.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. arXiv preprint arXiv:2104.08833 2021.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Push- ing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning, 2023.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint vrXiv:2303.10522 2023a.
Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Jie Zhou. Emergent modularity pre-trained transformers, 2023b.
13

---

Published as a conference paper at ICLR 2024
A
ADDITIONAL RESULTS
A. ABLATION ON THE NUMBER OF TASKS
To understand how approach scales with the number of tasks, performed experiments with varying task numbers 10, 50, and 100 from SuperNI dataset. Table Table and 6how the performance of Ta-Large, WannTT--Laree and GLM-10B models respectively. These results affirm that our method C-Poly is highly effective and scalable across various task numbers. Our approach shows only superior performance in lower task settings but also maintains its competitive edge as the number of tasks increases. It is worth mentioning that FLAN-TS-Larg is trained on FLAN dataset, and it may have some similarities or overlaps with the SuperNI dataset, which makes its performance relatively high.
Table 4: Performance of T5-Large on different numbers of tasks from SuperNI dataset. We report the average Rouge-1, Rouge-I., and Rouge-I.Sur .Sum for all tasks. Higher is better for all metrics. # of Tasks
PEFT Method Rouge-1
Rouge-L
Rouge-LSum
LoRA
14.22
14.12
14.19
MoE-LoRA
16.75
16.71
16.76
10
Poly
17.34
17.31
17.38
MHR
17.17
17.11
17.16
Our Method
42.62
42.47
42.60
LoRA
32.58
31.64
31.59
MoE-LoRA
39.50
38.47
38.46
50
Poly
46.13
44.25
44.28
MHR
47.27
45,43
45.39
Our Method
53.39
51.68
51.63
LoRA
34.16
33.64
33.65
MoE-LoRA
36.82
36.13
36.15
100
Poly
43.04
42.05
42.09
MHR
44.24
43.32
43.34
Our Method
49.34
48.50
48.51
Table 5: Performance of FLAN-TS-Large on different numbers of tasks from SuperNI dataset. We report the average Rouge-1, Rouge-L, and Rouge-LSum for all tasks. Higher is better for all metries.
# of Tasks PEFT Method
Rouge-1
Rouge-L.
Rouge-I.Sum
LoRA
67.82
67.01
67.03
MoE-LoRA
67.95
67.12
67.15
10
Poly
68.10
67.29
67.33
MHR
77.49
77.25
77.36
Our Method
77.73
77.58
77.61
LoRA
70.66
69.10
69.03
MoE-LoRA
70.81
69.25
69.21
50
Poly
70.76
69.23
69.15
MHR
70.92
69.39
69.33
Our Method
71.17
69.68
69.62
LoRA
68.26
67.42
67.42
MoE-LoRA
68.59
67.76
67.75
100
Poly
68.45
67.60
67.58
MHR
68.84
67.77
67.78
Our Method
68.69
67.80
67.82
14

---

Published as a conference paper at ICLR 2024
Table 6: Performance of GLM-10B different numbers of tasks from SuperNI dataset. We report the average Rouge-1, Rouge-L, and Rouge-LSum for all tasks. Higher is better for all metrics # of Tasks
PEFT Method
Rouge-1
Rouge-I.
Rouge-L.Sum
LoRA
30.64
30.40
30.42
MoF-LoRA
33.92
33.79
33.77
10
Poly
34.53
34.41
34.31
MHR
33.63
33.47
33.47
Our Method
43.74
43.72
43.65
LoRA
34.16
33.00
32.98
MoE-LoRA
39.87
38.63
38.55
50
Poly
44.81
43.09
43.07
MHR
45.32
43.62
43.56
Our Method
53.17
51.27
51.32
LoRA
43.16
42.04
42.09
MoE-LoRA
45.97
44.79
44,89
100
Poly
47.96
46.80
46.80
MHR
48.53
47.34
47.33
Our Method
49.53
48.45
48.45
A.2 ABLATION MODEL SCALE
We conducted an ablation study across various scales of the T5 model, including T5-Large (0.78B), T5-XL (3B), and T5-XXL (1 (11B), as well as FLAN-T5 variants. Table 7 and Table & summarised the experiments conducted on 100 tasks from SuperNI. The results demonstrate clear trend: as the model scale increases, there is consistent improvement in performance for all methods, includ- ing ours. Notably, our method C-Poly shows significant gains over other PEFT methods across all scales of the T5 and FLAN-T5 models, suggesting that approach effectively leverages the increased model capacity. The performance improvement is more pronounced larger models (T5- XXL and FLAN-T5-XXL), indicating that while model capacity plays role, method might be particularly effective at utilizing the additional capacity.
Table 7: Performance of T5 models on the 100 randomly selected tasks from SuperNI dataset. We report the average Rouge-1, Rouge-L, and Rouge-LSum for all tasks. Higher is better for all metrics.
Base Model
PEFT Method
Rouge-1
Rouge-I.
Rouge-LSum
LoRA
34.16
33.64
33.65
MoE-LoRA
36.82
36.13
36.15
La-Large
Poly
43.04
42.05
42.09
MHR
44.24
43.32
43.34
Our Method
49.34
48.50
48.51
LoRA
34.93
34.34
34.40
MoE-LoRA
39.78
38.83
38.87
TS-XL
Poly
43.61
42.61
42.62
MHR
45.53
44.62
44.61
Our Method
50.57
49.74
49.76
LoRA
49.97
48.89
48.93
MoE-LoRA
52.14
51.12
51.15
T5-XXL
Poly
55.42
54.65
54.64
MHR
55.81
55.01
55.01
Our Method
62.23
61.44
61.47
15

---

Published as a conference paper at ICLR 2024
Table 8: Performance of FLAN-TS models on the 100 randomly selected tasks from SuperNI dataset. We report the average Rouge-1, Rouge-L, and Rouge-LSum for all tasks. Higher is better for all metries
Base Model
PEFT Method
Rouge-1
Rouge-L
Rouge-LSum
LoRA
68.26
67.42
67.42
MoE-LoRA
68.59
67.76
67.75
FLAN-TS Largge
Poly
68.45
67.60
67.58
MHR
68.84
67.77
67.78
Our Method
68.69
67.80
67.82
LoRA
71.01
70.21
70.24
MoE-LoRA
71.08
70.29
70.33
FLAN-T5-XL
Poly
71.12
70.31
70.35
MHR
71.18
70.36
70.40
Our Method
71.57
70.72
70.74
LoRA
71.89
71.07
71.08
MoF-LoRA
72.05
71.25
71.26
FLAN-T5-XXI.
Poly
72.55
71.76
71.78
MHR
72.40
71.61
71.63
Our Method
73.09
72.27
72.28
A.3 ABLATION THE SKILL ALLOCATION
We designed an experiment to investigate the impact of the relative size of common skill and task specific skill on the results under the same parameter quantity, shown in Table 9 The result shows that the gain is most significant when the number of task specific skills is 1, indicating that explicitly separating the general skills and proprietary skills of skills is the main factor that brings improvement, which is consistent with the methods and viewpoints explained in paper.
Table 9: Performance of T5-Large, FLAN-T5-Large and GLM-10B on the 100 randomly selected tasks from SuperNI dataset. For each model, apply C-Poly with different settings of skill allocation. We report the average Rouge-1, Rouge-L, and Rouge-LSum for all tasks. Higher is better for all metrics.
Base Model
Common Skills
Specific Skills
Rouge-1
Rouge-L
Rouge-LSum
0
43.04
42.05
42.09
3
1
49.34
48.50
48.51
T5-Large
2
2
48.79
47.94
47.94
1
3
47.46
46.60
46.59
4
0
68.45
67.60
67.58
3
1
68.69
67.80
67.82
FLAN-TS-Large
2
2
68.47
67.63
67.65
3
68.21
67.36
67.41
4
0 47.96
46.80
46.80
3
1
49.53
48.45
48.45
GLM-10B
2
2
49.19
48.16
48.16
3
46.85
45.74
45.73
A.4 EXPERIMENTS OF FLAN-T5-XL ON SUPERGLUE
We conducted SuperGLUE experiment FLAN-T5-XL. and the result is summarized in Figure Experiments show that compared to BELAN-T5-LaRE (0.78B) in Figure 2 FLAN-T5-XX (2B) can achieve more stable improvement the SuperGLUE benchmark. As the model gets larger, its gen- eralization ability becomes stronger and can adapt to more tasks, meanwhile because of C-Poly, the phenomenon of negative transfer gets weakened.
16

---

Published as a conference paper at ICLR 2024
LoRA
95
MOE-LoRA
Poly
90
MHR
Our Method
Accuracy
85
80
75
70
AVERAGE
BoolQ
CB
COPA
MultRCC
RTE
WiC
WSC
SubTask: in SuperGLUE
Figure 4: FLAN-T5-XL with different PEFT methods on SuperGLUE benchmark. We reported
overall averaged (AVERAGE) and task-specific accuracy for all sub-tasks.
A.5 ANALYSIS ON TASK-COMMON SKILL ALLOCATION
To further analyze our approach, we visualize the allocation weights for task-common skills, in Figure [5] Figure 66] and Figure 7 We reuse the GLMM-10 models trained on different number of SuperNI tasks in Table 6 and select layer 0-10-20-30-40 for visualization. These figures show that learned task-common weights for different tasks are clearly separated and differentiated.
To compare C-Poly with original Poly, y. we clustered tasks based on the task-common skill allo- cations learned in all layers of the GLM-10B model trained on SuperNI- 100. The clustering results showed in Figure g and Figure P From the dendrograms, are more reasonable and balanced than those from Poly. It shows that our method C-Poly has stronger discrimination the differences and similarities among tasks.
see that the C C-Poly clusters
17

---

Published as a conference paper at ICLR 2024
3681
1950
D882
03
loyort JARTO
Bytta pRCC
lastro 2012 a
lowe, 2013
Eyen 201
Figure 5: Visualization the common skill allocation matrix WA A of selected transformer layers in GLM- 10B after training on SuperNI-10.
18

---

Published as a conference paper at ICLR 2024
task2
task3
0.6
task4
task6
task7
task8
task10
task11
task12
task13
0.5
task14
task15
task16
task17
task18
task19
task20
task21
task22
task23
task24
task25
task26
task27
task28
task29
task30
task31
task32
task33
0.3
task35
task36
task37
task38
task39
task40
task41
task42
task43
0.2
task44
task45
task46
task47
task48
task49
task50
M Tilps leeer
M eilliy laverr
M ElIlSS Quvvr
M Lill>s "nrrri
M Cillss aaerro
M Elllys оτ
M tillys aeer7o
M 21ll's aver2o
M Till's оg
M Eallis dr
M ElllsS ayerao
M TIIls oneero
M EiIlss Byerroo
M Eling aaeeio
Ellias aevrra
Figure 6: Visualization of the common skill allocation matrix WA of selected transformer layers in GLM-10B after training on SuperNI-50.
19

---

Published as a conference paper at ICLR 2024
Figure 7: Visualization of the common skill allocation matrix WA of selected transformer layers in GLM-10B after training on SuperNI-100.
20

---

Published as a conference paper at ICLR 2024
and phities
gomnte prowr gare dorettion pera, poe0aree
con kot aonn adtin commard, short
gww.atoonnnnnn buman
anthmotc
pooo pe ereaion mnreinng
11 coift, yontt and conseiaamss
a087775 rerent arower gererttinn
533 meeeeend aond geveration
conooneeeo Jomee genrration
you postive mesane tranter
pint conmunto severtorily Caseiiction
Rell rainsseer pepurect Jact, gensration Jasted phive, paragrash
privers pilliy purpose generation
oet woon cott zereou te, pappo
sulii sest cenataatin
enno enner generation
canitictnn musifiction
aeki op aooee cretii
1440 fimes entgos
classiiion distuiicaioon cataapoyyy
comerntie prrno eroty costtioon corpon
bes at anthmetic
no peention
ho0 amaccorreiie surmnary Classticaaton
penerationn
tong eitments
scan long, tost, generation actionn command asd
an moe snr offornn mesning centere generttionn
Moxeyy dream, amonnd gemerstion
min tee serllctton
ceneucte and sen gut
200
WNLC22 conala inder, addition
026322 jugaw, Suevilicationn threst
information, retrevcal
Jope supczteet, passoiication
te0iC221 code.
court.reeeency words
couadication jdentity
penraatio
taaSS15 dalegue slanu/teution
4b13. suestion geseration
Eccsom
fewsejpe EGE
tack505 preposition diasstication
Aypseque
Aquunvd "itureu CEn
eouginal
101575 corfanty disiigg,
jammiu gnswer, generation schost nuthenattcs
1as0055
servevat, 2018 tank) plassfication
saaiig "aust 1300
sel (1.
bas-511 arplonation, genersbion
1209 atumi, Dlesutication ctrertuee
style, transfer
berk cragpies burgans ctasoitiation
paeti rosuiin LSE
Our Jentoval componts Inversion
1.21 mathos atter
quares
give prppc Derpptraing dask 222
serseral 2010 sez10 dows iec.auuuaryy mathrmaatiooll ansoer, generration
jeconent anrwer
Woat Srg Letomun
aattro what is mhe ettecs
aate prrm ahoee generation Honh acchoo prochoogg
an curd snaee geneatton
eni cetodying deriingg Coneiattin Buenyy
153 toma Bid locatio Jund Sunt_chan chaan
215 caror, Seet sontence in, desate
stanee, clasttcatio t00 mnsatio snove, procrtton
yaar Oifermuer
peorret arowe. penorsttttt
disagre
montty a atrresation
oumntoo penerntion
conernttion hige cneed Sempess Deograsty
jubymeet
anotion eNnEE
spaserrs
Figure 8: Task clustering dendrogram for common skill allocation matrix WA of C C-Poly using GLM-10B as the base model in SuperNI-100 experiment. Tasks are grouped into the same category they share a similar subset of skills.
21

---

Published as a conference paper at ICLR 2024
merre sooe pentrrenn Hig. anted pnteolpreaaa peoerry
semtoee, gevrrttee
Jence purmenee AND contoo
aee conte colutt ditii sescenn copss
suret weth semsst survss
reveraa 2012 usec gone cosso
eaory, eosuuaion corpet Basl proyattin
ctrmet
petteen
poureee peperve 11H Vrevder
sad the ci siouutaation cottttatio
covariiee corennte pereds
decorre oecaeess fur gevenctioo balted gres poreprant
eneecctoonn parsttaaaa Dantuicttion
sisoypo clendication
groven peturrtion
ctsorticatin, paappry
geveration
antronte
la
1213, aboeni dwweffatin
and Jurd, reah
ger geriette teesacoa cuanciin
2717 vit jrunt.r
srpwer yennerttion pmunt stycheley
arcest
Japeal tultsces
yesk 12/0, sone, plnnriccoon wetoore
yor facoorts revertalcyy petateesiinn
00o4/22 Svasnc Svaskcmoun) soutt gogerrationn
erernd, pert, uppatoti
puseer amerstin
291, strv penryroooeeee
/obrnt Entuurory, cends
212 antreer www.aattttooooo varatnn morae agping DeC
4.1,
Jaave. cinoaiiation Plerityy attes
.dive entorvation prwoall
altin, citueend et promann
derk georration
goieet tost prmpse an
pemuramnnn miraj
ty verticttoo
conoable colvrnnt short
enteaainn Fortoy
Figure 9: Task clustering dendrogram for common skill allocation matrix WA of Poly using GLM- 10B as the base model in SuperNI-106 100 experiment. Tasks are grouped into the same category if they share a similar subset of skills.
22


Reviews:
[{"review_id": "3NEMVvJpL0", "reviewer": "Reviewer_yvMZ", "summary": "This paper proposes an approach called Customized Polytropon (C-Poly) for multi-task learning using parameter-efficient modules. The key idea is to explicitly separate task-common skills that can be shared across tasks, and task-specific skills that are unique to each task.  Note that this augments the previously published Poly method (by introducing task-specific adapters, combining subsets of shared and specific tasks, and can allow better interpretation by the parameters learned for selection and weighting).\n\nThe model consists of components related to task-common skills (shared low-rank across tasks), and low-rank adapters for each task.  Low-rank adapters (E.g. LoRA) are used to improve param efficienty also.  This approach appears to mitigate negative transfer effects, and improve learning over compared methods.  \n\n\n//Having read the responses: I think its a nice idea, the results are good, but some more analysis / polishing of the paper could be useful before this paper is published (See also response to comments).  To be frank I`m still borderline about this.  I will however increase my score to reflect the authors effort in responses and updates.", "questions": "How is the number of common and task-specific skills determined? Is there a systematic way to set these hyperparameters?\n\nHow does performance scale with increasing number of tasks?\n\nHow are skills initialized?  does this affect selection?\n\nCan you provide ablation studies controlling for common vs task-specific skills? \n\nMore in-depth discussion / results on interpretability\n\nIf new tasks appear after training, can the model adapt?\n\nResults are comparable largely to previous work", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- this paper presents an intuitive approach to combine shared and specialized skills\n\n- results are good in comparison to previous methods\n\n- sample efficiency is improved; the approach of explicitly separating task-common and task-specific skills to mitigate negative transfer\n\n- offers some more interpretability due to selecting particular skills", "weaknesses": "The method introduces additional hyperparameters like number of common/task-specific skills which may require tuning\n\nIt is not clearly analyzed if certain tasks benefit more from common or specialized skills\n\nThe interpretability via learned task hierarchies is not explored much in experiments\n\nResults are comparable largely to previous work\n\nThis paper is an extension of a previous work, with some (nice) but perhaps small increments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MuOsFWZo0k", "reviewer": "Reviewer_soi2", "summary": "The paper tackles the task of parameter efficient fine-tuning via Low-Rank Adapter (LoRA) which parametrized the weight updates as low-rank composition to significantly reduce the number of learnable parameters and computation. \nThe work proposes a novel approach, called Customized Polytropon, which extends LoRA to Multi-Task Learning setup by learning multiple LoRA adapter corresponding to different tasks.\nThe key insight is to decompose the learnable parameter into Task-Common adapters, which is shared among all tasks, and Task-Specific adapters specifically trained for different tasks.\nGiven this set of adapters, the proposed framework would learn to combine them for different task, thus enable knowledge transfer among tasks while efficiently learning these adapters.\nThe paper conducts experiments on SuperCLUE and Super Natural Instruction datasets as well as on T5-Large and GLM-10B models to show its effectiveness.", "questions": "+ Could the authors clarify on how they combine different adapters? This would help with understanding as well as reproducibility.\n+ Experiments on the effects of the number of adapters/ranks/tasks could provide better insight on the robustness and limitations of the proposed methods\n+ Stronger base model can be used to stress-test the generalizability of the proposed framework.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is self-contained which makes it accessible to a wide range of reader. Moreover, the paper is also easy to follow.\n+ The proposed method of decomposing learnable parameter into a task-common and task-specific portions is sensible as well as easy to develop in practical setting.\n+ The problem of parameter efficient finetuning is impactful which helps to democratize LLM technology on consumer device.", "weaknesses": "+ As the adapter in LORA are low-rank linear projection of parameters in attention modules, a combination of task-common and task-specific LORA adapters seems to be equivalent to just Mixture of LORA as addition of linear transformations is still a linear transformation (rows 2 and 3). Thus, it is unclear why decomposing learnable parameters would improve performance.\n \n+ The experimental section misses some studies to show the effective of hyper-parameters in the model. For examples, what is the how number of adapters in task-common and task-specific modules effect the performances? What is the impact of rank or the number of tasks toward final performance? These experiments would offer better insight into how robust the proposed method is under different setting.\n\n+ The performance seems to saturate when being applied to strong base model such as T5-Large. It seems to suggest that the effect of task-common components vanish when the base model can generalize well toward different downstream tasks. This could be an interesting phenomena can be study using stronger base model such as llama and llama2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qcB1dtM0v8", "reviewer": "Reviewer_WuXZ", "summary": "The paper proposes a new multi-task learning architecture based on LoRA finetuning framework. They propose the task-common skill sets and task-specific skill sets. Also they learn the task-specific combination weights of task common skill sets using Gumbel-Sigmoid. In the experiment, they adopt two different architectures (encoder-decoder -- T5 and decoder-only -- GLM) and show their performance on several multi-task benchmark in NLP.", "questions": "1. Since you have used Gumbel Sigmoid to optimize the w_i, what is the distribution of all w_i's learnt in the model? Is there any specific loss to force w_i to be close to 0 or 1? What is the performance variance if you need sample the w_i based on Bernouli distributions in the final evaluation?\n\n2. In the experiment results, it is clear that the method with GLM-10B outperforms baselines a lot and the method with T5 stands close to the baselines. In the paper, the authors claim the difference is due to the different model architecture. However, GLM-10B and T5_large also differ a lot in the model capacity. How do you know the difference of performance compared to the baselines is due to the architecture instead of model capacity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper clearly states the difference between their model architecture and previous baseline models.\n2. With GLM model, their proposed method outperforms the baseline models for a significant gap.\n3. They compare to other baseline models in a fair way.\n4. The paper is well-written and easy to follow.", "weaknesses": "1. The method does not achieve significant improvement with T5 architecture.  (But I am not expert in NLP tasks and I am open to other reviewers' opinions regarding the performance.)\n2. The model design seems incremental to the previous methods Poly and MHR. \n3. It is unclear about the optimization loss function and the main paper does not discuss this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yjllqJrLf4", "reviewer": "Reviewer_ZWwD", "summary": "The paper mixes different ideas from the multi-task learning and parameter-efficient mixture of experts literature: Methods such as `MoLora` introduced the idea of using low-rank LoRA modules as \"experts\". Methods such as `Poly` or `MHR` show that a new task can be learned as a combination of adapters modules (~ mixture of experts) where the combination weights are task-specific.\n\nThis paper additional proposes to explicitly capture the fact that each task can have shared knowledge (shared across all tasks) as well as task-specific knowledge at the adapters level. In this setting, a new task is now learned as a soft combination of **(i)** $A$ LoRa modules shared across all tasks and **(ii)** $B$ task-specific LoRA modules ($B$ separate modules for each task). In this case, all combination weights are task-specific, and when $B = 0$, we recover previous approaches such as `Poly`. \n\nIn the proposed framework, every adapter is a LoRA module and the combination weights are binary, i.e. each expert can be activated or not. During training, the binary decisions are approximated with Gumbel-sigmoids to allow for backpropagation.\n\nThe proposed approach is evaluated on `T5-Large` and `GLM-10B` as backbone, on the SuperGlue and Super Natural Instructions datasets, and compared to a set of recent baselines (LoRa, MoE-LoRA, Poly and MHR)", "questions": "- **Minor suggestion on writing:** \n  * I found the introduction hard to read as it mixes introduction with some related work towards the end, and introduces several concepts without really contextualizing them (e.g. in two paragraphs, the text jumps from MoLora $\\rightarrow$ Poly $\\rightarrow$ MHR $\\rightarrow$ CGC and PLE).\n  * Using $A$ and $B$ in Equation 5 is a bit confusing as these were introduced earlier as the number of shared and task-specific modules", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **interesting research direction:** I think the overall goal of the paper is clear and interesting. Parameter-efficient Mixture-of-experts with LoRA modules is a recent and interesting research direction, and generalizing this framework to multi-task/domain learning as was done for other \"adapters\" type of work seems very natural. Similarly, the idea of separating task-specific and shared knowledge has shown some success in multi-task learning literature, and makes a lot of sense in that setup.\n\n- **Diversity of the experimental setup:** The experiments consider two backbones, as well as two multi-task benchmarks with quite different number of tasks. The proposed method is also compared to several recent baselines\n\n- **Experimental improvement**: While the improvement in experiments is not always consistent, it does show significant gains in some benchmarks, in particular with the decoder-only `GLM` backbone.", "weaknesses": "- **Explicit vs Implicit separation of task-specific and shared modules**: To my understanding,  the proposed method can be seen as subcase of some of the baselines mentioned : If the number of adapters in `Poly` is set to $A + BT$, then in theory the model could learn to make certain modules task-specific by setting $w_i^t$ to 1 for only certain pairs of $(i, t)$. In contrast, the proposed method makes this structure explicit by defining task-specific modules; However, this still requires additional $BT$ task-specific adapter modules, hence an increased number of parameters. From the experiments, it is not clear to me if the number of parameters/modules used by the baselines and the proposed method are comparable.\n\n- **Experimental analysis**: In general, I find the proposed idea interesting, but I lack some experimental insights to better understand the proposed method. For instance, some points I found unclear are the following:\n  -  1) For instance, the number of task-specific modules $B$ for each task seems to be an important parameter, but I did not see a lot of discussion/ablation about it; It seems that $B$ is set to 1 throughout the paper\n  - 2) Similarly, the proposed method can use $A + B$ modules per task, while baselines only consider $A$ task-shared modules: It is not clear to me if the setup is fair to the baselines. Considering other  configurations would be interesting.\n  - 3) From the results in **Table 2**, it seems that the performance improvement is not consistent: For instance with the encoder-decoder `T5` model, improvement mainly comes from the `WiC` task, which could mean this task is more likely to interfere with others. However for the decoder-only `GLM`, we see that adding a task-specific module yield significant improvement across all tasks. It is not clear to me why that is the case: Since the tasks are the same in both case, it does not seem to be directly related to the multi-task setup; could it be related to the number of LoRA modules with respect to the architecture (*see point 2*) ?\n\n**Note post-rebuttal:** I still have some reserves about the method's robustness and sensitivity to the task specific/shared setup ratio, but in light of the rebuttal addressing my main concerns and added ablation experiments, I'll raise my score from 5 to 6", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)