Dynamic Task Vector Grouping for Efficient Multi-Task Prompt Tuning
Authors: Peiyi Zhang, Richong Zhang, Zhijie Nie, Ziqiao Wang
Accepted at: acl25

Dynamic Task Vector Grouping for Efficient Multi-Task Prompt Tuning

Peiyi Zhang1, Richong Zhang1,2*, Zhijie Nie1,3, Ziqiao Wang4,
1CCSE, School of Computer Science and Engineering, Beihang University, Beijing, China
2Zhongguancun Laboratory, Beijing, China
3Shen Yuan Honors College, Beihang University, Beijing, China
4School of Computer Science and Technology, Tongji University
{zhangpy,zhangrc,niezj}@act.buaa.edu.cn
ziqiaowang@tongji.edu.cn

Abstract

Multi-task prompt tuning utilizes multiple high-resource source tasks to improve performance on low-source target tasks. Existing approaches transfer the soft prompt trained by combining all source tasks or a single “high-similar” source task one-time-only. However, we find that the optimal transfer performance often comes from a combination of source tasks, which is neither one nor all. Further, we find that the similarity between source and target tasks also changes dynamically during fine-tuning after transferring, making similarity calculation in the initiation stage inadequate. To address these issues, we propose a method called Dynamic Task Vector Grouping (DTVG), whose core ideas contain (1) measuring the task similarity with task vectors instead of soft prompt, (2) grouping the optimal source task combination based on two metrics: target similarity and knowledge consistency; (3) dynamically updating the combination in each iteration step. Extensive experiments on the 26 NLP datasets under different settings demonstrate that DTVG effectively groups similar source tasks while reducing negative transfer, achieving the start-of-art performance.

1 Introduction

Full parameter fine-tuning (FT) of large pre-trained language models (PLMs) has shown significant success in addressing various natural language processing (NLP) tasks. However, the conventional fine-tuning paradigm requires substantial memory and computational resources. Recently, parameter efficient fine-tuning (PEFT) (Houlsby et al., 2019; Li and Liang, 2021; Lester et al., 2021; Zaken et al., 2022; Hu et al., 2022) aims to achieve comparable results of FT by updating a significantly small set of the model parameters.

Figure 1: In the upper part, we use performance on the RTE validation set to study potential conflicts of source tasks. We incrementally add source tasks with a random order and train soft prompt by examples-proportional mixing (Raffel et al., 2020). In the bottom part, we calculate the cosine similarity between the average pooled representations of the prompt tokens (Vu et al., 2022). We initialize the RTE soft prompt using the source task’s soft prompt with the highest similarity. The legend marker denotes the source task with the highest similarity, which shifts from QNLI to MNLI during fine-tuning.

Soft prompt tuning (PT) (Lester et al., 2021), as an effective PEFT method, achieves a trade-off between effectiveness and efficiency. During training, a series of learnable soft prompt vectors prepended to the input are updated while the original PLMs are frozen. Unlike methods such as LoRA (Hu et al., 2022) and Adapter (Houlsby et al., 2019), PT is independent of the model architecture and can be applied to various models without modification. Although promising, the existing study (Asai et al., 2022) demonstrates PT still underperforms compared to FT, particularly in the case of low-resource tasks. An additional issue with PT is sensitivity to the initialization and needs longer tuning for converge (Lester et al., 2021).

Recent works (Vu et al., 2022; Asai et al., 2022; Feng, 2023; Wang et al., 2023) address the above limitations by transferring soft prompt from high-
resource source task to low-resource target task.

Specifically, they initialize the soft prompt for the target task by either (1) learning a common soft prompt across all source tasks or (2) learning a soft prompt for each source task and selecting one with the task similarity. Subsequently, the soft prompt is tuned exclusively using limited training samples from the target task. These transfer approaches effectively maintain the parameters efficiency of soft prompts and demonstrate superior performance compared to vanilla prompt tuning.

Despite substantial progress, we challenge the rationality of some straightforward ideas in existing approaches. We first check whether existing methods achieve optimal performance. In the upper part of Figure 1, we observe that a subset of source tasks achieves the best transfer performance, neither all source tasks nor a source task. Additionally, Vu et al. (2022) demonstrates MNLI, QNLI, and QQP positively transfer to the RTE dataset, while we find that their gradual addition does not yield a consistent monotonic improvement due to the potential conflicts among source tasks. These observations revealed that we should find a group of source tasks for each target task and consider potential conflicts between source tasks besides the similarity to target tasks.

Further, we check whether “the most similar source task” will change in the tuning stage of the target task. We study a single-task version of SPOt (Vu et al., 2022), which transfers the soft prompt from a source task to initiate the target task via similarity measure between their learned soft prompt. In the bottom part of Figure 1, we find that “the most similar source task” of RTE shifts from QNLI to MNLI over time. Recall that the low-resource characteristics of the target task hinder sufficient convergence of soft prompt; therefore, it is unsurprising that we cannot select the truly most similar task with an unconverted soft prompt of the target task. This observation suggests that dynamically updating the selected source task during the target task’s fine-tuning may enhance the sustainable acquisition of knowledge.

Motivated by these valuable empirical observations, we propose a method called Dynamic Task Vector Grouping (DTVG). Specifically, We first introduce a novel task similarity metric, the dot product between task prompt vectors (TPV), which steadily achieves a better transfer performance than the current metric, the cosine similarity between soft prompts. Based on this metric, we introduce a source task grouping method to select the transfer source task group for each target task with two metrics, including target similarity and knowledge consistency. Then, a multi-task merging method is used to weighted sum the task vectors from the target task and the selected source tasks, synthesizing the initialization soft prompt for the target task. During the fine-tuning stage of the target task, we track the task similarity changes and dynamically update the source task group, which will effectively improve transfer performance.

In summary, our major contributions are to:

• We present an effective task similarity metric, based on the task prompt vectors, to measure the transfer performance between tasks.

• We propose DTVG, a dynamic task vector grouping method that assembles and updates a source task group for each target task throughout the iterative training process to ensure sustainable acquisition of knowledge.

• We confirm the effectiveness of DTVG on the 26 datasets based on T5 and Llama3 under different settings, surpassing the advanced models and achieving SOTA performance.

2 Background

Soft Prompt Tuning Soft Prompt Tuning (PT) (Lester et al., 2021) proposes strategically inserting the learned soft prompt into the input. Formally, for a task \( t \) with the dataset \( \mathcal{D} = \{(x_i, y_i)\}_{i=1}^{|\mathcal{D}|} \), we fine-tuning a pre-trained model \( F_\Theta \) to perform better in the task \( t \) with its parameter \( \Theta \) frozen. Instead, the learnable soft prompt \( P \in \mathbb{R}^{d \times r} \) is introduced, where \( d \) is the hidden state dimension of \( F_\Theta \) and \( r \) is the soft prompt length. The soft prompt \( P \) and the token embedding matrix \( E(x_i) \) are spliced as the input of \( F_\Theta \). Then, the soft prompt \( P^* \) is learned to boost the posterior probability of correct output \( y_i \):

\[
P^* = \arg\max_P \mathbb{E}_{(x_i, y_i) \in \mathcal{D}}[\mathbb{P}(y_i|P; E(x_i))]
\]

Although the PT method has shown great success in various NLP tasks, it still faces the low-source challenge: Too few training samples prevent the soft prompts from converging, which can result in huge performance differences under different soft prompt initializations.
Figure 2: An overview of methods for comparison. One For One, initialize a target task by retrieving the task-specific prompt from one of the most similar source tasks based on task similarity. ALL For One, initialize a target task by learning appropriately across all source tasks based on prompt or data mix. Our Method: Part For One, dynamic group a subset of source tasks and merge their task prompt vectors.

Multi-Task Prompt Tuning  Multi-Task Prompt Tuning (Mahabadi et al., 2021; Vu et al., 2022; Asai et al., 2022; Wang et al., 2023) is proposed to address the low-source challenge of PT. Formally, given a high-resource source task set \( \mathcal{S} = \{s^1, s^2, \ldots, s^n\} \), where \( n \) is the number of source tasks, Multi-Task Prompt Tuning improve the performance of a low-source target task \( t \) by transfer learning from \( \mathcal{S} \). Current methods usually contain two stages: (1) Learning the transferable soft prompts \( P_{mix} \) from \( \mathcal{S} \), defined as \( P_{mix} = G(S, t) \) where \( G \) is the learning method; (2) Adopting \( P_{mix} \) to \( t \) and re-tuning \( P_{mix} \) with maximum training steps \( N_{max} \) on the training set of task \( t \).

Multi-task Prompt Tuning does not impose restrictions on \( G \) to get \( P_{mix} \) and how to adopt \( P_{mix} \) on \( t \), excepting that the transfer ones must be soft prompts. Therefore, there are two representative lines of work to be highlighted. One For One: \( G \) serves as a retriever and selects the learned soft prompt of the most similar \( s \) to initialize for \( t \). SPoT (Vu et al., 2022) regards the soft prompts as the task embeddings and measures task similarity via cosine similarity between soft prompts. Feng (2023) learns \( G \) to predict transfer gain by randomly sampling soft prompt pairs. All For One: \( G \) serves as a blender and learns the task-shared prompt from source task set \( \mathcal{S} \) via different mix strategies. SPoT (Vu et al., 2022) also learns a single soft prompt through multi-task learning by mixing data. ATTEMPT (Asai et al., 2022) trains an attention module and mixes instance-wise prompts from all source tasks \( \mathcal{S} \). MPT (Wang et al., 2023) extends the multi-task training method of SPoT by learning task-shared and task-specific modules. TPT (Wu et al., 2023) propose to retrieve token-wise soft prompt from the prompt bank.

Task Arithmetic  Task Arithmetic (Ilharco et al., 2023; Zhang et al., 2024; Ortiz-Jimenez et al., 2024) as a newly emerged cost-effective approach demonstrates the effectiveness of multi-task training by operating task vectors derived from different tasks, where task vectors are given as the relative difference between the initialized parameters and those obtained after fine-tuning, capturing the changes induced by the adaptation process in weight space. Our proposed approach is inspired by Task Arithmetic. Similar to task vectors, the task prompt vectors (TPV) \( T = [v_1, \ldots, v_r] \in \mathbb{R}^{d \times r} \) are defined as the difference between \( P_{init} \) and \( P^* \), i.e. \( T = P^* - P_{init} \), where \( v_i \in \mathbb{R}^{d \times 1} \) represent the \( i \)-th vector in \( T \). Concurrent work (Belanec et al., 2024) uses TPV to enable generalization to new target tasks without training. In contrast, we introduce TPV to address the issue of potential negative transfer in multi-task prompt tuning.

3 Method

3.1 Overview

We propose a novel multi-task prompt tuning approach, Dynamic Task Vector Grouping (DTVG), which dynamically groups a subset from the source task set to transfer to the target task. Therefore, \( G \) in our method serves as a grouper, allowing a specific target task to selectively leverage partially related source tasks, mitigating the risk of negative transfer. As shown in Figure 2, DTVG actually follows the idea of Part For One and distinguishes itself from existing methods.

DTVG consists of two stages: (I) Task Prompt Vector Learning to obtain a tuned TPV for each source and target task and (II) Multi-Task Prompt Transfer to group source tasks’ TPV and merge it with the target vector’s TPV. Note that the first stage only needs to be performed once, while the second
Figure 3: DTVG is to learn dynamic grouping partially related source tasks, including two stages: I) Task prompt vector Learning; II) Multi-task Prompt Transfer. In the first stage, we obtain task prompt vectors via vanilla prompt tuning. In the second stage, Source Task Grouping and Multi-task Merging are executed at each iteration step.

stage is iterative, and the source task group will be dynamically updated during the fine-tuning process of the target task. An algorithm-style process of DTVG is provided in Appendix F.

3.2 Task Prompt Vector Learning

In the first stage, we obtain the soft prompt by individually tuning both the source and target task via the same initialization \( P_{init} \) and calculate their task prompt vectors \( T \). Therefore, we have \( n + 1 \) task prompt vectors from \( S \cup \{ t \} \).

We propose using the average token-wise task prompt vectors to compute their dot product, allowing us to predict task similarity. This method enables a quantitative assessment of task relationships, as illustrated at the top of Figure 3. Specifically, given two task prompt vectors \( T_1 \) and \( T_2 \) from \( s^1 \) and \( s^2 \), we can calculate the similarity between tasks \( s^1 \) and \( s^2 \). The task similarity scores \( sim \) between tasks is defined as follows:

\[
sim(T_1, T_2) = \frac{1}{r^2} \left( \sum_{i=1}^{r} v_i^1 \right)^{\top} \left( \sum_{j=1}^{r} v_j^2 \right)
\]

where \( r \) denotes the length of soft prompt tokens.

To evaluate the effectiveness of this metric, we conduct transfer experiments on the SuperGLUE benchmark. As shown in Table 1, TPV demonstrates consistent positive transfer, whereas SPoT exhibits negative transfer on WSC and CB, showing the superiority of our metric. Please refer to Appendix C.1 and I for the experiment details and visual analysis, respectively.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="5">SuperGLUE</th>
  </tr>
  <tr>
    <th>Multi</th>
    <th>Bool</th>
    <th>WiC</th>
    <th>WSC</th>
    <th>CB</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>PT</td>
    <td>72.7</td>
    <td>76.0</td>
    <td>62.6</td>
    <td>67.3</td>
    <td>82.1</td>
    <td>72.1</td>
  </tr>
  <tr>
    <td>SPoT</td>
    <td>74.9 ↑</td>
    <td>80.6 ↑</td>
    <td>65.2 ↑</td>
    <td>63.5 ↓</td>
    <td>78.6 ↓</td>
    <td>72.6</td>
  </tr>
  <tr>
    <td>TPV</td>
    <td>74.2 ↑</td>
    <td>81.3 ↑</td>
    <td>66.1 ↑</td>
    <td>67.3 -</td>
    <td>92.9 ↑</td>
    <td>76.4</td>
  </tr>
</table>

Table 1: Performance on SuperGLUE benchmark.

3.3 Multi-task Prompt Transfer

In the second stage, we introduce an iterative process for multi-task prompt transfer. As shown in the bottom of Figure 3, for each iteration, Source Task Grouping and Multi-Task Merging are executed sequentially to obtain \( P_{mix} \).

Source Task Grouping  Source task grouping aims to group a subset of source tasks \( S' \subseteq S \). Source tasks in \( S' \) should not only be similar to the target task but also possess consistency of knowledge. We propose two metrics to characterize the source task group quantitatively, including Target Similarity and Knowledge Consistency.

Target Similarity: To measure the transferability of multiple source tasks to the target task, we define a target similarity score TS as the average of
the similarity between each source and target task prompt vector pair \((T_i, T_t)\), which is formulated as

\[
\mathrm{TS}(S, t) = \frac{1}{|S|} \sum_{s^i \in S} sim(T_i, T_t)
\] (3)

**Knowledge Consistency**: In multi-task transfer learning scenarios, conflicts among source tasks are prevalent. For example, in NLP, words crucial for sentiment (e.g., “good”) may have varying significance in topic classification, leading to ambiguity and reduced performance in the target task. We propose to quantify the conflicts within a task group by calculating the average pairwise \(sim\) between tasks. More formally, we defined the Knowledge Consistency Score (KC):

\[
\mathrm{KC} = \begin{cases}
\frac{2}{n(n-1)} \sum_{i < j} sim(T_i, T_j) & \text{if } |S| \geq 2, \\
0 & \text{otherwise}
\end{cases}
\] (4)

Therefore, the objective for selecting a source task group can be defined using \(TS\) and \(KC\):

\[
\max_{S' \subseteq S} \left( \mathrm{TS}(S', t) + \lambda \mathrm{KC}(S') \right)
\] (5)

subject to, \( \forall s^i, s^{j} \in S',\ s^i \neq s^{j} \)

where \(\lambda\) is the hyperparameter to achieve the trade-off between TS and KC.

However, the process to find the optimal \(S'\) is equivalent to the Set Cover problem, which is the NP-Hard. As the number of tasks increases, the selection from \(2^{|S|}\) subsets becomes infeasible. Therefore, we use a heuristic algorithm to find a suboptimal subset, achieving a balance between efficiency and effectiveness. As shown in Figure 3, the algorithm consists of two steps: (1) \(sim(T_i, T_t)\) between each source task \(s^i\) and target task \(t\) is computed. Then \(\{sim(T_i, T_t)\}_{i=1}^n\) is ranked in the descending order, obtaining a rank list \(\Pi = \{\pi^1, \pi^2, \ldots, \pi^n\}\), where \(\pi^j\) is the source task with \(j\)-th highest \(sim(T_i, T_t)\); (2) The source tasks with \(sim(T_i, T_t)\) are added to the set \(S'\) one by one in the order in T until the \(\mathrm{KC}(S')\) is no longer increasing. The implementation details are provided in the Appendix G.

**Multi-task Merging** Multi-task merging aims to merge the task prompt vectors from the source task group and the target task to get a final soft prompt \(P_{mix}\). Specifically, \(P_{mix}\) is obtained by the sum of (1) the rescaled soft prompt task vectors of \(S' \cup \{t\}\) and (2) a common initialization prompt, which can be denoted as

\[
P_{mix} = \underbrace{P_{init}}_{\text{Initialization}} + \underbrace{\alpha_t T_t + \sum_{s \in S'} \alpha_s T_s}_{\text{Merged Task Prompt Vector}}
\] (6)

where \(\alpha \in \mathbb{R}^l\) is token level scaling term initialized to all-ones vector. In practice, we employ rescaled task prompt vectors to compute the task similarity score (Equation 2).

**Iteration Update** In each training step, we sequentially execute the above two steps to ensure the correct source task group \(S'\) selection to compute \(P_{mix}\) with in-batch. In practice, we observe that in the early stages of training, the grouping of source tasks exhibits significant fluctuations due to the insufficient convergence of the target task prompt vectors. As the iterations progress, the dynamic grouping gradually stabilizes and ultimately maintains consistency (see Section 4.4 for details).

4 Experiments

4.1 Experiment Setup

**Datasets** We evaluate the model’s natural language understanding capabilities using the GLUE and SuperGLUE benchmarks. In addition, we also use four question-answering datasets from the MRQA 2019 benchmark and four datasets from the “other” benchmark. In the following, we introduce the source tasks and target tasks separately. Further details can be found in Appendix B.

**Source Tasks**: Following Wang et al. (2023), we set \(n\) to 6, and use the same large-scale datasets as source tasks, including MNLI, QNLI, QQP, SST2 from GLUE, ReCoRD from SuperGLUE, and SQuAD from MRQA 2019.

**Target Tasks**: we use all 8 datasets from GLUE, 5 datasets (excluding ReCoRD) from SuperGLUE, 4 datasets (excluding SQuAD) from MRQA 2019, and 4 datasets from the “other” benchmark.

**Models** We adopt the model setup from (Lester et al., 2021) for prompt tuning. Our experiments mainly utilize T5-base with the soft prompt of length 100, while in ablation studies, we also explore other scales of T5 in Section 4.4.

**Baselines** We compare our method with several baseline methods. (1) no transfer learning, which updates model parameters for the each target task without source task, including Finetuning (FT),
Prompt Tuning (PT) (Lester et al., 2021), BitFit (Zaken et al., 2022), Adapter (Houlsby et al., 2019), LoRA (Hu et al., 2022), DePT (Shi and Lipani, 2024), as well as multi-task versions of FT, Adapter, HyperFomer (Mahabadi et al., 2021), and HyperDecoder (Ivison and Peters, 2022). Note that we exclude ACCEPT (Lin et al., 2024) due to the lack of accessible open-source code, which prevents an evaluation of its ability to address sensitivity to prompt initialization. (2) transfer learning + one for one, where transfer soft prompt from one source task to each target task, such as SPoT (Vu et al., 2022) (3) transfer learning + all for one, where transfer soft prompt from all source tasks to each target task, including ATTEMPT (Asai et al., 2022), MPT (Wang et al., 2023), TPT (Wu et al., 2023) as well as multi-task versions of ATTEMPT, and MPT. For a fair comparison, we directly quote the results of the baselines reported in previous works (Asai et al., 2022; Wu et al., 2023; Wang et al., 2023; Shi and Lipani, 2024) whenever possible, and utilize publicly available source code to ensure consistent experimental settings.

Implementation Details For both the Task Prompt Vector Learning and Multi-task Prompt Transfer stage, we train on high-resource source tasks for 300K steps, following Vu et al. (2022). For the target tasks, we set \( N_{\text{max}} \) to 30K. Aligning with standard prompt tuning methods (Lester et al., 2021), we use a default learning rate of 0.3 and select checkpoints with the highest validation set scores to extract task prompt vectors. In the Multi-task Prompt Transfer stage, we apply two-speed learning rates for different modules. We conduct transfer experiments four times and report the average results. Please see Appendix C for details.

Parameter Efficiency For both source and target tasks, we compute the task prompt vector \( T \in \mathbb{R}^{r*d} \), where \( r \) is the length of the soft prompt and \( d \) is the model dimension. For each source task, we introduce a learned scaling term \( \alpha \in \mathbb{R}^r \). Our framework enables knowledge transfer from partial source tasks to the target task, therefore, the total number of learned parameters ranges from \( r + r * d = r*(d+1) \) to \( (n+1)*r+r*d = r*(d+n+1) \), where \( n \) is the number of source tasks. We compare different methods’ trainable parameters under the least favorable conditions of DTVG in Table 2.

4.2 Main Results

Full-dataset Transfer Table 2 provides the performance and parameter comparison for each dataset on the GLUE and SuperGLUE benchmarks across different baselines. Additionally, we visualize the result on GLUE (see Appendix A). Notably, our proposed method, DTVG, outperforms others by achieving the highest average performance on GLUE and SuperGLUE with a minimal parameter tuning fraction of 0.035%, in contrast to the fine-tuning. When compared to prompt tuning in terms of low-resource datasets, DTVG significantly improves the performance of the target task, such as CoLA (10.6% vs. 69.1%) and CB (67.9% vs. 97.6%). Simultaneously, our multiple experiments demonstrate that DTVG is robust for addressing inappropriate soft prompt initialization leading to performance degradation. Please see Appendix D for details on MRQA and “Other” benchmarks.

Few-shot Adaptation We compare our method with other baselines on BoolQ, CB, and SciTail in Table 3. On average, our method outperforms the baselines in low-resource settings with only (\( k=4,16,32 \)) shots, indicating that our DTVG is adept at harnessing knowledge from multiple source tasks for effective transfer in scenarios with limited training samples. More details about GLUE and SuperGLUE are given in Appendix E.

4.3 Ablation Study

Source Task Grouping Strategy We conduct ablation experiments on the SuperGLUE benchmark to study the impact of two different perspectives for source task grouping. For a) Target Similarity (TS), we only merge TPV with \( sim \geq 0 \). For b) Knowledge Consistency (KC), we select the source task group with the highest \( KC \) among all source task combinations. As shown in Figure 4, these strategies can improve performance consistently. KC improves the average performance on SuperGLUE from 74.8 to 75.1, suggesting that mitigating the conflict among multiple source tasks is critical for effective multi-task prompt tuning, even when the task combinations may not be directly related to the target tasks. ST improves the average performance on SuperGLUE from 74.8 to 75.9, indicating that \( sim \) can effectively evaluate and leverage similar source tasks for transferring.

Multi-task Prompt Transfer Strategy We conduct a study to ablate different multi-task prompt
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">param\ task</th>
    <th colspan="8">GLUE</th>
    <th colspan="5">SuperGLUE</th>
  </tr>
  <tr>
    <th>MNLI</th>
    <th>QQP</th>
    <th>QNLI</th>
    <th>SST2</th>
    <th>STS-B</th>
    <th>MRPC</th>
    <th>RTE</th>
    <th>CoLA</th>
    <th>Avg.</th>
    <th>Multi</th>
    <th>Bool</th>
    <th>WiC</th>
    <th>WSC</th>
    <th>CB</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td colspan="17">no transfer learning</td>
  </tr>
  <tr>
    <td>Finetuning<sub>1</sub></td>
    <td>220M</td>
    <td>86.8</td>
    <td>91.6</td>
    <td>93.0</td>
    <td>94.6</td>
    <td>89.7</td>
    <td>90.2</td>
    <td>71.9</td>
    <td>61.8</td>
    <td>84.9</td>
    <td>72.8</td>
    <td>81.1</td>
    <td>70.2</td>
    <td>59.6</td>
    <td>85.7</td>
    <td>73.9</td>
  </tr>
  <tr>
    <td>PT<sub>1</sub></td>
    <td>76.8K</td>
    <td>81.3</td>
    <td>89.7</td>
    <td>92.8</td>
    <td>90.9</td>
    <td>89.5</td>
    <td>68.1</td>
    <td>54.7</td>
    <td>10.6</td>
    <td>72.2</td>
    <td>58.7</td>
    <td>61.7</td>
    <td>48.9</td>
    <td>51.9</td>
    <td>67.9</td>
    <td>57.8</td>
  </tr>
  <tr>
    <td>BitFit<sub>1</sub></td>
    <td>280K</td>
    <td>85.3</td>
    <td>90.1</td>
    <td>93.0</td>
    <td>94.2</td>
    <td>90.9</td>
    <td>86.8</td>
    <td>67.6</td>
    <td>58.2</td>
    <td>83.3</td>
    <td>74.5</td>
    <td>79.6</td>
    <td>70.0</td>
    <td>59.6</td>
    <td>78.6</td>
    <td>72.5</td>
  </tr>
  <tr>
    <td>Adapter<sub>1</sub></td>
    <td>1.9M</td>
    <td><b>86.5</b></td>
    <td>90.2</td>
    <td>93.2</td>
    <td>93.8</td>
    <td>90.7</td>
    <td>85.3</td>
    <td>71.9</td>
    <td>64.0</td>
    <td>84.5</td>
    <td><b>75.9</b></td>
    <td><b>82.5</b></td>
    <td>67.1</td>
    <td>67.3</td>
    <td>85.7</td>
    <td>75.7</td>
  </tr>
  <tr>
    <td>LoRA<sub>1</sub></td>
    <td>3.8M</td>
    <td>86.3</td>
    <td>89.0</td>
    <td>93.2</td>
    <td>94.3</td>
    <td>90.9</td>
    <td>90.1</td>
    <td>75.5</td>
    <td>63.3</td>
    <td>85.3</td>
    <td>72.6</td>
    <td>81.3</td>
    <td>68.3</td>
    <td>67.3</td>
    <td>92.9</td>
    <td>76.5</td>
  </tr>
  <tr>
    <td>DePT<sub>1</sub></td>
    <td>76.8K</td>
    <td>85.0</td>
    <td>90.4</td>
    <td>93.2</td>
    <td>94.2</td>
    <td>90.8</td>
    <td><b>90.7</b></td>
    <td>79.1</td>
    <td>63.8</td>
    <td>85.9</td>
    <td>74.3</td>
    <td>79.3</td>
    <td>68.7</td>
    <td>67.3</td>
    <td>92.9</td>
    <td>76.5</td>
  </tr>
  <tr>
    <td>Finetuning<sub>1</sub></td>
    <td>28M</td>
    <td>85.7</td>
    <td>91.1</td>
    <td>92.0</td>
    <td>92.5</td>
    <td>88.8</td>
    <td>90.2</td>
    <td>75.4</td>
    <td>54.9</td>
    <td>83.8</td>
    <td>74.4</td>
    <td>81.1</td>
    <td>70.0</td>
    <td>71.2</td>
    <td>85.7</td>
    <td>76.1</td>
  </tr>
  <tr>
    <td>Adapters<sub>1</sub></td>
    <td>1.8M</td>
    <td>86.3</td>
    <td><b>90.5</b></td>
    <td>93.2</td>
    <td>93.0</td>
    <td>89.9</td>
    <td>90.2</td>
    <td>70.3</td>
    <td>61.5</td>
    <td>84.4</td>
    <td>72.6</td>
    <td>82.3</td>
    <td>66.5</td>
    <td>67.3</td>
    <td>89.3</td>
    <td>75.6</td>
  </tr>
  <tr>
    <td>HyperFomer<sub>1</sub></td>
    <td>638K</td>
    <td>85.7</td>
    <td>90.0</td>
    <td>93.0</td>
    <td>94.0</td>
    <td>89.7</td>
    <td>87.2</td>
    <td>75.4</td>
    <td>63.7</td>
    <td>84.8</td>
    <td>72.9</td>
    <td>82.5</td>
    <td>69.0</td>
    <td>67.3</td>
    <td>85.7</td>
    <td>75.4</td>
  </tr>
  <tr>
    <td>HyperDecoder<sub>1</sub></td>
    <td>1.8M</td>
    <td>86.0</td>
    <td><b>90.5</b></td>
    <td><b>93.4</b></td>
    <td>90.4</td>
    <td>90.5</td>
    <td>87.7</td>
    <td>71.7</td>
    <td>55.9</td>
    <td>83.7</td>
    <td>70.4</td>
    <td>78.8</td>
    <td>67.1</td>
    <td>61.5</td>
    <td>82.1</td>
    <td>72.0</td>
  </tr>
  <tr>
    <td colspan="17">transfer learning + one for one</td>
  </tr>
  <tr>
    <td>SPoT<sub>1</sub></td>
    <td>76.8K</td>
    <td>85.4</td>
    <td>90.1</td>
    <td>93.0</td>
    <td>93.4</td>
    <td>90.0</td>
    <td>79.7</td>
    <td>69.8</td>
    <td>57.1</td>
    <td>82.3</td>
    <td>74.0</td>
    <td>77.2</td>
    <td>67.0</td>
    <td>50.0</td>
    <td>46.4</td>
    <td>62.9</td>
  </tr>
  <tr>
    <td colspan="17">transfer learning + all for one</td>
  </tr>
  <tr>
    <td>ATTEMPT<sub>1</sub></td>
    <td>232K</td>
    <td>84.3</td>
    <td>90.3</td>
    <td>93.0</td>
    <td>93.2</td>
    <td>89.7</td>
    <td>85.7</td>
    <td>73.4</td>
    <td>57.4</td>
    <td>83.4</td>
    <td>74.4</td>
    <td>78.8</td>
    <td>66.8</td>
    <td>53.8</td>
    <td>78.6</td>
    <td>70.5</td>
  </tr>
  <tr>
    <td>MPT<sub>3</sub></td>
    <td>77.6K</td>
    <td>85.9</td>
    <td>90.3</td>
    <td>93.1</td>
    <td>93.8</td>
    <td>90.4</td>
    <td>89.1</td>
    <td>79.4</td>
    <td>62.4</td>
    <td>85.6</td>
    <td>74.8</td>
    <td>79.6</td>
    <td>69.0</td>
    <td>67.3</td>
    <td>79.8</td>
    <td>74.1</td>
  </tr>
  <tr>
    <td>TPPT<sub>2</sub></td>
    <td>539K</td>
    <td>85.5</td>
    <td>90.1</td>
    <td>93.2</td>
    <td><b>94.7</b></td>
    <td>89.8</td>
    <td>89.7</td>
    <td>82.3</td>
    <td>59.8</td>
    <td>85.6</td>
    <td>74.4</td>
    <td>80.1</td>
    <td>69.8</td>
    <td>67.3</td>
    <td>94.6</td>
    <td>77.2</td>
  </tr>
  <tr>
    <td>ATTEMPT<sub>1</sub></td>
    <td>96K</td>
    <td>83.8</td>
    <td>90.0</td>
    <td>93.1</td>
    <td>93.7</td>
    <td>90.8</td>
    <td>86.1</td>
    <td>79.9</td>
    <td>63.4</td>
    <td>85.2</td>
    <td>74.4</td>
    <td>78.3</td>
    <td>66.5</td>
    <td>69.2</td>
    <td>82.1</td>
    <td>74.1</td>
  </tr>
  <tr>
    <td>MPT<sub>3</sub></td>
    <td>10.5K</td>
    <td>84.3</td>
    <td>90.0</td>
    <td>93.0</td>
    <td>93.3</td>
    <td>90.4</td>
    <td>89.2</td>
    <td>82.7</td>
    <td>63.5</td>
    <td>85.8</td>
    <td>74.8</td>
    <td>79.2</td>
    <td>70.2</td>
    <td>67.3</td>
    <td>89.3</td>
    <td>76.1</td>
  </tr>
  <tr>
    <td colspan="17">transfer learning + part for one</td>
  </tr>
  <tr>
    <td>DTVG (ours)</td>
    <td>77.5K</td>
    <td>86.0<sub>1.2</sub></td>
    <td>90.3<sub>0.1</sub></td>
    <td>93.1<sub>0.0</sub></td>
    <td>93.2<sub>0.0</sub></td>
    <td>91.0<sub>0.2</sub></td>
    <td>90.4<sub>0.2</sub></td>
    <td>86.3<sub>1.6</sub></td>
    <td>69.1<sub>1.0</sub></td>
    <td>87.4<sub>1.7</sub></td>
    <td>74.5<sub>0.7</sub></td>
    <td>81.4<sub>1.0</sub></td>
    <td>71.1<sub>1.6</sub></td>
    <td>69.9<sub>1.6</sub></td>
    <td>97.6<sub>1.6</sub></td>
    <td>78.9</td>
  </tr>
</table>

Table 2: Results on GLUE and SuperGLUE benchmark. “param\ task” denotes the number of learnable parameters for each task on the GLUE. * denotes multi-task learning on target tasks. 1 sourced from Asai et al. (2022), 2 sourced from Wu et al. (2023), 3 sourced from Wang et al. (2023) and 4 sourced from Shi and Lipani (2024). We differentiate high-resource and low-resource tasks using gray and blue, respectively, to highlight our contribution.

<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">k</th>
    <th colspan="6">Method</th>
  </tr>
  <tr>
    <th>FT</th>
    <th>PT</th>
    <th>HF</th>
    <th>ATP</th>
    <th>MPT</th>
    <th>DePT</th>
    <th>Our</th>
  </tr>
  <tr>
    <td rowspan="3">BoolQ</td>
    <td>4</td>
    <td>50.5</td>
    <td>61.6</td>
    <td>48.0</td>
    <td>61.8</td>
    <td>62.2</td>
    <td><b>62.7<sub>5.4</sub></b></td>
    <td>60.6<sub>1.5</sub></td>
  </tr>
  <tr>
    <td>16</td>
    <td>56.5</td>
    <td>61.9</td>
    <td>50.2</td>
    <td>60.0</td>
    <td>63.3</td>
    <td>66.9<sub>4.4</sub></td>
    <td><b>72.3<sub>1.4</sub></b></td>
  </tr>
  <tr>
    <td>32</td>
    <td>58.4</td>
    <td>61.7</td>
    <td>58.3</td>
    <td>65.3</td>
    <td>68.9</td>
    <td>67.2<sub>4.1</sub></td>
    <td><b>75.1<sub>5.1</sub></b></td>
  </tr>
  <tr>
    <td rowspan="3">CB</td>
    <td>4</td>
    <td>57.7</td>
    <td>53.5</td>
    <td>60.7</td>
    <td><b>82.1</b></td>
    <td>73.6</td>
    <td>75.0<sub>5.1</sub></td>
    <td><b>86.9<sub>1.7</sub></b></td>
  </tr>
  <tr>
    <td>16</td>
    <td>77.0</td>
    <td>63.5</td>
    <td>76.3</td>
    <td>78.5</td>
    <td>78.6</td>
    <td>82.1<sub>2.9</sub></td>
    <td><b>82.1<sub>2.9</sub></b></td>
  </tr>
  <tr>
    <td>32</td>
    <td>80.0</td>
    <td>67.8</td>
    <td>81.4</td>
    <td><b>85.7</b></td>
    <td>82.1</td>
    <td>82.1<sub>2.1</sub></td>
    <td>84.5<sub>1.7</sub></td>
  </tr>
  <tr>
    <td rowspan="3">SciTail</td>
    <td>4</td>
    <td>79.6</td>
    <td>57.7</td>
    <td><b>82.0</b></td>
    <td>80.2</td>
    <td>80.2</td>
    <td>78.1<sub>2.5</sub></td>
    <td>78.3<sub>1.1</sub></td>
  </tr>
  <tr>
    <td>16</td>
    <td>80.0</td>
    <td>60.8</td>
    <td>86.5</td>
    <td>79.5</td>
    <td><b>87.3</b></td>
    <td>78.5<sub>1.4</sub></td>
    <td>82.1<sub>2.9</sub></td>
  </tr>
  <tr>
    <td>32</td>
    <td>81.9</td>
    <td>60.2</td>
    <td>85.8</td>
    <td>80.2</td>
    <td><b>86.3</b></td>
    <td>85.4<sub>3.1</sub></td>
    <td>85.3<sub>2.5</sub></td>
  </tr>
</table>

Table 3: Few-shot adaptation on BoolQ, CB, and SciTail datasets, where FT, HF, ATP denote Finetuning, HyperFomer, and ATTEMPT, respectively.

![Ablation study for the source task grouping.](page_1012_1042_410_180.png)

Figure 4: Ablation study for the source task grouping.

![Ablation study for multi-task prompt transfer.](page_1012_1232_410_180.png)

Figure 5: Ablation study for multi-task transfer strategy.

transfer strategies, including 1) only target: This strategy focuses solely on learning the target task prompt and its associated scaling term for the task prompt vectors. 2) fix group: This strategy fixes the initial source task group, thus eliminating the effect of dynamic grouping, which relies on the specific grouping of source tasks. Figure 5 shows that using a fixed group of source tasks results in a performance drop (77.4 vs. 75.2), suggesting that the choice of the source task group is important. This emphasizes the need for our approach, namely DTVG’s ability to efficiently group source tasks by dynamic iteration, thereby improving performance.

4.4 Additional Analysis

We extend our experiments to comprehensively evaluate the performance of DTVG, including model scaling, natural language generation, generalization to other LLMs, and dynamic grouping during training. However, for some experiments without a standard evaluation protocol, we analyze DTVG only against some fundamental baselines.

Model Scaling Figure 6 illustrates the results on three SuperGLUE datasets with different scales of the T5 model. We observe that as the model size increases, performance across different tasks improves. This indicates that our method indeed benefits from a larger model capacity. Please refer
Figure 6: Model Scaling on BoolQ, MultiRC, and WiC.

<table>
  <tr>
    <th>Task</th>
    <th>Method</th>
    <th colspan="4">Metris</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>BLEU</td>
    <td>R-1</td>
    <td>R-2</td>
    <td>R-L</td>
  </tr>
  <tr>
    <td>E2E</td>
    <td>PT</td>
    <td>0.274</td>
    <td>62.1</td>
    <td>36.3</td>
    <td>47.0</td>
  </tr>
  <tr>
    <td></td>
    <td>DTVG</td>
    <td>0.331</td>
    <td>63.6</td>
    <td>37.5</td>
    <td>47.9</td>
  </tr>
  <tr>
    <td>CommonGen</td>
    <td>PT</td>
    <td>0.056</td>
    <td>33.3</td>
    <td>9.9</td>
    <td>27.6</td>
  </tr>
  <tr>
    <td></td>
    <td>DTVG</td>
    <td>0.067</td>
    <td>36.6</td>
    <td>11.0</td>
    <td>29.1</td>
  </tr>
  <tr>
    <td>WebNLG</td>
    <td>PT</td>
    <td>0.293</td>
    <td>64.4</td>
    <td>39.6</td>
    <td>52.3</td>
  </tr>
  <tr>
    <td></td>
    <td>DTVG</td>
    <td>0.363</td>
    <td>66.3</td>
    <td>41.4</td>
    <td>53.4</td>
  </tr>
</table>

Table 4: Performance on NLG tasks. R-1, R-2, and R-L denote Rouge-1, Rouge-2, and Rouge-L, respectively.

to Appendix C.4 for experiment details.

Natural Language Generation As shown in Table 4, we observe that DTVG consistently outperforms PT on three natural language generation tasks (namely, E2E (Dušek et al., 2019), CommonGen (Lin et al., 2020), and WebNLG (Gardent et al., 2017)), suggesting DTVG works not only for NLU but also for NLG. Interestingly, although we transfer TPV from NLU tasks to NLG tasks, DTVG’s performance on NLG tasks does not degrade, which aligns with the same observation (Wang et al., 2023). We suspect that this phenomenon might be related to T5’s text-to-text framework. Please see Appendix C.5 for details.

Generalization to Other LLMs We experimentally analyze the performance of DTVG on the latest decoder-based models using Llama-3.2-1B, Llama-3.2-3B and Llama-3-8B (Dubey et al., 2024). As shown in Table 5, DTVG outperforms vanilla prompt tuning across various target tasks. When compared with SPoT, DTVG demonstrates consistent positive transfer across various LLMs, whereas SPoT exhibits negative transfer, such as on RTE with Llama-3.2-1B (74.8% vs. 57.6%). These results suggest that DTVG’s generalizability to other types of LLMs. Moreover, we observe that DTVG performs better on Llama-3.2-3B than Llama-3.2-1B, indicating that it benefits from more powerful LLMs. Please see Appendix C.6 for experiment details.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Task</th>
  </tr>
  <tr>
    <th>RTE</th>
    <th>CoLA</th>
    <th>CB</th>
    <th>WSC</th>
  </tr>
  <tr>
    <th colspan="5">LLama-3.2-1B</th>
  </tr>
  <tr>
    <td>PT</td>
    <td>74.8</td>
    <td>59.2</td>
    <td>60.7</td>
    <td>63.5</td>
  </tr>
  <tr>
    <td>SPoT</td>
    <td>57.6</td>
    <td>67.5</td>
    <td>64.3</td>
    <td>67.3</td>
  </tr>
  <tr>
    <td>DTVG</td>
    <td>84.1</td>
    <td>63.4</td>
    <td>82.1</td>
    <td>67.3</td>
  </tr>
  <tr>
    <th colspan="5">LLama-3.2-3B</th>
  </tr>
  <tr>
    <td>PT</td>
    <td>60.4</td>
    <td>67.2</td>
    <td>64.3</td>
    <td>67.3</td>
  </tr>
  <tr>
    <td>SPoT</td>
    <td>63.3</td>
    <td>71.7</td>
    <td>60.7</td>
    <td>67.3</td>
  </tr>
  <tr>
    <td>DTVG</td>
    <td>89.2</td>
    <td>73.1</td>
    <td>89.3</td>
    <td>69.2</td>
  </tr>
  <tr>
    <th colspan="5">LLama-3-8B</th>
  </tr>
  <tr>
    <td>PT</td>
    <td>83.5</td>
    <td>69.2</td>
    <td>57.1</td>
    <td>67.3</td>
  </tr>
  <tr>
    <td>SPoT</td>
    <td>84.9</td>
    <td>70.3</td>
    <td>60.7</td>
    <td>67.3</td>
  </tr>
  <tr>
    <td>DTVG</td>
    <td>84.9</td>
    <td>72.6</td>
    <td>86.3</td>
    <td>71.4</td>
  </tr>
</table>

Table 5: Results on Llama-3.2-1B, Llama-3.2-3B and Llama-3-8B

Dynamic Grouping Figure 7 illustrates the variations of dynamic grouping for RTE during the training process. Compared to prompt tuning, DTVG achieves better performance on RTE.

From the task grouping perspective, we observe the source task combination shifts from [top1: MNLI, top2: SST2] to [top1: MNLI, top2: QNLI] over time. This result suggests that a) Target Similarity: two NLI source tasks become more aligned to the target task RTE (NLI); and b) Knowledge Consistency: conflicts exist between MNLI and SST2 (replaced by QNLI) are reduced.

From an iterative training perspective, we observe that source task groups fluctuate frequently during the early stages of training. As training progresses, the task group converges, resulting in a stable selection of tasks in the final stage. This supports our hypothesis that insufficient convergence is attributed to the low-resource characteristics of the target tasks. Additionally, we report the grouping results of MRPC, NQ, and SciTail in Appendix H.

5 Conclusion

In this paper, we present DTVG, a novel approach for addressing potential negative transfer in multi-task prompt tuning based on task prompt vectors. Compared to vanilla transfer of the soft prompt from all source tasks, we dynamically group a subset of source tasks and merge their task prompt vectors to avoid an unrelated source task inducing performance degradation of the target task. Extensive
Figure 7: Validation performance on RTE with source task grouping. The source tasks are arranged in each patch legend from left to right, ordered by their similarity to the target task, from highest to lowest.

experiments demonstrate that DTVG effectively groups related source tasks to further optimize the performance of the target task.

References

Akari Asai, Mohammadreza Salehi, Matthew E Peters, and Hannaneh Hajishirzi. 2022. Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6655–6672.

Robert Belanec, Simon Ostermann, Ivan Srba, and Maria Bielikova. 2024. Task prompt vectors: Effective initialization through multi-task soft-prompt transfer. arXiv preprint arXiv:2408.01119.

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.

Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, pages 107–124.

Dorotyya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. arXiv preprint arXiv:1809.02922.

Bill Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005).

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179.

Ondřej Dušek, David M. Howcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural language generation. In Proceedings of the 12th International Conference on Natural Language Generation, pages 421–426, Tokyo, Japan. Association for Computational Linguistics.

Lingyun Feng. 2023. Learning to predict task transferability via soft prompt. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8829–8844.

Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from RDF data. In Proceedings of the 10th International Conference on Natural Language Generation, INLG 2017, Santiago de Compostela, Spain, September 4-7, 2017, pages 124–133. Association for Computational Linguistics.

Sebastian Gehrmann, Tosin Adewumi, Karmanyaa Aggarwal, Pawan Sarasaka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinneye Emzeue, Varun Gangal, Cristina Barbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahmood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjian Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96–120, Online. Association for Computational Linguistics.

Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9.

Neil Houlby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea
Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Hamish Ivison and Matthew E Peters. 2022. Hyperdecoders: Instance-specific decoders for multi-task nlp. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1715–1730.

Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252–262.

Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of the AAAI conference on artificial intelligence.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059.

Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, pages 552–561.

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4582–4597. Association for Computational Linguistics.

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. Commongen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1823–1840. Association for Computational Linguistics.

Yu-Chen Lin, Wei-Hua Li, Jun-cheng Chen, and Chu-Song Chen. 2024. Accept: Adaptive codebook for composite and efficient prompt tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15345–15358.

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 565–576.

Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. 2024. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36.

Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.

Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267–1273.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

P Rajpurkar. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.
Zhengxiang Shi and Aldo Lipani. 2024. Dept: Decomposed prompt tuning for parameter-efficient fine-tuning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200.

Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2022. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5039–5059.

Alex Wang. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.

Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogério Feris, Huan Sun, and Yoon Kim. 2023. Multitask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

A Warstadt. 2019. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471.

Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122.

Muling Wu, Wenhao Liu, Jianhan Xu, Changze Lv, Zixuan Ling, Tianlong Li, Longtao Huang, Xiaoqing Zheng, and Xuanjing Huang. 2023. Parameter efficient multi-task fine-tuning by learning to transfer token-wise prompts. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 8734–8746. Association for Computational Linguistics.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380.

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9.

Frederic Z Zhang, Paul Albert, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbaspnejad. 2024. Knowledge composition using task vectors with learned anisotropic scaling. arXiv preprint arXiv:2407.02880.

Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885.

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.

Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308.
Appendix

A Performance and Parameter Comparison

We visualize the average score (y-axis) and parameter (x-axis) on the GLUE benchmark across various baselines in Figure 8. We observe DTVG surpassing other baselines and achieving SOTA performance with minimal parameters.

![Performance & parameter comparison.](page_184_370_563_324.png)

Figure 8: Performance & parameter comparison.

B Dataset Details

We use 26 datasets in total from 5 benchmarks. We use GLUE and SuperGLUE benchmarks to test the model’s natural language understanding. MNLI (Williams et al., 2018), QNLI (Demszky et al., 2018), QQP (Wang, 2018), SST2 (Socher et al., 2013), RTE (Giampiccolo et al., 2007), CoLA (Warstadt, 2019), STS-B (Cer et al., 2017), MRPC (Dolan and Brockett, 2005) are derived from GLUE. MultiRC (Khashabi et al., 2018), BoolQ (Clark et al., 2019), WiC (Pilehvar and Camacho-Collados, 2019), WSC (Levesque et al., 2012), and CB (De Marneffe et al., 2019), ReCoRD (Zhang et al., 2018) are from SuperGLUE. We use four question-answering datasets from the MRQA 2019 benchmarks, including Natural Questions (NQ) (Kwiatkowski et al., 2019), HotpotQA (HQ) (Yang et al., 2018), NewsQA (News) (Trischler et al., 2017), and SearchQA (SQA) (Dunn et al., 2017), SQuAD (Rajpurkar, 2016). WinoGrande (WG) (Sakaguchi et al., 2021), YelpPolarity (Yelp) (Zhang et al., 2015), SciTail (Khot et al., 2018) and PAWS-Wiki (PAWS) (Zhang et al., 2019) are from the ‘other’ benchmark to test model’ generalizability across different domains. We also use CommonGen (Lin et al., 2020), E2E (Dušek et al., 2019), and WebNLG (Gardent et al., 2017) sourced from the GEM (Gehrmann et al., 2021) benchmark to test the model’s performance on natural language generation. We download all datasets from the huggingface dataset1. Table 6 lists more details about each dataset.

C Implementation Details

We use PyTorch2, huggingface transformers3 to implement our method. We validate the effectiveness of DTVG based on the open-source repository 4. All of the experiments are conducted with a single GPU with 32 GB of memory. Following Asai et al. (2022), we use the original T5 checkpoint. We set the batch size for T5-base as 32 for most datasets. We set the batch size to 16 and the gradient accumulation step to 2 for the MRQA benchmark with a long context. Due to the different input lengths of various datasets, we set the maximum token length of 256 for most datasets that have a context of fewer than 200 tokens. We set the maximum token length of 348 for MultiRC and 512 for MRQA datasets. We limit the maximum training data number of YelpPolarity to 100k. We maintain the same hyperparameter settings (Lester et al., 2021) to reinitialize and retrain all tasks, aiming to reconstruct the corresponding soft prompts and task prompt vectors. Similar to (Mahabadi et al., 2021), for datasets lacking publicly available test sets, we use the validation set as the test set or partition it to create separate test and validation sets.

C.1 Comparison of Task Prompt Vectors and Soft prompt

We used the reconstructed soft prompts with the same initialization to compare SPoT (Vu et al., 2022) and TPV. Specifically, we initialize the target task prompt with the soft prompt that obtained the highest metric score from six source tasks (namely, MNLI, QNLI, QQP, SST-2, ReCoRD, and SQuAD). Note that the difference between the implementations of the two methods SPoT and TPV is only in the task similarity metric. SPoT uses the traditional cosine similarity of soft prompts, while TPV uses Eqn.2 to compute task similarity.

C.2 Full-dataset Transfer

We set warmup steps to be 500, weight decay to be \( 1 \times 10^{-5} \), and use Adam (Kingma and Ba, 2015) for optimization with a linear learning rate scheduler.

1https://github.com/huggingface/datasets
2https://pytorch.org/
3https://github.com/huggingface/transformers
4https://github.com/AkariAsai/ATTEMPT
<table>
  <tr>
    <th>Dataset</th>
    <th>Source</th>
    <th>Target</th>
    <th>Benchmark</th>
    <th>Task Type</th>
    <th>Domain</th>
    <th>Metric</th>
  </tr>
  <tr>
    <td>MNLI</td>
    <td>✓</td>
    <td>✓</td>
    <td>GLUE</td>
    <td>Natural Language Inference</td>
    <td>Various</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>QQP</td>
    <td>✓</td>
    <td>✓</td>
    <td>GLUE</td>
    <td>Paraphrase Detection</td>
    <td>Social QA</td>
    <td>Accuracy & F1</td>
  </tr>
  <tr>
    <td>QNLI</td>
    <td>✓</td>
    <td>✓</td>
    <td>GLUE (QA)</td>
    <td>Natural Language Inference</td>
    <td>Wikipedia</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>SST2</td>
    <td>✓</td>
    <td>✓</td>
    <td>GLUE</td>
    <td>Sentiment Analysis</td>
    <td>Movie Reviews</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>STS-B</td>
    <td>×</td>
    <td>✓</td>
    <td>GLUE</td>
    <td>Sentence Similarity</td>
    <td>Various</td>
    <td>Pearson & Spearman corr.</td>
  </tr>
  <tr>
    <td>MRPC</td>
    <td>×</td>
    <td>✓</td>
    <td>GLUE</td>
    <td>Paraphrase Detection</td>
    <td>News</td>
    <td>Accuracy & F1</td>
  </tr>
  <tr>
    <td>RTE</td>
    <td>×</td>
    <td>✓</td>
    <td>GLUE</td>
    <td>Natural Language Inference</td>
    <td>News & Wikipedia</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>CoLA</td>
    <td>×</td>
    <td>✓</td>
    <td>GLUE</td>
    <td>Acceptability</td>
    <td>Various</td>
    <td>Matthews corr.</td>
  </tr>
  <tr>
    <td>ReCoRD</td>
    <td>✓</td>
    <td>×</td>
    <td>SuperGLUE</td>
    <td>Question Answering (QA)</td>
    <td>News</td>
    <td>F1 & EM</td>
  </tr>
  <tr>
    <td>MultiRC</td>
    <td>×</td>
    <td>✓</td>
    <td>SuperGLUE</td>
    <td>Question Answering (QA)</td>
    <td>Various</td>
    <td>F1 & EM</td>
  </tr>
  <tr>
    <td>BoolQ</td>
    <td>×</td>
    <td>✓</td>
    <td>SuperGLUE</td>
    <td>Question Answering (QA)</td>
    <td>Wikipedia</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>WiC</td>
    <td>×</td>
    <td>✓</td>
    <td>SuperGLUE</td>
    <td>Word Sense Disambiguation</td>
    <td>Lexical databases</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>WSC</td>
    <td>×</td>
    <td>✓</td>
    <td>SuperGLUE</td>
    <td>Common Sense Reasoning</td>
    <td>Fiction books</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>CB</td>
    <td>×</td>
    <td>✓</td>
    <td>SuperGLUE</td>
    <td>Natural Language Inference</td>
    <td>Various</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>SQuAD</td>
    <td>✓</td>
    <td>✓</td>
    <td>MRQA 2019</td>
    <td>Question Answering (QA)</td>
    <td>Wikipedia</td>
    <td>F1 & EM</td>
  </tr>
  <tr>
    <td>NQ</td>
    <td>×</td>
    <td>✓</td>
    <td>MRQA 2019</td>
    <td>Question Answering (QA)</td>
    <td>Wikipedia</td>
    <td>F1 & EM</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>×</td>
    <td>✓</td>
    <td>MRQA 2019</td>
    <td>Question Answering (QA)</td>
    <td>Wikipedia</td>
    <td>F1 & EM</td>
  </tr>
  <tr>
    <td>SearchQA</td>
    <td>×</td>
    <td>✓</td>
    <td>MRQA 2019</td>
    <td>Question Answering (QA)</td>
    <td>Search snippets</td>
    <td>F1 & EM</td>
  </tr>
  <tr>
    <td>NewsQA</td>
    <td>×</td>
    <td>✓</td>
    <td>MRQA 2019</td>
    <td>Question Answering (QA)</td>
    <td>News</td>
    <td>F1 & EM</td>
  </tr>
  <tr>
    <td>WinoGrande</td>
    <td>×</td>
    <td>✓</td>
    <td>‘Other’</td>
    <td>Common Sense Reasoning</td>
    <td>WikiHow</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>YelpPolarity</td>
    <td>×</td>
    <td>✓</td>
    <td>‘Other’</td>
    <td>Sentiment Analysis</td>
    <td>Yelp reviews</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>SciTail</td>
    <td>×</td>
    <td>✓</td>
    <td>‘Other’</td>
    <td>Natural Language Inference</td>
    <td>Science exams</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>PAWS</td>
    <td>×</td>
    <td>✓</td>
    <td>‘Other’</td>
    <td>Paraphrase Detection</td>
    <td>Wikipedia</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>WebNLG</td>
    <td>×</td>
    <td>✓</td>
    <td>GEM</td>
    <td>Data to Text (NLG)</td>
    <td>Various</td>
    <td>Automated Evaluation</td>
  </tr>
  <tr>
    <td>E2E</td>
    <td>×</td>
    <td>✓</td>
    <td>GEM</td>
    <td>Data to Text (NLG)</td>
    <td>Restaurant</td>
    <td>Automated Evaluation</td>
  </tr>
  <tr>
    <td>CommonGen</td>
    <td>×</td>
    <td>✓</td>
    <td>GEM</td>
    <td>Data to Text (NLG)</td>
    <td>Commonsense</td>
    <td>Automated Evaluation</td>
  </tr>
</table>

Table 6: Details about 26 datasets from 5 Benchmarks in total. GLUE (QA) denotes the QNLI derived from the Question Answering Dataset (SQuAD). Lexical databases contain WordNet, VerbNet, and Wiktionary. Search snippets denote question answering from the search engine. Automated Evaluation includes BLEU, Rouge-1, Rouge-2, and Rouge-L. Following Shi and Lipani (2024), we use the metric marked with an underline as the primary evaluation metric.

C.3 Few-shot Adaptation

In few-shot adaptation experiments, followed by (Mahabadi et al., 2021), we run experiments three times with different random seeds and take the mean of the performance. In each trial, we train 1k steps on the target task for both task prompt vector learning and multi-task prompt transfer stage, which we found to be able to achieve full convergence. We evaluate every 50 steps on the original validation set. For the rest, we report on the original test sets based on the best checkpoint on the validation set.

C.4 Model Scale

For model scaling experiments, we set the batch sizes are 100 and 16 for T5-small and T5-large, respectively.

C.5 Other LLMs

We use Llama-3.2-1B, Llama-3.2-3B and Llama-3-8B to test DTVG’s generalizability on other types of LLMs. In our experiment, we use the same 6 source tasks as our main experiments setting on the T5-base and select RTE, CoLA, CB, and WSC as target tasks. We set the length of the soft prompt to 100 for both models and set the batch size to 16, 4 and 2 for Llama-3.2-1B, Llama-3.2-3B and Llama-3-8B, respectively. Compared to encoder-decoder-based models, we observe that decoder-based autoregressive models require a smaller learning rate. Therefore, we set the learning rate of the soft prompt and its corresponding scaling term to 0.001 for Llama-3.2-1B and Llama-3.2-3B, and to 0.0001 for Llama-3-8B.

C.6 Natural Language Generation

We select E2E, CommonGen, and WebNLG sourced from the GEM benchmark to evaluate DTVG’s performance on natural language generation (NLG) tasks. We use T5-base as the backbone and reuse the task prompt vectors sourced from 6 natural language understanding (NLU) source tasks. We set the maximum 128 token length for both the input and output. We use the target as a simple reference to compute metrics for both PT and DTVG and report the best result on the valida-
C.7  Two Speed Learning Rate

For the full-dataset transfer setting, we search the learning rate within the set {3e-1, 4e-1, 5e-1} for the target task prompt and corresponding scaling term. For the scaling term of the source prompt task vectors, we search the learning rate within the set {4e-1, 6e-1, 8e-1, 1}. For few-shot adaptation and others, we set the learning rate of 0.3 for both the target task prompt and corresponding scaling term, and 0.4 for the scaling term of the source task prompt vectors.

C.8  Prompt Initialization

We initialized the soft prompt by randomly sampling the top 5000 vocabulary words for all tasks. In both full-dataset transfer and few-shot adaptation experiments, we utilize soft prompt tasks vectors from source tasks by full-dataset prompt tuning. In few-shot adaptation setting, we exclude the corresponding task prompt vectors when adapt to source tasks in GLUE.

D  MRQA and ‘Other’ Benchmark

As shown in Table 7, DTVG realizes significant improvements over the vanilla prompt tuning with a 3.7%, 14.2% increase on MRQA and ‘Other’ in terms of relative average performance. Compared to other baselines, DTVG also achieves comparable or better performance on MRQA and ‘other’ benchmarks.

E  Few-shot adaptation On GLUE and SuperGLUE benchmark

We compare our method with no transfer baseline PT, one for one baseline DePT, and all for one baseline MPT, and Table 8 shows the evaluation results on GLUE and SuperGLUE benchmarks. Our method can substantially improve the few-shot adaptation results in the most of settings. Specifically, compared to PT, our method on average improves the results across only (k= 4,16,32) shots. Meanwhile, our method also surpasses MPT and DePT, in terms of performance.

F  Algorithm Details about DTVG

We give all the implementation details about DTVG for multi-task prompt tuning on Algorithm 1.

![Validation Performance on MRPC, SciTail, and NQ with source task grouping.](page_1012_120_393_393.png)

Figure 9: Validation Performance MRPC, SciTail, and NQ with source task grouping.

G  Details about Source Task Grouping

Implementation details about Source Task Grouping for addressing optimization objective 5 are presented in Algorithm 2.

H  Source Task Grouping

Figure 9 demonstrates that DTVG consistently outperforms vanilla prompt tuning in terms of performance across the MRQA, SciTail, and NQ datasets. Meanwhile, DTVG achieves dynamic grouping a appropriate task subset for the different target task. This indicates that DTVG is capable of effectively group a related source task combination tailored to different target tasks, thereby reduce negative transfer.

I  Task similarity

TPV represents the change in parameters after fine-tuning from its initial parameters on a specific task and reflects the specific optimization direction of a task in the weight space. When we fix a unified initialization for all tasks, effectively constraining them to the same weight space, it means that when two TPVs are closer, their optimization directions are more aligned. As a result, when transferring between tasks, there will be fewer conflicts.

We conduct a case study using 5 source tasks and 6 target tasks with the same initialization to analyze the effectiveness of TPV in capturing the relation-
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">param \ task</th>
    <th colspan="4">MRQA</th>
    <th colspan="5">Other</th>
  </tr>
  <tr>
    <th>NQ (100k)</th>
    <th>HP (72K)</th>
    <th>SQA (117K)</th>
    <th>News (74K)</th>
    <th>Avg.</th>
    <th>WG (40K)</th>
    <th>Yelp (100k)</th>
    <th>SciTail (27K)</th>
    <th>PAWS (49K)</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Finetuning<sub>1</sub></td>
    <td>220M</td>
    <td>75.1</td>
    <td>77.5</td>
    <td>81.1</td>
    <td>65.2</td>
    <td>74.7</td>
    <td>61.9</td>
    <td>96.7</td>
    <td>95.8</td>
    <td>94.1</td>
    <td>87.1</td>
  </tr>
  <tr>
    <td>Adapters<sub>1</sub></td>
    <td>1.9M</td>
    <td>74.2</td>
    <td>77.6</td>
    <td>81.4</td>
    <td>65.6</td>
    <td>74.7</td>
    <td>59.2</td>
    <td>96.9</td>
    <td>94.5</td>
    <td>94.3</td>
    <td>86.2</td>
  </tr>
  <tr>
    <td>BitFit<sub>1</sub></td>
    <td>280K</td>
    <td>70.7</td>
    <td>75.5</td>
    <td>77.7</td>
    <td>64.1</td>
    <td>72.0</td>
    <td>57.2</td>
    <td>94.7</td>
    <td>94.7</td>
    <td>92.0</td>
    <td>84.7</td>
  </tr>
  <tr>
    <td>PT<sub>1</sub></td>
    <td>76.8K</td>
    <td>67.9</td>
    <td>72.9</td>
    <td>75.7</td>
    <td>61.1</td>
    <td>69.4</td>
    <td>49.6</td>
    <td>95.1</td>
    <td>87.9</td>
    <td>55.8</td>
    <td>72.1</td>
  </tr>
  <tr>
    <td>LoRA<sub>3</sub></td>
    <td>3.8M</td>
    <td>72.4</td>
    <td>62.3</td>
    <td>72.5</td>
    <td>56.9</td>
    <td>66.0</td>
    <td>58.2</td>
    <td>97.1</td>
    <td>94.7</td>
    <td>94.0</td>
    <td>86.0</td>
  </tr>
  <tr>
    <td>SPoT<sub>1</sub></td>
    <td>76.8K</td>
    <td>68.2</td>
    <td>74.8</td>
    <td>75.3</td>
    <td>58.2</td>
    <td>69.1</td>
    <td>50.4</td>
    <td>95.4</td>
    <td>91.2</td>
    <td>91.1</td>
    <td>82.0</td>
  </tr>
  <tr>
    <td>ATTEMPT<sub>1</sub></td>
    <td>232K</td>
    <td>70.4</td>
    <td>75.2</td>
    <td>77.3</td>
    <td>62.8</td>
    <td>71.4</td>
    <td>57.6</td>
    <td>96.7</td>
    <td>93.1</td>
    <td>92.1</td>
    <td>84.9</td>
  </tr>
  <tr>
    <td>MPT<sub>2</sub></td>
    <td>77.6K</td>
    <td>72.0<sub>1</sub>, 75.8<sub>1</sub>, 77.2<sub>1</sub>, 63.7<sub>1</sub>, 72.2</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>56.5<sub>0.9</sub>, 96.4<sub>0.0</sub>, 95.5<sub>0.1</sub>, 93.5<sub>0.1</sub>, 85.5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DePT<sub>3</sub></td>
    <td>76.8K</td>
    <td>73.2<sub>0.1</sub>, 76.8<sub>0.3</sub>, 77.6<sub>0.2</sub>, 64.4<sub>0.1</sub>, 73.0</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>59.0<sub>0.2</sub>, 96.8<sub>0.1</sub>, 95.6<sub>0.2</sub>, 93.7<sub>0.1</sub>, 86.3</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DTVG (ours)</td>
    <td>77.5K</td>
    <td>73.1<sub>0.1</sub>, 76.7<sub>0.0</sub>, 77.8<sub>0.3</sub>, 64.6<sub>0.1</sub>, 73.1</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>58.0<sub>0.0</sub>, 96.6<sub>0.1</sub>, 97.0<sub>0.1</sub>, 93.7<sub>0.0</sub>, 86.3</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 7: Performance on MRQA2019 and ‘Other’ benchmarks. “param \ task” denotes the number of learnable parameters for each task. 1 sourced from (Asai et al., 2022), 2 sourced from (Wang et al., 2023) and 3 sourced from (Shi and Lipani, 2024).

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">k-shot</th>
    <th colspan="7">GLUE</th>
    <th colspan="7">SuperGLUE</th>
  </tr>
  <tr>
    <th>MNLI</th>
    <th>QQP</th>
    <th>QNLI</th>
    <th>SST-2</th>
    <th>SST-B</th>
    <th>MRPC</th>
    <th>RTE</th>
    <th>CoLA</th>
    <th>Avg.</th>
    <th>Mult</th>
    <th>Bod</th>
    <th>WIC</th>
    <th>WiC</th>
    <th>CB</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>PT</td>
    <td>4</td>
    <td>61.5</td>
    <td>63.0</td>
    <td>80.5</td>
    <td>82.5</td>
    <td>55.7</td>
    <td>55.7</td>
    <td>55.7</td>
    <td>55.7</td>
    <td>55.7</td>
    <td>61.8</td>
    <td>61.6</td>
    <td>61.8</td>
    <td>61.8</td>
    <td>61.8</td>
    <td>55.7</td>
  </tr>
  <tr>
    <td>MPT</td>
    <td>4</td>
    <td>59.4</td>
    <td>82.0</td>
    <td>86.2</td>
    <td>56.5</td>
    <td>89.1</td>
    <td>68.1</td>
    <td>62.6</td>
    <td>34.8</td>
    <td>67.3</td>
    <td>62.2</td>
    <td>62.2</td>
    <td>52.9</td>
    <td>67.3</td>
    <td>63.6</td>
    <td>63.6</td>
  </tr>
  <tr>
    <td>DePT</td>
    <td>4</td>
    <td>44.0<sub>1</sub>, 77.4<sub>1</sub>, 85.8<sub>1</sub>, 59.3<sub>1</sub>, 84.1<sub>2</sub>, 73.5<sub>2</sub>, 63.5<sub>1</sub>, 29.3<sub>1</sub>, 64.6</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>62.3<sub>1.3</sub>, 62.7<sub>1.3</sub>, 57.5<sub>1.1</sub>, 67.9<sub>0.9</sub>, 75.0<sub>1.1</sub></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Our</td>
    <td>4</td>
    <td>49.3<sub>1</sub>, 87.5<sub>1</sub>, 80.2<sub>1</sub>, 81.8<sub>1</sub>, 87.9<sub>1</sub>, 68.1<sub>1</sub>, 72.7<sub>1</sub>, 22.4<sub>1</sub>, 68.7</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>61.4<sub>1.2</sub>, 60.6<sub>1.2</sub>, 59.4<sub>1.1</sub>, 45.2<sub>1.0</sub>, 86.9<sub>1.1</sub></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PT</td>
    <td>16</td>
    <td>41.5</td>
    <td>62.3</td>
    <td>87.4</td>
    <td>50.9</td>
    <td>87.8</td>
    <td>68.1</td>
    <td>54.7</td>
    <td>28.5</td>
    <td>56.7</td>
    <td>60.3</td>
    <td>61.9</td>
    <td>48.9</td>
    <td>44.2</td>
    <td>63.5</td>
    <td>55.8</td>
  </tr>
  <tr>
    <td>MPT</td>
    <td>16</td>
    <td>61.6</td>
    <td>84.1</td>
    <td>90.6</td>
    <td>65.2</td>
    <td>89.1</td>
    <td>71.1</td>
    <td>64.1</td>
    <td>32.1</td>
    <td>69.6</td>
    <td>64.5</td>
    <td>63.9</td>
    <td>49.8</td>
    <td>67.6</td>
    <td>64.7</td>
    <td>64.7</td>
  </tr>
  <tr>
    <td>DePT</td>
    <td>16</td>
    <td>61.0<sub>1</sub>, 80.3<sub>1</sub>, 91.2<sub>1</sub>, 65.2<sub>1</sub>, 87.1<sub>1</sub>, 78.1<sub>1</sub>, 71.9<sub>1</sub>, 27.1<sub>1</sub>, 71.9</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>64.9<sub>1.0</sub>, 59.6<sub>1.0</sub>, 57.7<sub>1.0</sub>, 78.0<sub>1.0</sub>, 64.7</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Our</td>
    <td>16</td>
    <td>58.8<sub>0.6</sub>, 81.9<sub>1</sub>, 89.8<sub>1</sub>, 88.4<sub>0.4</sub>, 86.9<sub>0.4</sub>, 76.8<sub>0.6</sub>, 31.3<sub>1</sub>, 74.8</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>61.4<sub>1.1</sub>, 72.3<sub>1.4</sub>, 60.7<sub>0.4</sub>, 67.3<sub>0.8</sub>, 82.1<sub>0.9</sub>, 68.8</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PT</td>
    <td>32</td>
    <td>37.0</td>
    <td>62.3</td>
    <td>56.7</td>
    <td>50.9</td>
    <td>87.5</td>
    <td>68.1</td>
    <td>54.7</td>
    <td>23.2</td>
    <td>55.1</td>
    <td>59.2</td>
    <td>61.7</td>
    <td>52.6</td>
    <td>67.3</td>
    <td>61.7</td>
    <td>61.7</td>
  </tr>
  <tr>
    <td>MPT</td>
    <td>32</td>
    <td>63.6</td>
    <td>88.5</td>
    <td>91.0</td>
    <td>75.9</td>
    <td>89.7</td>
    <td>74.5</td>
    <td>59.7</td>
    <td>30.8</td>
    <td>71.7</td>
    <td>63.3</td>
    <td>68.9</td>
    <td>53.9</td>
    <td>67.3</td>
    <td>82.1</td>
    <td>67.1</td>
  </tr>
  <tr>
    <td>DePT</td>
    <td>32</td>
    <td>63.3<sub>0.1</sub>, 80.1<sub>0.1</sub>, 91.5<sub>0.1</sub>, 80.4<sub>0.2</sub>, 89.2<sub>0.1</sub>, 81.4<sub>0.1</sub>, 72.7<sub>0.1</sub>, 28.6<sub>1</sub>, 73.4</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>60.1<sub>1.7</sub>, 67.2<sub>1.4</sub>, 58.0<sub>1.0</sub>, 63.1<sub>1.0</sub>, 82.1<sub>1.0</sub>, 66.4</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Our</td>
    <td>32</td>
    <td>61.2<sub>0.1</sub>, 85.3<sub>0.1</sub>, 91.2<sub>0.1</sub>, 88.3<sub>1</sub>, 83.2<sub>1</sub>, 83.1<sub>1</sub>, 74.1<sub>1</sub>, 29.3<sub>1</sub>, 74.5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>66.3<sub>1.1</sub>, 73.5<sub>1.1</sub>, 60.2<sub>1.0</sub>, 67.3<sub>0.9</sub>, 84.5<sub>1.0</sub>, 70.4</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 8: Few-shot adaptation on GLUE and SuperGLUE benchmark

Algorithm 1: DTVG

Input: source tasks set \( S = \{s^1, s^2, \ldots, s^n\} \), target task t, initialization soft prompt parameters \( P_{init} \), maximum training steps \( N_{max} \)

Output: Trained multi-task soft prompt parameters \( P_{mix}^* \)

1. **Stage 1: Task prompt vector Learning** ;
2. Initialize \( P_{init} \) for both sources and target task;
3. Boost the posterior probability and obtain their task prompt vectors;
4. **Stage 2: Multi-task Prompt Transfer** ;
5. **for each iterative k \( \leftarrow 1 \) to \( N_{max} \) do**
   6. Source Task Grouping: Group a subset of relevant source tasks \( S' \) from \( S \);
   7. Multi-Task Merging: Merge task prompt vectors from \( S' \cup \{t\} \) to get \( P_{mix} \);
   8. Boost the posterior probability on target task t with \( P_{mix} \)
9. **return** \( P_{mix}^* \)
Algorithm 2: Source Task Grouping

Input: source tasks set \( S = \{s^1, s^2, \ldots, s^n\} \), target task \( t \)
Output: selected task group \( S' \)

Step 1: Rank Similarity to Target ;
Compute Similarity Ranking list \( \Pi \);

Step 2: Maximize Knowledge Consistency ;
Initialize an empty source task group \( S' \leftarrow \emptyset \);
for each index \( \pi^i \) from similarity rank list \( \Pi \) do
    Let \( s^{\pi^i} \) be the task corresponding to index \( \pi^i \);
    Calculate the contribution of \( s^{\pi^i} \) to \( S' : \Delta(S', s^{\pi^i}) \leftarrow KC(S' \cup \{s^{\pi^i}\}) - KC(S') \);
    if \( sim(t, s^{\pi^i}) \geq 0 \) and \( \Delta(S', s^{\pi^i}) \geq 0 \) then
        Add \( s^{\pi^i} \) to \( S' : S' \leftarrow S' \cup \{s^{\pi^i}\} \);
return \( S' \)

![Task similarity dendrogram and heatmap showing pairwise dot product results between 5 source tasks and 6 target tasks](page_347_682_495_312.png)

Figure 10: Task similarity of dot product result via TPV. We visualize the task similarity between 5 source tasks and 6 target tasks. We apply min-max normalization to reflect the relative relation among tasks.

ships between different tasks. Figure 10 shows the cluster map by computing pairwise task similarity score based on TPV (Eqn. 2). We observe that tasks perceived as similar are clustered together. Specifically, in the source tasks partition, SQuAD and ReCoRD are grouped in the QA cluster. QNLI and QQP belong to QA datasets. This clustering pattern is also observed in the target tasks partition. NQ and HQ are in the QA cluster, MRPC PAWS are Paraphrase Detection, and WSC and WG are Common Sense Reasoning. Furthermore, all target tasks show a consistently high relative task similarity with MNLI, a widely used intermediate task for fine-tuning PLMs (Phang et al., 2018). This highlights the TPV’s ability to capture less obvious positive transfer. More details can be found in Figure 11.

J Computation and Time Costs

Dynamically calculating the task combinations during each parameter update does indeed introduce additional time and computation costs during training. However, this computation does not involve gradients, so it ultimately does not lead to a significant increase in time and computation burden. We visualize the result and training speed on RTE in Table 9. We observe that DTVG demonstrates a 16.5% improvement in performance while incurring only a 8.7% decrease in training speed compared to prompt tuning.
Figure 11: Task similarity visualizations of task prompt vectors. We conduct four experiments with different seeds in {22, 32, 42, 52}. We apply Min-Max normalization to ensure the relative relationships in the results are maintained.

<table>
  <tr>
    <th>Method</th>
    <th>Test Acc on RTE</th>
    <th>Traning samples per second</th>
  </tr>
  <tr>
    <td>PT</td>
    <td>74.1</td>
    <td>64.2</td>
  </tr>
  <tr>
    <td>DTVG</td>
    <td>86.3</td>
    <td>58.6</td>
  </tr>
</table>

Table 9: Test result and training speed on RTE. We use T5-base as backbone

Reviewer paper summaries: In their work, the authors introduce a novel soft-prompt-based PEFT method called Dynamic Task Vector Grouping (DTGV) and new metrics called Target Similarity and Knowledge Consistency Score to determine the positive transfer of added tasks. Authors motivate the introduction of new metrics with their study on how adding new tasks includes negative transfer and that cosine similarity between source and target soft prompts changes over time. Moreover, the authors show that cosine similarity between the source and target soft prompt does not always positively correlate with positive and negative transfer.

Target Similarity and Knowledge Consistency Score are based on Task Similarity, essentially a cosine similarity between vectors created from soft prompts, aggregated into a single by the sum across task vector tokens. Target Similarity is then calculated as a sum of Task Similarity between the source and target tasks. The knowledge Consistency score is calculated as Task Similarity on permutations of source tasks. 

DTGV method first finds a proper subset of tasks to merge via Multi-Task Grouping. Task Similarity between the source and target task is then used to rank the source tasks and to group them based on their rank. The group with the highest Knowledge Consistency is chosen to be merged by arithmetic addition of Task Vectors as described by Ilharco et al., 2023 with different scaling factors for the target task and the source task. Source Task Grouping and Multi-Task Merging steps are repeated during training for each parameter update.

Authors compare the DTGV method with various PEFT methods and achieve state-of-the-art best performance across compared methods.
Summary of strengths: **S1**: The paper deals with an interesting topic of multi-task transfer of soft prompts and introduces a novel method that achieves the best performance across all compared state-of-the-art methods.

**S2**: Authors provide an evaluation of how source task selection inhibits the negative and positive transfer and that including more tasks doesn’t necessarily mean better performance. Moreover, they provide empirical evidence that the cosine similarity of source and target task soft prompt changes with the convergence of target soft prompt.

**S3**:  Authors also include extensive experiments and ablation study of the DTGV method on 6 source tasks and 21 target tasks, where the results are based on averaging across multiple runs with standard deviations provided.
Summary of weaknesses: **W1**: Lack of generalization. Although the DTGV method achieves excellent results on the T5-base model, it is unclear how it would perform on decoder-based autoregressive models or the latest encoder-decoder-based models. The same goes for evaluating natural language generation (NLG) tasks. Consider including more recent models (e.g., LLaMa 3, Mixtral, Zephyr, mt0) and NLG tasks in evaluating the method, at least for some of the experiments, to strengthen the results.

**W2**: Missing intuition for Task Similarity. Both Target Similarity and Knowledge Consistency Scores are based on the cosine similarity of aggregated task vectors (Task Similarity). Still, the intuition of why it should be better than the cosine similarity between soft prompts is missing. A short explanation of why it is better than just flattening the vectors and why it should inform better if two tasks are similar would support the empirical findings. 

**W3**: Presentation and paper structure. The presentation and structure of the method could be improved. See the Comments Suggestions And Typos section. While reading, I staggered across different variables used in the formal definitions, which were not adequately defined, explained, or referenced.
Comments, suggestions and typos: **Q1**: Why do you define Target Similarity but never use it in training due to the NP-hardness of Set Cover problem? I understand that it shows the thought process behind the method, but it would save you space and readers time. I suggest moving the Target Similarity to the appendix and replacing the naming in Figure 3 with just Task Similarity between source and target tasks.

**Q2**: You show experiment results in Table 1 without prior experiment setup. What source task set was used to determine the source task with the highest similarity? I suggest moving this experiment after the Experiment Setup section.

**Q3**: You mentioned in the experiment setup that you compared different baselines in different sources to target task ratios (1 to 1, 1 to many, many to many). In the table, you only differentiate between multi-task training and single-task training. Would it be possible to include and highlight this (using different colors)?

**C1**: You named the task vectors created from soft prompts Soft Prompt Task Vectors, which is a valid and logical naming. However, the prior work of Belanec et al., 2024  already used Task Prompt Vectors naming. I suggest replacing Soft Prompt Task Vectors (SPTV) with Task Prompt Vectors (TPV) to avoid future misunderstandings and inconsistencies.

**C2**: You define task similarity as a function of s1 and s2, but there is no formal mapping of v1 and v2 that would tell us that v1 and v2 are per token vectors of task vector T1 and T2 (and this also confirms the lack of Task Similarity intuition in W2). I suggest using straight task vectors as input to the function because you defined it earlier in line 208.

**C3**: I suggest including an anonymized repository or providing compressed source code directly in the submission to increase reproducibility.

**C4**: You define the rule that SS > 0 at line 314 but do not define SS. The same goes for parameter lambda in Equation 5.

**C5**: Line 87, *we* should start with capital letter.

Generally, I like the idea of DTGV, and the quality of the paper is above average. If you meaningfully address some of the mentioned weaknesses, I am willing to increase the overall score from 3 to 3.5 and soundness from 3.5 to 4.


**Comments after Author Discussion**

I would like to thank authors for providing a rebuttal reply. Unfortunately, the response was provided literally minutes before the end of the rebuttal period so no additional discussion was possible. Nevertheless, after considering the provided replies, I decided to increase the overall score to 3.5.
Overall assessment scores: 3.5