Connectivity Patterns are Task Embeddings
Year: 2023

Abstract:
Task embeddings are task-specific vectors designed to construct a semantic space of tasks, which can be used to predict the most transferable source task for a given target task via the similarity between task embeddings. However, existing methods use optimized parameters and representations as task embeddings, resulting in substantial computational complexity and storage requirements. In this work, we draw inspiration from the operating mechanism of deep neural networks (DNNs) and biological brains, where neuronal activations are sparse and taskspecific, and we use the connectivity patterns of neurons as a unique identifier associated with the task. The proposed method learns to assign importance masks for sub-structures of DNNs, and accordingly indicate the task-specific connectivity patterns. In addition to the storage advantages brought by the binary masking mechanism and structured sparsity, the early-bird nature of the sparse optimization process can deliver an efficient computation advantage. Experiments show that our method consistently outperforms other baselines in predicting intertask transferability across data regimes and transfer settings, while keeping high efficiency in computation and storage.

heading: 1 Introduction
text: With the rapid development and excellent performance of large pre-trained language models (PLMs), the most prevalent paradigm in natural language processing (NLP) has become pretraining then fine-tuning (Peters et al., 2018; Devlin et al., 2019a; Brown et al., 2020; Lewis et al., 2020; Raffel et al., 2020). Extending upon the two-step training procedure, previous works show that intermediate-task transfer, i.e., fine-tuning the model on an intermediate source task before the target task, can yield further gains (Phang et al.,
∗Equal contribution. †Corresponding author.
2018; Wang et al., 2019a). Nevertheless, the improvement by intermediate-task transfer heavily relies on the selection of a proper intermediate task because some source tasks lead to performance degradation (Yogatama et al., 2019; Pruksachatkun et al., 2020). One straightforward approach is to enumerate every possible (source, target) task combination, but it is extremely expensive. Therefore, recent works explore methods to predict inter-task transferability accurately with high efficiency.
The current state-of-the-art (SOTA) works are established on task embeddings, (i.e., leveraging a single vector to represent a task). They predict inter-task transferability by computing the similarity between task embeddings. Task2Vec (Achille et al., 2019; Vu et al., 2020) develops task embeddings based on the Fisher information matrix while requiring fine-tuning the full model and consuming a large amount of storage (Zhou et al., 2022). Recently, researchers propose that the efficiently tuned parameters like prompts (Li and Liang, 2021; Liu et al., 2021) and LoRA (Hu et al., 2022) encode rich information for a task and thus can serve as task embeddings (Poth et al., 2021; Vu et al., 2022;
11993
Zhou et al., 2022). However, these tuned parameters are sensitive to model initialization and stochasticity (Li and Liang, 2021; Lester et al., 2021), and optimizing these parameters consumes significantly more computational resources than traditional finetuning (Ding et al., 2022).
Different from them, we draw inspiration from the shared working mechanisms of DNNs and biological brains to develop high-quality task embeddings. We start by considering which parts of knowledge within the model are being utilized for a given task. Typically, recent works in sparse optimization and model pruning have shown that sub-structures (e.g., neurons, attention heads, channels, and layers) from different parts of the model exhibit specialization in distinct knowledge and possess varying degrees of importance for a particular task (Dalvi et al., 2020; Liu et al., 2017; Voita et al., 2019a; Glorot et al., 2011; Georgiadis, 2019; Li et al., 2022). These are consistent with the findings in neuroscience that activities of neurons and connectivities in biological brains are sparse (Kerr et al., 2005; Poo and Isaacson, 2009; Barth and Poulet, 2012) and task-specific (Duncan, 2010; Fox et al., 2005; Crinion et al., 2003; Newton et al., 2007). The aforementioned remarkable findings motivate us to use task-specific connectivity patterns in DNNs to represent tasks.
In this work, we propose a novel task embedding, namely Connectivity Patterns as Task Embedding (COPATE), and apply it to predict the inter-task transferability, as illustrated in Figure 1. Our key insight is that in over-parameterized DNNs, there exist connectivity patterns (i.e., the structures of subnetworks) that are functional for one certain task, and can capture high-density task-specific information. Concretely, we assign importance masks to attention heads and intermediate neurons of PLMs, jointly train the masks and the model, and extract task embeddings according to the learned masks. Our method has two strengths in efficiency: 1) it is computation-friendly as we extract connectivity patterns early in the training; 2) it is storage-friendly because our embedding granularity is coarse-grained, and COPATE can be represented by a binary mask. Experiments show that compared to other approaches, COPATE has superior inter-task prediction capability across data regimes and transfer settings. Our codes are available at Github1.
1https://github.com/WooooDyy/CoPaTE
Our contributions can be summarized as follows:
• Inspired by the working mechanisms of DNNs and biological brains, we propose COPATE, a novel task embedding that represents tasks with sparse connectivity patterns.
• We propose a method to obtain COPATE with sparse optimizing techniques, and show the significant positive correlation between embedding similarity and task transferability.
• We conduct thorough experiments on 342 transfer combinations with different settings to show the effectiveness of our method. We further explore an intermediate-curriculum transfer setting to investigate whether there is a beneficial curriculum for a target task.

heading: 2 Identifying Sparse, Task-specific Connectivity Patterns
text: In this section, we demonstrate the framework to identify task-specific connectivity patterns. We represent the task-specific connectivity patterns via the structure of essential subnetworks found by sparse optimizing and pruning techniques (Liu et al., 2017; Chen et al., 2021a; Zheng et al., 2022), including the searching stage (Sec 2.1) and the extracting stage (Sec 2.2).

heading: 2.1 Finding Connectivity Patterns
text: Typically, BERT is constructed by multiple transformer encoder layers that have uniform structure (Vaswani et al., 2017). Each layer has a multi-head self-attention (MHA) block, a feed-forward network (FFN), and residual connections around each block. The MHA is formulated as:
MHA(x) =
Nh∑
i=1
AttW iK ,W i Q,W i V ,W i O (x), (1)
where x is input, Nh is the number of heads, and the projections W iK , W i Q, W i V ∈Rdh×d, W iO ∈ Rd×dh denote the key, query, value and output matrices in the i-th attention head. Here d is the hidden size (e.g., 768), and dh = d/Nh denotes the output dimension of each head (e.g., 64).
An FFN parameterized by WU ∈ Rd×df and WD ∈ Rdf×d comes next:
FFN(x) = gelu(XWU ) ·WD, (2)
where df = 4d.
Learnable Importance Masks We adopt a coarse-grained structured pruning strategy to shape connectivity patterns. Specifically, we use the modified network slimming (Liu et al., 2017; Chen et al., 2021a) to find which heads and intermediate neurons are essential for a given task. We first assign learnable importance masks to each head and intermediate neuron:
MHA(x) =
Nh∑
i=1
miH ·AttW iK ,W iQ,W iV ,W iO(x),
(3)
FFN(x) = mF · gelu(XWU ) ·WD, (4)
where mH denotes the masks for heads, i is the index of head, and mF denotes the masks for FFN. Then, we can jointly train BERT with importance masks but with a sparsity-inducing regularizer:
R(m) = λH∥mH∥1 + λF∥mF∥1, (5)
where m = {mH,mF}, λH and λF denote regularization strength for the two kinds of masks respectively. Hence, the final optimizing objective is:
min θ,m
L(θ,m) +R(m), (6)
where L is the original loss function of fine-tuning.

heading: 2.2 Extracting Connectivity Patterns
text: Early-stopping Strategy Note that the joint training is still as expensive as traditional finetuning. Fortunately, (You et al., 2020) and (Chen et al., 2021b) point out that the importance masks converge early in the searching stage. This inspires us to stop the joint training early and dig out earlybird connectivity patterns to generate task embeddings. Nevertheless, it is difficult to determine the exact search termination time as the termination moments of different tasks are different. Moreover, masks of MHA and FFN typically have different convergence rates. Hence, we adopt a termination metric following (Xi et al., 2022) which terminates the searching process when the normalized mask distances between several consecutive miniepochs are all smaller than a threshold γ2.
Pruning Strategy After the joint training, we can perform pruning to the original models to extract important connectivity patterns that encode taskspecific information. Specifically, the self-attention heads and intermediate neurons with the smallest importance masks are believed to contribute the least to the task and the corresponding masks are set to 0, while the masks of the surviving elements
2see Appendix C for more details of the termination metric.
Algorithm 1: COPATE Generation Input: model parameters θ, learnable
importance masks m, learning rate η, sparsity for self-attention heads pH, and sparsity for intermediate neurons pF.
1 Procedure TASK-SPECIFIC CONNECTIVITY PATTERNS SEARCHING 2 Initialize θ to pre-trained weights; 3 Initialize m = {mH,mF} to 1; 4 repeat 5 θ = θ − η∇θ(L(θ, c) +R(c)); 6 m = m− η∇m(L(θ,m) +R(m)); 7 until the convergence condition in Sec.2.2 is
satisfied, or the fine-tuning is done; 8 Procedure GENERATING COPATE WITH
LEARNED MASKS 9 Reset mH and mF to binary form with pH
and pF according to mask magnitudes, respectively;
10 Emb = [mH;mF].
are set to 1. Therefore, we can generate storageefficient task embeddings with the resulting model structure.

heading: 3 COPATE: Connectivity Patterns as Task Embedding
text: In this section, we first show how we generate task embeddings with task-specific connectivity patterns at hand (Sec 3.1). Next we provide empirical evidence for the appropriateness of using the obtained task embeddings to predict inter-task transferability in Sec. 3.2.

heading: 3.1 Task Embedding Generating
text: Typically, the structure of a neural network can be represented as a mask vector:
m = [m1,m2, ...,mN ], mi ∈ {0, 1}, (7)
where N denotes the number of elements (i.e., substructures) that construct the network and the value of mask mi indicates whether the i-th element is pruned or not. In our framework, the elements are self-attention heads and intermediate neurons, so the structured subnetworks are represented by:
mH = [m 0 H,m 1 H, ...,m NL×Nh H ], (8)
mF = [m 0 F,m 1 F, ...,m NL×Nf F ], (9)
where NL denotes the number of transformer layers, Nh denotes the number of heads in each layer and Nf is the number of intermediate neurons in each layer. Hence, the resulting task embedding is:
Emb = [mH;mF]. (10)
We summarize the procedure of generating COPATE in Algorithm 1. COPATE is quite storageefficient owing to its binary form. For example, BERTBASE consumes only 4626 bytes to store3.

heading: 3.2 Positive Correlation between COPATE Similarity and Task Transferability
text: We first calculate the similarity between COPATEs of different tasks with Hamming Similarity, which is defined as the number of positions at which the corresponding symbols are the same:
Sim(V1, V2) =
∑n i=1 σ(V1[i], V2[i])
n , (11)
where σ(v1, v2) = 1 if v1 = v2 else 0. Since the numbers of self-attention heads and intermediate neurons differ significantly, we calculate the similarity of the two types of elements separately, and each contributes equally to the final similarity.
We then explore whether the similarity between COPATEs is correlated with task transferability. We calculate related transfer gain to measure the impact of transfer learning. Specifically, given a source task s and a target task t, if a baseline PLM that is directly fine-tuned on the target dataset (without any intermediate transferring) achieves a performance of T (t), while a transferred model achieves a performance of T (s, t), the relative transfer gain can be expressed as: G(s, t) = T (s, t)− T (t)
T (t) .
Figure 2 shows how the relative transfer gain changes as a function of the similarity between the source and target task embeddings. Overall, there is a significant positive correlation between the similarity of task embeddings and task transferability on the majority of the target tasks (16 out of 19). It is possible for the correlation coefficient to attain a high magnitude in many cases, such as on the DROP task, where the correlation coefficient is 0.78 (p = 0.00013).
The exciting results suggest that COPATE is promising in accurately predicting inter-task transferability. Concretely, for a novel target task, we
3BERTBASE has (12 × 12) heads and (3072 × 12) intermediate neurons, and requires 37008 bits = 4626 bytes to store.
rank the candidate source tasks in descending order by the COPATE similarity and select the top-ranked task for intermediate fine-tuning.

heading: 4 Predicting Task Transferability
text: In this section, we perform thorough experiments to empirically demonstrate the capability of COPATE in predicting inter-task transferability.

heading: 4.1 Experimental Setup
text: Datasets We conduct experiments with 8 tasks of text classification or regression (CR) and 11 tasks of question answering (QA) following previous works (Vu et al., 2020; Zhou et al., 2022). We list the datasets in Appendix A.
Data Regimes For every (source, target) dataset pair, we perform transfer experiments in three data regimes to simulate real-world situations: FULL → FULL , FULL → LIMITED , and LIMITED → LIMITED. The FULL regime includes all training data, while in LIMITED settings, we limit the amount of training data by randomly selecting 1K training examples.
Baselines We compare our method with following strong baselines: (1) TEXTEMB (Vu et al., 2020) averages sentence representations by BERT over the whole dataset. (2) TASKEMB (Achille et al., 2019; Vu et al., 2020) embeds tasks based on the Fisher information matrix which captures the curvature of the loss surface. (3) PTUNING (Vu et al., 2022) interprets the fine-tuned soft prompts in each transformer layer as task embeddings. (4) LORA (Zhou et al., 2022) injects trainable rank decomposition matrics into layers of the model and takes the fine-tuned matrics as task embeddings.
Evaluation Metrics We use the following metrics to evaluate the performance of methods: (1) Normalized Discounted Cumulative Gain (NDCG) (Järvelin and Kekäläinen, 2002) is a broadly used information retrieval metric aiming to evaluate the quality of a ranking with attached relevances, and it penalizes top-ranked and bottomranked mismatches with different weight4. (2) Regret@k (Renggli et al., 2022) measures the relative performance difference between the top k selected
4See Appendix D for more details about NDCG.
source tasks and the optimal source task5. In our experiments, we include k = 1 and k = 3.
Implementation Details We perform transfer experiments with all (source, target) combinations and use BERTBASE (Devlin et al., 2019b) as the backbone. All the intermediate tuning and target tuning take 3 epochs. For FULL → FULL regime, we use the results from (Vu et al., 2020). We implement all baseline methods according to their opensource codes and the Transformers library (Wolf et al., 2020). When searching for connectivity patterns in our method, we jointly train the masks and the BERT model for 5 epochs. When extracting early-bird embeddings (i.e., EARLY-EMB), we set the max searching epoch number to 1. We perform 5 restarts for stable results in LIMITED regimes. See Appendix F for more details.

heading: 4.2 Experimental Results
text: Table 1 demonstrates the detailed evaluating results. Overall, the proposed COPATE achieves superior performance across task types, transfer scenarios and data regimes, revealing that it is a robust and accurate predictor of beneficial transfer.
FULL → FULL In this regime, our method attains impressive performance compared to other baselines. For example, in the setting of in-class transfer of Classification tasks, COPATE exceeds the most competitive baseline by 1.0 in NDCG, and the Regret@3 score achieves 1.2. It is also observed that excessive training steps for identifying task-specific connectivity patterns do not necessarily result in large performance improvement in this regime. The efficient EARLY-EMB performs slightly worse than LTH EP=5, but still performs comparably.
5See Appendix E for more results about Regret@k
FULL → LIMITED In this few-shot regime, our method achieves comparable performance to SOTA baselines. However, we find that in QA tasks, the performance of COPATE degrades sharply as the number of training steps utilized during the search stage decreases. Compared to LTH EP=5, EARLYEMB’s NDCG on in-class and all-class decreased by 10.1 and 8.9, respectively. This trend is also observable in LIMITED → LIMITED regime. It is not surprising as QA tasks are typically more complex and the connectivity patterns require more training steps to converge better. This suggests a trade-off between performance and efficiency when facing limited examples, and additional training resources should be allocated to the search stage to extract high-quality task embeddings.
LIMITED → LIMITED In this regime, COPATE demonstrates exceptional performance and surpasses other existing baselines by a significant margin. For instance, our method outperforms the strongest baseline by 9.5 in terms of NDCG on in-class transfer of QA tasks, and 4.6 on all-class transfer of QA tasks.

heading: 5 Discussion

heading: 5.1 Ablation Study
text: In this section, we perform ablation studies to show the contribution of each component of our method.
Head v.s. FFN Previous experiments utilize both masks of attention heads and intermediate neurons to compute similarity. Here, the contribution of each component is evaluated individually by separately using them to calculate similarity and subsequently assessing the NDCG. Table 2 shows that both components play essential roles in ranking source tasks. We observe that on CR tasks, heads outperform FFN by a large margin, revealing that heads are more important in such tasks.
Impact of Sparsity Figure 3 illustrates the relationship between the level of sparsity and the performance of the obtained embeddings. The performance of the model is significantly impacted by variations in the pruning ratio of heads or FFN when the target tasks are CR, while such variations have a limited effect when the target tasks are QA, revealing that CR tasks are more sensitive to embedding sparsity. After comprehensive consideration, we believe that 1/3 and 0.4 are reasonable sparsity for heads and FFN, respectively.
We include more ablation studies of pruning strategies, early-stopping thresholds, and the sparsity-inducing regularizer in Appendix I.

heading: 5.2 Computation and Storage Consumption
text: Table 3 lists the computational and storage cost of each method. COPATE demonstrates efficiency in both aspects thanks to proper designs ( i.e., earlystopping, structured pruning and binary form of embeddings), particularly EARLY-EMB, which exhibits the fastest generation speed and only requires 4.6K bytes to store. TASKEMB is also computation-efficient, but it requires much more storage than COPATE. While TEXTEMB is the only method that is comparable to our approach in terms of efficiency, it falls behind EARLY-EMB with an average difference of 1.6 in NDCG.
Further Storage-efficiency with Task-specific Layers Previous studies have established that in BERT, layers are redundant (Dalvi et al., 2020), and that shallower transformer layers contain more general information while deeper layers contain more task-specific information (Voita et al., 2019a; Kim et al., 2020; Sajjad et al., 2020). These in-
sights shed light on further reducing the storage of COPATE by representing tasks using a select number of layers, or even a single layer. Figure 4 illustrates the evaluated performance. We observe that: (1) Using a select number of layers does not result in a significant decrease in performance, and sometimes delivers better performance. (2) Top-down strategy outperforms bottom-up strategy, and consistently exceeds the full model in few-shot settings, showing that deep layers can effectively encode task-specific information, which is in line with previous studies. As a result, if we adopt the last six layers for embedding generation, 50% of the storage can be saved, while little decrease in performance is incurred. We also explore the potential of generating embeddings using a single layer, while sacrificing little performance in Appendix J.

heading: 5.3 COPATE Captures Task Relationships
text: The heatmap in Figure 5 illustrates the hierarchical clustering of the similarities between COPATEs. The results indicate that the obtained embeddings effectively capture various intuitive task relationships. We observe that tasks with similar characteristics congregate in clusters, such as QA tasks (WikiHop, SQuAD-1, SQuAD-2, DuoRC-s, DuoRCp, NewsQA, and HotpotQA), similarity and paraphrasing tasks (STS-B and MRPC), NLI tasks (QNLI and MNLI), and single sentence classification tasks (SST-2 and CoLA). In particular, a closer examination of the clustering reveals that SQuAD1 and SQuAD-2 are closely grouped together, with the latter being an extension of the former (Rajpurkar et al., 2016, 2018). Furthermore, the tight clustering of DuoRC-p and DuoRC-s is also noteworthy, as they are variations of the same movie plots with different lengths (Saha et al., 2018).

heading: 5.4 Intermediate-curriculum Transfer
text: Here, we extend the boundary of intermediate-task transfer and examine the potential benefits of a specific intermediate task curriculum (i.e., a particular order to arrange several tasks) to a target task using COPATE. Three distinct curriculum strategies are considered: (1) Similar-first strategy which selects the three tasks that are most similar to the
target task and arranges the intermediate tasks in a sequential order of similarity. (2) Different-first strategy which also selects the three tasks that are most similar to the target task, but arranges the intermediate tasks in an order of dissimilarity. (3) Recursive-similar strategy which starts from the target task, recursively finds the task that is most similar to the current task three times, stacks them, and then sequentially pops these found tasks for intermediate fine-tuning. The results in Table 4 show that: (1) Each curriculum can boost the target task, validating the value of intermediate-task transfer. (2) The recursive-similar strategy yields the most performance gain, suggesting that making each intermediate task learned better can deliver more benefits to target tasks. (3) The different-first strategy performs better than the similar-first, implying that intermediate tasks that are similar to the
target task should be assigned later.

heading: 6 Related Work
text: Predicting Beneficial Intermediate Tasks It has been shown that intermediate-task transfer can deliver performance gains for many target tasks (Phang et al., 2018; Wang et al., 2019a; Talmor and Berant, 2019; Liu et al., 2019), but improper intermediate tasks can result in negative transfer results (Yogatama et al., 2019; Pruksachatkun et al., 2020). Hence, researchers try to accurately identify the most beneficial source task based on metadata or extracted representations of tasks (Alonso and Plank, 2017; Vu et al., 2020; Poth et al., 2021). Recent works represent tasks with embeddings that are generated from data representations (Vu et al., 2020), model weight information (Achille et al., 2019; Vu et al., 2020), and efficiently tuned parameters (Poth et al., 2021; Vu et al., 2022; Zhou et al., 2022). Different from them, we start from a model architecture perspective and use connectivity patterns to represent tasks.
Techniques to Obtain Sparse Subnetworks Researchers have explored a variety of techniques to obtain sparse networks by removing sub-structures like weights (Louizos et al., 2018; Frankle and Carbin, 2019; Sanh et al., 2020; Xu et al., 2021), channels (He et al., 2017; Luo et al., 2017; Liu et al., 2017; Molchanov et al., 2019), attention heads (Voita et al., 2019b; Michel et al., 2019; Li et al., 2021) and layers (Fan et al., 2020; Sajjad et al., 2020). These approaches first identify unimportant sub-structures and subsequently remove them. With the increasing size of PLMs, sparse subnetworks have become increasingly important for efficient deployment and inference in NLP, lead-
ing to a proliferation of related research (Prasanna et al., 2020; Hou et al., 2020; Lagunas et al., 2021; Xia et al., 2022). Our proposed method, which uses connectivity patterns as task embeddings, is orthogonal to these existing techniques.

heading: 7 Conclusion
text: In this work, we propose COPATE, a novel task embedding that represents tasks with sparse connectivity patterns, and develop a method to get such embeddings. Comprehensive experiments show that the proposed method outperforms other competitive approaches in predicting inter-task transferability while achieving efficiency in both computation and storage. We hope that our work may motivate future work in introducing connectivity patterns as task embeddings to fields like meta learning, multi-task learning, and model interpretability.
Limitations
While the proposed method has demonstrated superior performance and high efficiency, there are several limitations that warrant further investigation: (1) In few-shot settings where the number of training examples is limited, the performance of our method and other baselines drops significantly. Future work should focus on uncovering essential features of the task in few-shot scenarios and generating embeddings of higher quality. (2) The storage consumption has been reduced to a small amount, however, the number of neurons is still relatively large compared to that of heads and therefore becomes a bottleneck for further decreasing storage requirements. As discussed in Sec 5.2, one possible solution is reducing the number of layers used to generate the embedding. Future work could also include assigning intermediate neurons into groups to make the embedding coarser in granularity, thus reducing storage requirements.

heading: Acknowledgements
text: The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62076069,62206057,61976056), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR1403500).

heading: A List of Datasets
text: See Table 5 for details of datasets.

heading: B More Results of Correlation between COPATE Similarity and Inter-task Transferability
text: See Figure 6 for more results of correlation between COPATE similarity and inter-task transferability. There is a significant positive correlation between the similarity of task embeddings and task transferability on most target tasks.

heading: C More Details of Early-stopping Strategy
text: We use Hamming distance to calculate the normalized normalized mask distance. We stop the searching stage when the normalized mask distances between consecutive 5 miniepochs are all smaller than γ. Each miniepoch consists of 0.05 epochs. We set γ to 0.05 in all settings. This is not the best choice for all transfer scenarios, but we unify the
value of hyper-parameters for the sake of generality.

heading: D More Details of NDCG
text: The NDCG is defined using the Discounted Cumulative Gain (DCG), which is a measure of the relevance score for a list of items, each discounted by its position in the ranking. The DCG of a ranking R at a particular rank position p can be calculated as:
DCGp(R) = p∑
i=1
2reli − 1 log2(i+ 1)
In our experiments, R refers to a ranking of source tasks where the relevance reli of the source task with rank i is set to the averaged target performance, i.e. reli ∈ [0, 100]. We set p = |S|, which is the number of intermediate tasks.
The NDCG finally normalizes the DCG of the ranking predicted by the task selection approach (Rpred) by the golden ranking produced by the empirical transfer results (Rtrue). An NDCG of 100% indicates the best ranking.
NDCGp(R) = DCGp(Rpred) DCGp(Rtrue)

heading: E More Details of Regret@k
text: Regret@k is defined as:
Regretk =
O(S,t) ︷ ︸︸ ︷ max s∈S E[T (s, t)]− Mk(S,t)︷ ︸︸ ︷ max ŝ∈Sk E[T (ŝ, t)]
O(S, t) × 100%
where T (s, t) means the performance on target task t when transferring from source task s. O(S, t) is the expected target task performance of the optimal selection. Mk(S, t) denotes the highest performance on t among the k top-ranked source tasks of the evaluated selection method. In our experiments, we include k = 1 and k = 3.

heading: F More Implementation Details
text: For classification/regression tasks, we set the max sequence length to 128. For question answering tasks, we set the max sequence length to 384. The batch size for all experiments is set to 32. Our experiments are performed on twelve NVIDIA GeForce RTX 3090 GPUs. We perform 3 restarts for our experiments and report the mean. For
PTUNING, we adopt P-Tuning v2 in (Liu et al., 2021), which implements a prompt tuning method by introducing additional attention prefix matrices to each transformer layer. We set the prefix length to 20. For LORA, we set the r to 8 and α to 8. For the searching stage of winning tickets, we set the regularization strength λH and λF to 1e− 4.

heading: G More Results of Head v.s. FFN
text: Table 6 and Table 7 show the results of Head v.s. FFN in FULL → LIMITED and LIMITED → LIMITED , respectively. We can still find that both of them are important for high-quality task embeddings.

heading: H More Results of Impact of Sparsity
text: Figure 7 and Figure 8 show the results of impact of sparsity in FULL → LIMITED and LIMITED → LIMITED , respectively. We can still find that 1/3 and 0.4 are reasonable sparsity for heads and FFN, respectively.

heading: I More Ablation Studies
text: I.1 Impact of Pruning Strategies
In this section, we investigate the impact of different pruning strategies to the embedding performance. Results in Table 8, Table 9 and Table 10 show that layerwise pruning and global pruning are proper strategies for self-attention heads and FFN, respectively.
I.2 Impact of Different Early-Stopping Thresholds
In this section, we investigate the impact of different values of the early-stopping threshold γ. Results in Figure 9 show that the performance of COPATE converges when γ reduces to near 0.05.
I.3 Importance of Sparsity-inducing Regularizer
In this section, we investigate the importance of the sparsity-inducing regularizer during the connectivity pattern searching stage. Results in Table 11 show that the regularizer is indispensable for
generating high-quality task embeddings.

heading: J Further Storage-efficiency with Single Layer
text: In this study, we examine the performance of COPATE when utilizing a single layer to generate task embeddings. The results, as illustrated in Figure 10, demonstrate the performance of each layer. The findings indicate that a single layer can yield performance comparable to that of the full model. Specifically, when the fifth layer is used to generate the embedding, there is a significant reduction of 91.7% in the storage space required for the embedding, while the final NDCG score is only slightly lower, at 0.67 on average, as compared to the full model.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The Limitations section is after the conclusion part.
7 A2. Did you discuss any potential risks of your work? We think there is no ethical statements need to be included.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: The abstract is at the beginning of the article and the introduction is Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5.1 (Experimental Settups), Appendix A

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.1 (Experimental Setup), Appendix A
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? They are all open-source artifacts that are publicly available.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? They are all open-source artifacts that are publicly available.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? They are all open-source artifacts that are publicly available, and do not contain this kind of private information.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.1 (Experimental Setup), Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1 (Experimental Setup), Appendix A
C 3 Did you run computational experiments? Section 4 (Predicting Task Transferability), Section 5 (Discussion)

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.1 (Experimental Setup), Section 5.2 (Computation and Storage Consumption) and Appendix F
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Yes, Section 4.1 (Experimental Setup) and Appendix F

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 (Predicting Task Transferability), Section 5 (Discussion) and Appendix F

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Yes, Section 4.1 (Experimental Setup) and Appendix F
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope that our work may motivate future work in introducing connectivity patterns as task embeddings to fields like meta learning, multi-task learning, and model interpretability. Limitations
While the proposed method has demonstrated superior performance and high efficiency, there are several limitations that warrant further investigation: (1) In few-shot settings where the number of training examples is limited, the performance of our method and other baselines drops significantly. Future work should focus on uncovering essential features of the task in few-shot scenarios and generating embeddings of higher quality. (2) The storage consumption has been reduced to a small amount, however, the number of neurons is still relatively large compared to that of heads and therefore becomes a bottleneck for further decreasing storage requirements. As discussed in Sec 5. 2, one possible solution is reducing the number of layers used to generate the embedding. Future work could also include assigning intermediate neurons into groups to make the embedding coarser in granularity, thus reducing storage requirements