One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning
Year: 2023

Abstract:
Fine-tuning pre-trained language models for multiple tasks tends to be expensive in terms of storage. To mitigate this, parameter-efficient transfer learning (PETL) methods have been proposed to address this issue, but they still require a significant number of parameters and storage when being applied to broader ranges of tasks. To achieve even greater storage reduction, we propose PROPETL, a novel method that enables efficient sharing of a single PETL module which we call prototype network (e.g., adapter, LoRA, and prefix-tuning) across layers and tasks. We then learn binary masks to select different sub-networks from the shared prototype network and apply them as PETL modules into different layers. We find that the binary masks can determine crucial information from the network, which is often ignored in previous studies. Our work can also be seen as a type of pruning method, where we find that overparameterization also exists in the seemingly small PETL modules. We evaluate PROPETL on various downstream tasks and show that it can outperform other PETL methods with approximately 10% of the parameter storage required by the latter.1

heading: 1 Introduction
text: With the release and wide application of numerous pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019), pre-training and subsequently fine-tuning them becomes prevalent in natural language processing (NLP). This yields good performance in many downstream tasks. However, such a paradigm requires the entire model to be updated and saved after fine-tuning. As PLMs grow in size, traditional fine-tuning becomes costly in storage, limiting the application to multi-task scenarios. In order to ameliorate this issue, many ParameterEfficient Transfer Learning (PETL) methods have
*The first two authors contributed equally. 1Our code is available at https://github.com/
ChaosCodes/ProPETL.
been proposed (Houlsby et al., 2019; Li and Liang, 2021; Hu et al., 2022). Rather than fine-tuning the entire model, they introduce new parameters and only fine-tune those additional parameters on downstream tasks, which drastically reduces the storage of the parameters required by each task. However, they still require a significant number of parameters when more tasks are considered.
In this paper, we continue this line of research and target using even less storage per task. We observe that recent advancements in the field of PETL focus on finding better ways to apply the additional parameters, such as the adapter module after each feed-forward layer (Houlsby et al., 2019; Pfeiffer et al., 2021), or the low-rank matrices in the query and value projection of the self-attention
7564
networks (Hu et al., 2022). However, limited works examine the impact of sub-network structure and integrate pruning methods with PETL methods. In fact, studies in network pruning have shown that the modeling ability of neural networks relies not only on the parameters but also on the sub-network structures that are decided by the pruning masks. For instance, Zhou et al. (2019) discovered that the sub-network of an untrained model can yield great performance without any parameter update. In light of this, we seek to incorporate the structural information of sub-networks into PETL. We believe that when enough structural information is injected into the network, we no longer need that many parameters in PELT modules and further improve the parameter efficiency.
To this end, we propose a novel PETL method dubbed PROPETL that enables efficient sharing of a single prototype adapter, prefix, or LoRA across layers and tasks. When sharing the prototype network, PROPETL learns binary masks to prune different sub-networks in different layers and tasks (Figure 1). The connections of the prototype network pruned in one layer can be used in another with a different pruning mask. In this way, a parameter can be used multiple times across different modules, achieving higher parameter efficiency. Previous methods (He et al., 2022b; Ma et al., 2022) only consider simply discarding (pruning) the useless parameters, while we focus on the structural information in masks by strategically dispatching the parameters in the single prototype network to different modules. We evaluate PROPETL on various downstream tasks, including GLUE (Wang et al., 2018), XSum (Narayan et al., 2018), and WMT16 Ro-En (Bojar et al., 2016). Experiments show that PROPETL achieves better performance than other PETL methods while using significantly fewer parameters.
Our contributions are summarized as follows:
• We propose PROPETL, a highly parameterefficient transfer learning method that injects structural information into PELT and allows for efficient sharing of a single prototype network across layers and tasks.
• Experiments show PROPETL is able to dramatically reduce the storage of the parameters while achieving better performance than conventional PETL methods.
• PROPETL offers an alternative view for net-
work pruning and sharing, where we use binary masks to decide when to discard or share the parameters. We hope to inspire more intriguing explorations in this direction.

heading: 2 Related Work
text: In this section, we briefly survey ideas that are related to our work from three fields: parameterefficient transfer learning, pruning methods, and multi-task learning.

heading: 2.1 Parameter-Efficient Transfer Learning
text: Recently, as the pre-trained language models get larger and larger, some parameter-efficient transfer learning methods that only update a few extra parameters while freezing the PLM backbone have been proposed. Adapter-tuning (Houlsby et al., 2019) fine-tuned adapter modules inserted after each attention and feed-forward layer. Prefixtuning (Li and Liang, 2021) placed an additional trainable prefix to the keys and values matrix in the attention module. LoRA (Hu et al., 2022) injected tunable rank decomposition matrices into each Transformer layer. Based on these parameter-efficient transfer learning methods, He et al. (2022a) gave a unified framework that allows for the transfer of design elements across various PETL approaches. However, when applied to larger PLMs and a broader range of tasks, these methods still require a large storage space because the number of extra parameters is directly proportional to the number of layers and tasks. Inspired by the parameter-sharing techniques of ALBERT (Lan et al., 2020a), we propose sharing the additional parameters in PETL modules across layers and tasks. Our method can thus obtain higher parameter-efficient efficiency with a significantly smaller portion of additional storage than existing PETL methods.

heading: 2.2 Pruning Methods
text: Pruning is one of the most popular methods to reduce unnecessary weights from over-parameterized neural networks while maintaining comparable performance. Recently, Frankle and Carbin (2019) proposed Lottery Ticket Hypothesis and stated that in a randomly initialized dense model, a sparse subnetwork exists that, when trained in isolation, can achieve performance comparable to dense models. Accompanied by this hypothesis, many pruningbefore-training methods have emerged (Lee et al.,
2019; Bai et al., 2022; Sreenivasan et al., 2022). Xu et al. (2021) further propose a method that prunes the backward gradient of the neural network, as opposed to pruning the network parameters themselves. Based on these methods, some works (He et al., 2022b; Ma et al., 2022) also proposed to combine pruning algorithms with parameter-efficient methods to further decrease the additional module size. However, those methods only focus on discarding redundant parameters. A parameter is either discarded or retained without any sharing. They fail to make full use of the additional parameters and cannot achieve highly sparse sub-networks without significantly compromising accuracy.

heading: 2.3 Multi-Task Learning
text: Multi-task learning (Zhang and Yang, 2022), which involves training a single model to perform well on multiple tasks, has gained popularity as a research direction in machine learning. However, this approach can be hindered by catastrophic forgetting (Kirkpatrick et al., 2016) and data imbalance among tasks, which will result in overfitting on lowresource tasks and underfitting on high-resource tasks (Arivazhagan et al., 2019). Houlsby et al. (2019) propose adapter tuning that only introduces and updates small additional parameters for each
task while freezing the pre-trained model. Based on such a parameter-efficient method, Mahabadi et al. (2021) train a hyper-network named Hyperformer, which generates task-specific weights for the adapter modules when fed with different task embeddings.

heading: 3 Methods
text: In this section, we first give an introduction to parameter-efficient transfer learning (PETL). We then present our method PROPETL, depicted in Figure 2, which combines the techniques of parametersharing and pruning to further improve the parameter efficiency compared to existing PETL methods.

heading: 3.1 Preliminaries
text: In parameter-efficient transfer learning, we freeze the parameters θlm of the pre-trained language model and then introduce additional fine-tunable parameters denoted as θt. Given dataset {Xi, Yi}Ni=1, the goal of parameter-efficient finetuning is to maximize the following likelihood of the label Y by only updating the additional parameters θt:
max θt
N∑
i=1
logP (Yi|Xi; θlm, θt) (1)
Such parameter-efficient methods suggest a more effective way to adapt pre-trained language models over downstream tasks than fully finetuning. We give a brief introduction of the three most widely used PETL modules, namely adapter (Houlsby et al., 2019), prefix-tuning (Li and Liang, 2021), and LoRA (Hu et al., 2022) in Appendix A. However, there is still a storage limitation when we handle a large range of tasks using these methods. In this paper, we investigate the potential for further enhancing parameter efficiency in neural network models by reducing storage requirements. While previous PETL methods have primarily focused on decreasing the number of parameters to improve efficiency, our approach posits that employing varying bit lengths (e.g., 1-bit, 8-bit, 32-bit) during storage can lead to significant improvements in parameter efficiency by reducing the overall number of bits used by the parameters. To this end, we use bits to measure the storage, which we call Bit-Level Storage (BLS), to take into account the fact that different parameters may have different bit lengths. Consider a neural model, where each parameter has a specific bit length. Then we divide these parameters into N distinct groups based on their respective bit lengths. Let {ρi}Ni=1 denote the number of parameters within the group i, with corresponding bit lengths {bi}Ni=1. The BLS for these parameters can subsequently be determined as follows:
Bit-Level Storage = N∑
i=1
ρibi (2)

heading: 3.2 Shared Prototype Network
text: Parameter-efficient methods like adapter and prefixtuning tend to introduce an additional module in each Transformer layer. Assuming the Transformer has L layers, we can split the parameters θt into [θt,1, ..., θt,L] according to their layer indexes. Therefore, we can rewrite Equation 2 as:
max θt,1,...,θt,L
N∑
i=0
logP (Yi|Xi; θlm, [θt,1, ..., θt,L])
(3) Inspired by ALBERT (Lan et al., 2020b), in our methods, we first introduce additional parameters for a single PETL module as our prototype network, denoted as θpro. Then, we share the prototype network across different layers. Assuming that the number of parameters in a single PETL module is
Algorithm 1: PROPETL training algorithm Input: Dataset D = (xi, yi), prototype network
learning rate λp, mask learning rate λm, number of layers L, sparsity k% ∈ [0, 1], pretrained parameters θlm
Output: Parameter of the prototype network θpro ∈ Rn, binary mask across layers m ∈ Rn
1 θpro ← randomly initialize in Rn 2 s← randomly initialize in Rn 3 for (xi, yi) in D do /* Apply masks in each layer */ 4 for l in 1, 2, ..., L do 5 ml ← h(sl, k) 6 θsub,l ← θpro ⊙ml 7 θpro ← θpro − λp∇θpro logP (yi|xi; θlm, θsub) 8 s← s− λm∇s logP (yi|xi; θlm, θsub) 9 for l in 1, 2, ..., L do
10 ml ← h(sl, k)

heading: 11 return θpro,m
text: n, we can decrease total parameters from nL to only L, which significantly improves the parameter efficiency. Therefore, we convert the objective function to a more concise one:
max θpro
N∑
i=0
logP (Yi|Xi; θlm, θpro) (4)

heading: 3.3 Masked Sub-Networks
text: Sharing the parameters alone will reduce the model’s capacity to capture meaningful representation in different layers, leading to suboptimal results. Inspired by Zhou et al. (2019) and Ramanujan et al. (2020), we believe that parameters and network structures are both crucial contributing factors to the model’s representative capacity. To this end, we introduce different binary masks ml = {0, 1}n in each Transformer layer l (left part of Figure 2), where n denotes the number of parameters in a single PETL module. Each mask represents a corresponding subnetwork of the shared prototype network. Even though the prototype is shared among all layers, we can use different masks to create different sub-networks for each layer l, whose parameter will be θsub,l = θpro⊙ml, where ⊙ indicates the element-wise product. With this, we can get our final objective function as:
max θpro,m1,m2,...,mL
N∑
i=0
logP (Yi|Xi; θlm, θsub) (5)
where θsub = [θpro ⊙m1, θpro ⊙m2, ..., θpro ⊙ mL].
To learn such masks, we develop our training algorithm based on the edge-popup approach (Ramanujan et al., 2020). Specifically, for each binary mask ml, we introduce floating-point scores sl ∈ Rn. In the forward pass, we generate the binary mask ml by setting the top-k% with the largest absolute value in sl as 1 and the rest as 0. We denote such top-k% thresholding function as h, and ml = h(sl). We refer to the hyperparameter k% as the sparsity ratio in subsequent sections of this paper. During the backpropagation, we use the straight-through gradient estimator (Bengio et al., 2013) to approximate the gradient of the scores, where function h(·) is treated as the identity function. In addition to training the masks, we also jointly optimize the prototype network.
Our approach employs learnable floating-point scores for each binary mask during fine-tuning, leading to nL parameters. When integrated with the prototype network, PROPETL updates a total of nL + n parameters during the training phase, which is marginally more than the conventional PEFT methods which have nL extra parameters. A comprehensive overview of the PROPETL algorithm can be found in Algorithm 1. After training, we discard the floating-point scores and retain only the binary masks (1-bit) together with the shared prototype network (32-bit). Assuming that the 32-bit prototype network requires p bit-level storage and the binary mask of the same dimension demands p/32, our PROPETL can achieve a substantial decrease in storage from around pL to p(1 + L/32). To reconstruct the network structure during inference, we adhere to the following steps: (1) first load the unpruned, shared 32-bit prototype network, (2) then load the binary masks (1-bit) for each layer/task, and (3) extract and use the pruned subnets from the shared prototype network based on specific binary masks during the inference step.

heading: 3.4 Hybrid Masks for Multi-Task Learning
text: Rather than just sharing a PETL module across layers under single-task learning, we can also allow for efficient sharing of the prototype network across multiple tasks. In our approach, we leverage layer masks, as introduced in the previous section, to support parameter sharing within the model. Additionally, we introduce task masks to support parameter sharing across multiple tasks. By per-
forming a logical OR operation2 on these masks, we can obtain a hybrid mask for a specific layer in a specific task, as shown in the right side of Figure 2.
mhybrid = mlayer ∨mtask (6)
With the design of the hybrid mask, given T tasks and N layers in the pre-trained language models, we only require one PROPETL module, N layer masks, and T task masks, further reducing the BLS from the single task scenario (e.g., 0.011% BLS as in Table 2). In addition, layer masks and task masks, which can be combined as hybrid masks, will potentially help infuse the knowledge into the shared prototype network from layers and tasks.

heading: 4 Experimental Setup
text: We briefly summarize the experimental setup in this section. More details can be found in Appendix C.
Datasets We evaluate PROPETL on a wide range of benchmarks, including language understanding (GLUE (Wang et al., 2018)), text summarization (XSum (Narayan et al., 2018)), and machine translation (WMT16 Ro-En (Bojar et al., 2016)).
Backbones We use RoBERTaBASE (Liu et al., 2019) for single-task learning on GLUE. During fine-tuning, we only tune our PROPETL module and the text classification head. For generation and multi-task learning benchmark, we use T5BASE (Raffel et al., 2020) and only tune the PROPETL module. Note that some previous works also tune the layer norms (Houlsby et al., 2019; Mahabadi et al., 2021) while we keep them frozen during fine-tuning.
PETL Modules and PROPETL We use the Pfeiffer adapter (Pfeiffer et al., 2021) as our adapter module and set the bottleneck dimension as 64 by default. For prefix-tuning, we follow Li and Liang (2021) and choose the prefix length to be 64. As for LoRA tuning (Hu et al., 2022), the bottleneck dimension and the scaling factor α are both set to 32. In PROPETL, we increase the value of α to 48 to scale the representation, as the sparse network will decrease the norm of the output representation. We give a brief summary of these PETL modules and how PROPETL is implemented on top of them in Appendix A. Following Ramanujan et al. (2020), we choose the sparsity ratio k% of PROPETL as 0.5, which we will also further discuss in Section 6.
2We provide ablations regarding the choice of mask combining methods in Appendix D.1.
In multi-task learning, we aim to maintain an expected k% around 0.5 for the hybrid mask, so we set the k% to 0.3 for both the layer and task masks.3
Evaluation For text generation, we report ROUGE-2 (Lin, 2004) on the XSUM test set and BLEU (Papineni et al., 2002) score on the Ro-En test set. Since the test sets of GLUE are not released publicly, following Zhang et al. (2021) and Mao et al. (2022), when the sample number of the datasets is fewer than 10k (RTE, MRPC, STS-B, CoLA), we divide the original validation set into halves – the first half for validation and the second for testing. As for the other datasets in GLUE, we randomly choose 1k samples from the training set as our validation data and test on the original validation set. we report both accuracy and F1 for MRPC and QQP in GLUE . For STS-B, we report both Pearson and Spearman correlation coefficients. For CoLA, we report Matthews correlation. For all remaining sub-tasks in GLUE, we report accuracy. Due to high training overhead for generation tasks, we report experimental results with one run for XSum and Ro-En. For GLUE, we report the mean of three different random runs.

heading: 5 Results

heading: 5.1 Single-Task Learning
text: Results in Language Understanding In Table 1, we report the performance of PROPETL and various baselines on the GLUE benchmark. Both PROPETLAdapter and PROPETLLoRA demonstrate superior performance compared to their respective counterparts (adapter and LoRA). Despite having
3Two random and independent binary masks whose elements have 30% probability to be one will produce a resulting mask with about 50% ones after the OR logical operation, which can be calculated using the equation P (A ∪ B) = P (A) + P (B)− P (A)P (B).
slightly more parameters during the fine-tuning process, PROPETL requires only 1/9 (0.11% v.s. 0.95%) of bit-level storage during inference, making them more efficient options. Specifically, PROPETL increases the average score of the adapter by 0.95 and improves the score of LoRA by 0.41. Besides, PROPETLAdapter remarkably outperforms the fully fine-tuned model (86.60 v.s. 86.16) while using 0.11% storage of the fully fine-tuned model. These results indicate that although with reduced parameters, PROPETL injected with the structure information from masks can make more use of the single prototype network and achieve better performance compared to their counterparts. However, we also found that PROPETLPrefix did not outperform prefix-tuning, which we believe is caused by the reparameterization in prefix-tuning that has a harmful effect on the mask learning.4 Overall, PROPETL increases the performance of the adapter to the greatest extent. PROPETLAdapter also achieves the highest performance among the three PROPETL variants. We will stick to PROPETLAdapter for the rest of the experiments.
Results in Language Generation To verify PROPETL can also be applied to harder tasks, we evaluate our method on two language generation datasets, XSum and WMT16 Ro-En. The results are presented in Figure 3 (a) and (b). We find that PROPETLAdapter can perform just as well as the regular adapter method while using significantly less bitlevel storage. Additionally, when consuming more than 1.6% of the storage, PROPETLAdapter is able to achieve competitive performance on the XSum dataset compared with the fully fine-tuned T5. However, both adapter tuning and PROPETLAdapter do not reach the level of the fully fine-tuned model on Ro-En. One potential explanation is Ro-En is
4See more details and explanation in Appendix A.3
harder because translation knowledge may not be covered a lot during the pre-training process of T5. To perform well on such tasks, the model needs to learn additional knowledge and requires more tunable parameters during fine-tuning. We note that such sub-optimal performance on hard generation tasks is not only a nature of PROPETL but generally exists in all PETL methods. Similar findings are also presented in Raffel et al. (2020) and He et al. (2022a). Overall, these experiments show that our PROPETL is also more parameter-efficient on text generation benchmarks compared to existing PETL methods.

heading: 5.2 Multi-Task Learning
text: We present the results of multi-task learning and also provide baseline results from single-task learning using the T5BASE in Table 2. Our bestperforming model, PROPETLAdapter with bottleneck dimension of 64, surpasses the fully fine-tuned T5. We also compare PROPETL with Hyperformer++ (Mahabadi et al., 2021), a hyper-network that is specifically designed to transfer knowledge across tasks. The latter uses significantly more task-specific bit-level storage (26×: 0.011% v.s. 0.29% per task), while only increasing the average score by 0.51. Compared to the vanilla adapter, PROPETLAdapter can be marginally better under a similar fine-tuning parameter budget but with 1/9 of the original storage. Besides, we experiment with an extreme case, where we set the bottleneck dimension to 6. Our results show that the accuracy of adapter tuning decreases from 85.48 to 83.94, while PROPETLAdapter still maintains a comparable performance to the fully fine-tuned model (85.32 v.s. 85.47) with a remarkably small percentage
(0.001%) of bit-level storage per task. This demonstrates that normal adapter tuning can not make full use of the parameters and may fail to perform well with a relatively small bottleneck dimension. However, in the case of PROPETLAdapter, even with a bottleneck dimension that is only 6, it can still achieve a reasonable result. To further validate that PROPETL is effective for larger-sized models, we also carry out experiments on the T5 3B variant and present the findings in Table 3. The outcomes align with our conclusions drawn from the T5 base model. We believe that, in PROPETLAdapter, the structure information learned by the mask can make up for the performance drop due to the shrink of the bottleneck dimension to a certain extent. We further compare adapter tuning with PROPETLAdapter with different percentages of bit-level storage by varying the bottleneck dimension. The results are presented in Figure 3 (c). It shows PROPETLAdapter is able to reach the fine-tuned performance with as few as 0.002% task-specific bit-level storage. We can also see that the curve shares a similar trend with those in Figure 3 (a) and (b), which we will further discuss in the next section.

heading: 6 Discussion

heading: How does PROPETL Scale Differently to
text: Adapter? Figure 3 presents the results when we
adjust the size of the adapter and PROPETLAdapter on three different datasets. Despite the difference in tasks and methods, we discover that adapter and PROPETLAdapter show similar scaling trends on all three datasets when we increase the proportion of the task-specific bit-level storage. Their performance increases linearly with respect to the log scale of the extra storage in the beginning. When the adapter and PROPETLAdapter reach close to the performance of the fully fine-tuned model, their performance gradually converges. Even though PROPETLAdapter and adapter tuning can slightly exceed the fully fine-tuned performance on some datasets, their performance is still bounded to the same level and cannot outperform the fully finetuned model by a large margin. For instance, the performance of both adapter and PROPETL starts to drop when the task-specific storage exceed 0.4% per task on GLUE (Figure 3 (c)). However, PROPETLAdapter is able to reach the fully finetuned level much earlier in the scaling curves. Given a fixed amount of task-specific storage, PROPETLAdapter is also able to achieve better results than the adapter. These results indicate that our share-and-mask method is overall more efficient than the adapter across almost all scales.

heading: Which is More Important, Sharing or Masking?
text: In this section, we discuss the effect of masking and sharing in the prototype network by comparing PROPETL with a random mask baseline and two alternative settings (only mask and only share). For the random mask setting, we randomly select the sub-network of the prototype module during the forward process rather than relying on the updated mask score. Only mask involves not sharing
the module across layers but only learning masks to prune different PETL modules into sparse subnetworks. Only share shares a single PETL module among layers without using masks for pruning. Masking without sharing will drastically increase the number of fine-tunable parameters if we keep the same bottleneck dimension and sparsity ratio. To keep the number of fine-tunable parameters on the same level, we either keep the bottleneck dimension the same and reduce the sparsity ratio k% or use the same sparsity ratio with reduced bottleneck dimension for the only mask setting. We also slightly increase the bottleneck dimension of the only share set up to compensate for the parameters of masks. Our results, as presented in Table 4, indicate that the random mask setting yields the poorest performance. Moreover, neither only mask nor only share reaches the same performance with PROPETL, highlighting the necessity to use both masking and sharing to achieve higher parameter efficiency.5 We believe masking injects crucial structural information into the sub-networks, while sharing is necessary to expand the size of each sub-network when the number of fine-tunable parameters is fixed. Therefore, our method can use the parameters more efficiently.
How do Sub-Networks’ Size and Structure Affect Each Other? The sparsity ratio k% is an important hyperparameter in PROPETL. We study the impact of such sub-network sparsity and present the results in Figure 4. The performance of our PROPETL improves as k% increases from 0.1 to 0.5 but then declines as k% continues to grow from
5Detailed setup can be found in Appendix D.2.
Method PROPETL Random Only Mask(Same bn) Only Mask (Same k%) Only Share
GLUE

heading: Ro-En
text: 0.5 to 1.0. Additionally, it can be seen that all these PETL methods can achieve the best accuracy when k% is set to 0.5. This is likely due to the fact that as the network becomes denser from 0.1 to 0.5, the sub-networks get to use more parameters and thus obtain better modeling ability. However, after 0.5, the network in different layers becomes more homogeneous as the sub-networks overlap more, leading to less distinctive structural information in each layer. The absence of enough structural information starts to harm the performance, which even outweighs the benefits of potential knowledge sharing across the layers. We also discover that sharing the PETL module without any pruning (k% = 1.0) results in the worst performance among all sparsity levels. These results suggest that given the fixed percentage of tunable parameters, it is crucial to find a good balance between the distinctive structural information of each sub-network and the number of parameters used in each sub-network.
How is PROPETL Conceptually Related to PETL? Other than the explanation of prototype network sharing and masking, our proposed PROPETL can also be considered as a PETL-onPETL approach, which we refer to as PETL2. In other words, the mask is to the prototype network (in our approach) as the PETL module is to the PLM (in conventional PETL approaches). Vanilla PETL methods, such as adapter and prefix-tuning, update specific additional parameters for each layer and downstream task while only sharing the parameters of the PLM. In contrast, PROPETL extends this approach by sharing not only the PLM but also the prototype PETL module among layers and tasks, resulting in a higher degree of parameter sharing. Our method uses binary masks that function like PETL modules on top of the prototype
PETL module to prune different structures in different sub-networks. These task-specific tunable parameters are thus an order of magnitude smaller than conventional PETL modules.

heading: 7 Conclusion and Future Work
text: In this paper, we introduce PROPETL, a method for sharing prototype PETL modules across different layers and tasks. Our method significantly improves the parameter efficiency by utilizing the prototype network and maintaining a binary mask for each layer and task. Extensive experiments show that our method achieves comparable performance to fully fine-tuned models and conventional PETL methods with a much smaller fraction of storage. For future works, we aim to study the interpretability of the masks in different layers and explore their potential relationships. We also intend to apply our method to the pre-training process of large language models to reduce the overall number of parameters.

heading: Limitations
text: Although our masks in different layers are binary and require significantly less storage compared to other PETL networks, we still need the underlying 32-bit scores for each mask during the training process. Therefore, PROPETL consumes slightly more memory in the training time than existing PETL methods. To fine-tune PROPETL, it takes a similar training time to conventional PETL modules, which means our method will normally take a longer time to converge compared to the fully fine-tuned model.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers, our meta-reviewer, and senior area chairs for their constructive comments and support on this work. This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Program (AISG Award No: AISG2-RP-2020-016), and AI Singapore Programme (AISG Award No: AISG2-PhD-2021-08-007).

heading: A.1 PROPETLAdapter
text: An adapter module modifies a model’s hidden representation through a down-sampling projection and an up-sampling projection with a non-linear layer in between:
h← h+ f(hWdown + bdown)Wup + bup (7)
where W represents the weight matrix, f denotes the non-linear layer and b is the bias term. In PROPETLAdapter, we apply our binary pruning masks on the up-sampling and down-sampling weights, respectively:
h← h+ f(hW̃down + bdown)W̃up + bup (8)
where W̃down = Wdown⊙mdown and W̃up = Wup⊙ mup.
The original Houlsby adapter (Houlsby et al., 2019) introduces the adapter module after each
multi-head attention and feed-forward layer. Pfeiffer et al. (2021) later propose a more efficient variant of the adapter that is only inserted after the feed-forward layer.

heading: A.2 PROPETLLoRA
text: LoRA (Hu et al., 2022), like adapter tuning, also includes a down-sampling projection and an upsampling projection. The difference is that LoRA does not have any non-linear layers, but it does have an additional scaling factor α ≥ 1:
h← h+ α · xWdownWup (9)
To modify PLMs’ hidden representation, LoRA is applied to the query and value representations of the attention modules in Transformers. In PROPETLLoRA, the network is pruned with binary masks:
h← h+ α · x(Wdown ⊙mdown)(Wup ⊙mup) (10)
The scaling factor α is an important parameter for LoRA and Hu et al. (2022) use different α for different datasets in their released code. In PROPETLLoRA, applying the pruning masks m will reduce the norm of the output of the LoRA module. We find that applying a larger α for PROPETLLoRA than that used in LoRA will remedy the issue of reduced feature norm and result in better performance.

heading: A.3 PROPETLPrefix
text: Li and Liang (2021) propose to insert tunable matrices, which they call prefixes, into the key-value pair of the Transformers’ attention modules:
H = Attention(Q, [Pk;K], [Pv;V ])
= softmax( Q[Pk;K]
T
√ d
)[Pv;V ] (11)
where [·; ·] stands for matric concatenation. They also find that directly updating the P parameters will lead to unstable training and a bit drop in performance. In order to ameliorate the problem, they propose to reparametrize the matrix:
P = P ′W (12)
where P ′ is a smaller learnable matrix and W is the weight of an up-projecting feedforward neural
network. In PROPETLPrefix, we apply our binary pruning masks on the reparametrized prefixes:
H = Attention(Q, [Pk ⊙mk;K], [Pv ⊙mv;V ]) (13) Note that different from adapter and LoRA tuning, the pruning masks in PROPETLPrefix do not directly operate on the parameters of the network. Instead, the masks are applied on Pk and Pv that are output activation depending collectively on W and P ′. Thus, it might be hard for the mask training process to identify good structures from Pk and Pv, which potentially explains the sub-optimal results of PROPETLPrefix in Table 1. To verify this claim, we further compare 3 PETL modules against their counterparts pruned with a sparsity of 0.5 in Table 5. We find that both adapter and LoRA tuning improve their performance when 50% of parameters are pruned, which substantiates the idea that structural information of sub-networks is important. However, for prefix tuning that uses reparameterization, its performance drops when we learn pruning masks on top of it. This shows that the masks fail to locate suitable structures for prefix tuning.

heading: B Parameter Efficiency in PROPETL

heading: B.1 Overview
text: We present an approximate calculation of the bits (space) required by PROPETL during inference in Table 6, in which we assume that a single shared PETL network contains p BLS and the model has L layers. In addition, the storage space required for the binary mask makes up 1/32 (≈ 0.031) of the 32-bit PETL module. Depending on the specific PETL module used, the calculations may vary a bit, and we detail them in the following sections.

heading: B.2 PROPETLAdapter
text: Given the number of layers (L), hidden dimension (d) of the pre-trained language model, and bottleneck dimension (bn) of the adapter module, the BLS consumed by PROPETLAdapter is:
32 · (2 · bn · d+ bn+ d)︸ ︷︷ ︸ Prototype adapter +2 · bn · d · L︸ ︷︷ ︸ Mask (14)
Note that our method does not apply any masks on the bias terms in the prototype adapter. Since our PROPETL will reuse the prototype network, we do not consider the pruning ratio k% when calculating the storage. Here we also discuss how to calculate the storage cost of parameters used under the only mask setting in Table 4, in which we do not share the PETL module across layers but only learn masks to prune them. In that case, we do not count the pruned parameters because they will never be reused. Therefore, the formula for the BLS required by only mask is:
32 · (2 · k% · bn · d+ bn+ d) · L (15)

heading: B.3 PROPETLLoRA
text: Similarly, given the number of layers (L), hidden dimension (d) of the pre-trained language model, and bottleneck dimension (bn) of the LoRA module, the formula for the storage needed by PROPETLLoRA is:
32 · 4 · bn · d︸ ︷︷ ︸ Prototype LoRA +4 · bn · d · L︸ ︷︷ ︸ Mask
(16)
Note that there is no bias term in LoRA. Under the only mask setup, given the sparsity ratio k%, the storage of parameters is:
32 · 4 · k% · bn · d · L (17)

heading: B.4 PROPETLPrefix
text: Given the number of layers (L), hidden dimension (d) of the pre-trained language model, and the prefix length (l) of the prefix module, the formula to calculate the BLS of PROPETLPrefix is:
32 · 2 · l · d︸ ︷︷ ︸ Prototype prefix +2 · l · d · L︸ ︷︷ ︸ Mask
(18)
Under only mask settings, given the sparsity ratio k%, the approximate required storage is:
2 · k% · l · d · L (19)

heading: C Experimental Details
text: We briefly introduce the benchmark datasets used in this work. Their statistics can be found in Table 7.

heading: C.1 Datasets
text: GLUE The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018) is widely used to benchmark models’ language understanding ability. It consists of a broad range of sentence-level tasks, including natural language inference (MNLI, QNLI, RTE), sentence similarity (MRPC), paraphrase detection (QQP, STS-B), and single sentence classification (SST-2, CoLA).
XSum The Extreme Summarization dataset (XSum) (Narayan et al., 2018) is designed to evaluate systems’ abstractive summarization ability of a single document. It is collected from the online articles of the British Broadcasting Corporation (BBC). The input is a single document with an average token count of 431.07, and the model is expected to generate a short, single-sentence summarization of this document.
WMT16 Ro-En WMT16 is the 2016 edition of the Workshop on Machine Translation (WMT) dataset (Bojar et al., 2016). This dataset is widely used to evaluate machine translation systems. To benchmark on WMT16 Ro-En, the model is supposed to take in a Romanian sentence and output translation in English.
C.2 Implementation Details Single-Task Learning on RoBERTaBASE Our models are implemented based on the AdapterHub package (Pfeiffer et al., 2020). We use the datasets (Lhoest et al., 2021) library to calculate each sub-task’s scores in GLUE.
The learning rate of the PETL module is set to 1e-4, and we detail the learning rate of pruning masks in Table 8. We find that it is important to set a higher learning rate for the masks than the learning rate of the PELT network. The batch size is set to 128 and the weight decay is 0.1 in all experiments with Roberta. When conducting experiments on GLUE datasets, we train 10 epochs when the dataset is large (MNLI, QNLI, QQP, SST2), while training 20 epochs on the small dataset (RTE, MRPC, STS-B, CoLA). Table 9 shows the training time taken on a single A100 GPU.

heading: Single and Multi-Task Learning on T5BASE
text: We implement T5 based on the transformers library (Wolf et al., 2020). We use the ROUGE package (Lin, 2004) for ROUGE-2 calculation and sacrebleu (Post, 2018) for the BLEU score. Table 10 shows the training time on a single A100 GPU and the detailed hyperparameters used. We mainly follow He et al. (2022a) and Mahabadi et al. (2021) to select the hyperparameters and do not perform an exhaustive search. The same set of hyperparameters is used across the fully-finetuning, adapter tuning, and PROPETL models. For GLUE, we follow Mahabadi et al. (2021) to sample data from each GLUE sub-task with a temperature of 10. Specifically, a sub-task is sampled with probability p 1/T τ where pτ = Nτ∑T
i=1 Nτ and T = 10.

heading: D Additional Ablation Studies

heading: D.1 Choice of Mask Combining Methods
text: As shown in Table 11, using the OR logical operation to combine the layer mask and the task mask
Method PROPETL Only Mask (Same bn) Only Mask (Same k%) Only Mask (Same k% and bn) Only Share
GLUE
Adapter 64/50% 64/11% 14/50% 64/50% 88/100% Prefix 64/50% 64/15% 15/50% 64/50% 88/100% LoRA 32/50% 32/15% 8/50% 32/50% 44/100% % BLS (0.11%) (0.11%) (0.11%) (0.48%) (0.11%)
Ro-En
Method PROPETL Only Mask(Same bn) Only Mask (Same k%) Only Mask (Same k% and bn) Only Share
GLUE
Adapter 86.60 85.40 84.70 86.56 84.32 Prefix 84.53 84.18 84.23 84.50 81.57 LoRA 85.37 83.46 84.75 85.44 82.53 % BLS (0.11%) (0.11%) (0.11%) (0.48%) (0.11%)
Ro-En
achieves the best performance. This is intuitive because given a specific task and a Transformer layer, parameters that contain the layer information and parameters that contain the task information should be both used in the forward pass.

heading: D.2 Choice of Sharing and Masking
text: We provide additional details of the experiments in Table 4 in this section. With the equations shown in Appendix B, we calculate the bottleneck dimen-
sion (bn) and sparsity ratio (k%) to ensure all the settings have a similar BLS to PROPETL. We also add one more setup, only mask with the same k% and bn as PROPETL. Note that this new setup will result in more storage than the other settings. The hyperparameters are listed in Table 12 and the results are detailed in Table 13.
We find that on the GLUE dataset, PROPETL achieves matched performance to this new setup, even though the latter has used more than 4 times the bit-level storage (0.11% v.s. 0.49%). We believe that, with masks, models with only 0.11% of storage can be enough to have a good performance while more storage cost will not have a significant
improvement or lead to overfitting under simple tasks like GLUE. However, on the more challenging dataset Ro-En, when we keep the k% and bn unchanged and only mask the adapter modules, the model indeed improves its performance by 0.65, but this comes at a cost of around 7× more storage of parameters.

heading: E Additional Results
text: We additionally present the experimental results from Figure 3 in table format in Table 14, 15, and 16.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations.
3 A2. Did you discuss any potential risks of your work? Section Ethics Statement.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 and Appendix C
3 B1. Did you cite the creators of artifacts you used? Section 4 and Appendix C
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Apache License 2.0 gives permission on Commercial use, Modification, Distribution, Patent use and Private use.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. The dataset we applied is a commonly used open-source benchmarks datasets in NLP.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix C
C 3 Did you run computational experiments? Section 5 and 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix C
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4, 5 and 6
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section C
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For future works, we aim to study the interpretability of the masks in different layers and explore their potential relationships. We also intend to apply our method to the pre-training process of large language models to reduce the overall number of parameters