Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning
Authors: Haobo SONG
Hao Zhao
Soumajit Majumder
Tao Lin
Venue: ICLR 2024 poster (2024)
Keywords: Parameter-efficient Fine-tuning
Model Capacity
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Fine-tuning large pre-trained foundation models, such as the 175B GPT-3, has become the prevailing approach for downstream tasks. While parameter-efficient fine-tuning methods have been proposed and proven effective without retraining all model parameters, their performance is limited by the capacity of incremental modules, especially under constrained parameter budgets.
To overcome this challenge, we propose CAPABOOST, a simple yet effective strategy that enhances model capacity by leveraging low-rank updates through parallel weight modules in target layers. By applying static random masks to the shared weight matrix, CAPABOOST constructs a diverse set of weight matrices, effectively increasing the rank of incremental weights without adding parameters. Notably, our approach can be seamlessly integrated into various existing parameter-efficient fine-tuning methods. We extensively validate the efficacy of CAPABOOST through experiments on diverse downstream tasks, including natural language understanding, question answering, and image classification. Our results demonstrate significant improvements over baselines, without incurring additional computation
or storage costs. We will make our code and benchmark publicly available.

Published as a conference paper at ICLR 2024
INCREASING MODEL CAPACITY FOR FREE: A SIMPLE STRATEGY FOR PARAMETER EFFICIENT FINE-TUNING
Haobo Song?.* Hao Zhao? Soumajit Majumder
Tao Lin
haobo. song@epfl.... ch: hac. zww.@epfl........ eeeee....c ch;
souma
jit. itt
ma jumder rhhuawe com; 1intao@westlake.een
WWestlake University 2EPFL HHuawei
ABSTRACT
Fine-tuning large pre-trained foundation models, such as the 175B GPT-3, have attracted more attention for downstream tasks recently. While parameter-efficien fine-tuning methods have been proposed and proven effective without retraining all model parameters, their performance is limited by the capacity of incremental modules, especially under constrained parameter budgets.
To overcome this challenge, propose CAPABOOST, simple yet effective strategy that enhances model capacity by leveraging low-rank updates through parallel weight modules in target layers. By applying static random masks to the shared weight matrix, CAPA BOOST constructs a diverse set of weight matrices, effectively increasing the rank of incremental weights without adding parameters. Notably, our approach can be seamlessly integrated into various existing parameter-efffiieet fine-tuning methods. We extensively validate the efficacy of CAPABOOST through experiments on diverse downstream tasks, including natural language understanding, question answering. and image classification. Our results demonstrate significant improvements over baselines, without incurring additional computation or storage costs. Our code is available at https::////e / httitggtttttttt thub. com/LINS:
INTRODUCTION
In recent years, the prevailing training paradigm has revolved around pre-training models on large- scale datasets and subsequently fine-tuning them for diverse downstream tasks, yielding remarkable achievements. However, the increasing size of popular pre-trained models, such as LLaMA2 (Touvron al., 2023) and GPT3 with size of 175B (Floridi & Chiriatti, 2020), poses significant challenges for full-sized model fine-tuning. Memory and storage limitations restrict its practicality and applicability.
Parameter Efficient Fine- (PEFT) emerges head-on. Unlike the resource-intensive nature of full-size fine-tuning, PEFT adopts judicious approach by either fine-tuning small subset of the original model's parameters introducing limited number of additional parameters during the fine-tuning process. This strategy effectively alleviates the memory and computation burdens, providing cost-effective alternative that match or surpass the performance of full-size fine-tuning. Prominent PEFT techniques employed today include Prompt Learning (Sun & Lai, 2020), Prefix-Tuning (Li & Liang, 2021), Adapters (Houlsby et al., 2019; Pfeiffer et al., 2021), 1,, and LoRA (Hu et al., 2022).
compelling solution to address these challenges
The core concept of these PEFT techniques lies in approximating single layer's heavy-weight matrix w E Rd1 x by two consecutive layers possessing much smaller inner dimension r such B Xr A dR This approach results in significant reduction in parameter count (evident in Adapters and LoRA), achieving performance comparable to full-size fine-tuning while retaining only 1% of trainable parameters. However, the inner dimension cannot be arbitrarily small, as its impact on performance is substantial. For instance, in LoRA, the trainable parameter B A is constrained by rank BA) < r (Hu et al., 2022), setting an upper bound on the model's capacity. These capacity constraints often lead to suboptimal performance when r is small (He et al., 2021).
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2024
87
130
CapaBooost Adupter
CapaBoost Adapter
86
104
Perfomance
85
O ProPETI Acapter
7epopuBooett LoRA
PAdapter
ProPETL Prefix.
CapaBoost LoRA
BAnk
84
PAdapter
ProPETL LORA
52
ProFETI Acapter
LoRA
AdA
PrOPETL Pistx
ProPETL LORA
0.65
0.74
0.83
0.92
1.01
1.1
0.85
0.74
0.83
0.82
1.01
% FT Params
% FT Params
Figure I: Rank and Performance comparison among several PEFT methods in GLUE test with RoBERTa base model. The left figure shows the performance and parameter numbers (shown as percentage to fully fine-tuning) of different methods and indicates CAPABOOST is the hest. The right figure shows the rank and parameter numbers of several methods and CAPABOOST has the highest rank among similar PEFT methods.
The pursuit of an innovative method capable of reducing parameters while preserving high model rank becomes paramount. To tackle this challenge, we present CAPABOOST, a versatile framework aimed at increasing model capacity for free in various PEFT approaches, c.g., Prefix-Tuning, Tuning, Adapters, LoRA, and their variants. Through the integration of model pruning and weight sharing, CAPABOOST significantly amplifies model capacity while simultaneously reducing parameter count, all without inflating FLOP numbers. Leveraging the elevated rank and expanded model capacity, CAPABOOST PEFT methods demonstrate substantial performance improvements over their original counterparts, as demonstrated in Figure 1. Our contributions are summarized as follows:
We introduce CAPABOOST, a plugin framework that seamlessly integrates with various PEFT methods, such as Adapters and LoRA. Through model pruning and weight-sharing, CAPABOOST enhances model rank without incurring additional costs.
Extensive results unequivocally demonstrate that CAPABOOST outperforms all state-of-ther -tee-errtt baselines while significantly reducing parameter count and maintaining the same or fewer FLOPs.
2 RELATED WORK
We provide compact summary here due to space issues. A complete discussion is in Appendix J.
Parameter-Efficien Fine-tuning (PEFT). the realm of PEFT, the initial approach involves selecting subset of the original model parameters for updating, such top layers (Donahue et al., 2014), specific model layers (Gheini et al., 2021), or internal modules (Zaken et al., 2022). While these brute-force selection methods effectively reduce the number of trainable parameters, they often result in sub-optimal performance. Consequently, various scoring functions have been employed to assess the importance of each parameter (Sung al., 2021; Ansell et al., 2022; Guo al., 2021). However, these scoring functions are typically task-specific and involve additional computational overhead.
Another line of work proposes sharing the pre-trained network and incorporating task-specific train- able modules, significantly reducing storage costs (Zhao et al., 2020). Notably, HAdapter (Houlsby al., 2019) introduces adapter modules after each feed-forward and attention layer, while PAdapter (Pfeiffer et al., 2021) further suggests using adapters after FFN and LayerNorm modules (Ba al., 2016) for improved efficiency. Drawing inspiration from textual prompting methods (Sun & Lai, 2020; Liu et al., 2019; Jiang et al., 2020; Shin et al., 2020), Prefix-Tuning (Li & Liang, 2021) employs additional prefix vectors that are prepended to the keys and values in the attention layer. Building upon this, He et al. (2021) conducts systematic exploration of existing advancements and proposes diverse range of Adapter variants by incorporating design elements from Prefix- -unnin. To alleviate the inference latency caused by incremental parameters, Hu et al. (2022) employ a bottle- neck structure with low-rank constraint, enabling the merging of learned weights into the pre-trained network. However, these methods are limited by the prescribed rank values, which impose constraints on the maximum rank of trainable parameter matrices, thus limiting the model's capacity. In contrast, our work aims to overcome such capacity limitations and enhance parameter efficiency for fine-tuning.
Low-rank Properties in Deep Neural Networks. In the over-paramettrrized regime, it has been widely observed in various deep learning tasks that neural networks exhibit low-rank properties following training (Oymak et al., 2019). Building upon this insight, several works (Jaderberg et al., 2014; Sainath et al., 2013; Khodak et al., 2020) propose explicitly incorporating low-rank constraints during neural network training, leading notable successes in CNNs. Following similar approach,
2

---

Published as a conference paper at ICLR 2024
LoRA (Hu et al., 2022) and subsequent studies (Dettmers et al., 2023; Zhang et al., 2022a; Chavan et al., 2023; Chen et al., 2023) adopt low-rank updates applied to a frozen pre-trained network for fine-tuning in downstream tasks.
While modern neural architectures like transformers have demonstrated (Aghajanyan et al., 2021; Wang et al., 2020) low-rank characteristics in their dimensions and representations, Bhojanapalli al. (2020) highlight that the low-rank property of key and query projections in the multi-heal attention modules becomes a bottleneck for transformer performance. Furthermore, experiments in Lialin et al. (2023) reveal that transformers with low-rank updates may under-perform that of full-rank baselines in some cases, especially when the fine-tuning weight rank is small.
2.1 PARAMETER PRUNING FOR PEFT METHODS
Weight-tied models, also referred to as weight- or weight-tying models, represent a parameter- efficient neural network architecture where the same set of weights is shared across different layers or segments of the input (Dehghani et al., 2019; Dabre & Fujita, 2019; Xia et al., 2019; Lan al., 2020; Li et al., 2021; Takase & Kiyono, 2021). Serving as the foundation for most implicit models, this architecture has garnered significant attention in recent years across wide range of tasks (Wang et al., 2019; Liu et al., 2020; Yang et al., 2018; Lan et al., 2020; Takase & Kiyono, 2021; Zhang et al., 2020; Bender et al., 2020; Xie et al., 2021; Li et al., 2021). Pruning, an orthogonal yet extensively employed strategy, aims enhance neural network efficiency by identifying and eliminating redundant parameters (Wang et al., 2022a; Frankle & Carbin, 2019; Lin et al., 2020; Su et al., 2020; Frankle et al., 2021; Wang et al., 2022b; He et al., 2022b). To the best of our knowledge, our idea of introducing various deterministic random masks to a parallel weight-tied model is novel.
The most relevant work our method might be Bai et al. (2022), which extends the codebook idea and learns masks on top of a fixed random weight vector to represent diverse dense layers. Specifically, masks are utilized to select values from a random vector (i.e., codebook), resulting in layers with distinct structures. Simultaneously, Zeng et al. (2023) introduce ProPETL, a method that also incorporates shared weights but employs different masks to encode layer-specific and susk-specific knowledge. Our work, in contrast to these approaches, learns shared weights with deterministic random binary masks, leading to stronger model capability.
AdaLoRA (Zhang et al., 2022a) offers an alternative approach to achieving parameter efficiency by dynamically allocating the parameter budget across different layers. This is accomplished through iterative pruning of singular values of incremental parameters based on an importance metric. However, is worth noting that AdaLoRA entails additional computational overhead and necessitates a higher initial budget of trainable parameters, making it less suitable for low-resource scenarios.
3 CAPABOOST LEARNING FRAMEWORK
3.1 MOTIVATION: RANK MATTERS IN PEFT METHODS
LoRA (Hu al., 2022), Adapter (Li & Liang, 2021), and Prefix-Tuning (Houlsby et al., 2019) the representative PEFT methods in the community. These PEFT methods share similar insights and can be unified (He et al., 2021). More specifically,
The core idea of LoRA is proposing to model the incremental update of the pre-trained weights W Wree-triied E dddx xd2 through low-rank approximations z = (W pre trained + BA), where BA is trainable and in low rank, x E gdi, B E Ri A E gr and < dd). In Adapter or Prefix-Tuning, a module is inserted into a pre-trained model with structure z = x xW pre-trained + (xB)A, where B E Rd: xr and A rx22 are trainable low-rank parameters. x E Rd is the input data, fwon-lincar( (-) is the non-linear function, and < (2),
While the compact dimensions of B and A offer notable parameter efficiency benefits, they also impose significant constraints on model capacity due to the inherent low-rank nature of their product, particularly when inner dimension is low. This restriction inevitably leads suboptimal model performance, as evidenced by the findings presented in Figure 4 of He et al. (2021), Figure 2 of Zhang et al. (2022a), and our Figure 3(b).
To unlock superior performance, a crucial imperative emerges: elevating the rank of BA.
3.2 INCREASING MODEL CAPACITY BY INCREASING THE RANK
Hereby, we propose the following theorem which indicates efficient potential increase rank.
3

---

Published as a conference paper at ICLR 2024
LoRA
Input
Input
(2) rankeri S.d
Pruning
1+1.2....
Add
Add
(1) ranked
(4) ranksmint, + (j.d)
Adapter
(8) rankery S
Input
Inpul X
(a)
(b)
Figure The framework of CAPABOOSK (a): Diagram of CAPA learning with d=2. After applying blue and red pruning masks to the original w (1), we obtain w Milu in (2) and w O Meed in (3), both sharing the same dense weight as (1). Since (2) and (3) have common pruned weights, we can exclude common pruned weights from the original and store the sparse weights in (4), benefitting from fewer parameter numbers. During the training. we retrieve weights from (4) and apply respective masks (m:} to obtain weight in (2) and (3). (b): Diagram CAPABOOSI example LoRA and Adapter.
Theorem 3.1. Assume two matrices and Y are randomly generated by X X X and
Y You yno respectively. X == [x]]] X2
,...,, ,..,, E Rdxr, where column vector basis
{xi") X22 ....,.,, ..,,,, are sampled from NO, Ia). Similarly, X = [[x .x2 ...., [..,,,"""] xx="] E
xco!
Bexd by sampling row vector basis {x("w x/2 ....., ;,xx from ((0, Ia). Ia E RRaxd denotes an identity matrix. For matrices X IRd and E Baxd we have
rank(X (x)) rank( X) (ank(()+randdi rankk( with probability equal to I almost surely when 2r d. (1)
Proof of this theorem is provided in Appendix A.
Remark 3.2. The matrix X in Theorem 3.1 is generated by the multiplication of two low-rank matrices, which share an identical form with trainable weight w = BA in the PEFT methods. The vectors in X and Nnw row are sampled from Gaussian distribution, while A is also initialized in Gaussian. Hence, the high similarity between w and X suggests that this theorem might be applicable and effective in PEFT methods well.
3.3 CAPABOOST: INCREASING MODEL CAPACITY FOR FREE!
In addition to increasing the rank of low-rank-bass -rased PEFT methods to enhance model capacity, we explore complementary approach in this section based on the insights from Theorem I.
add more parallel trainable parameters. In detail, the capacity of a general linear layer w with z wx + b can improved by:
A naive solution. Theorem 3.1 intuitively motivates
Ed w.x + b
Σi= w: b,
(2)
where rank(w) 2 d rank(w). once [W,} can satisfy the distribution assumption in Theorem 3. However, this approach becomes impractical due to the d-1 times increase in the number of additional parameters, rendering PEFT methods inefficient. As remedy, we introduce the CAPABOOST, cost-free solution, detailed below.
CAPABOOST framework. We leverage the idea of weight-tying to alleviate the parameter- inefficient overhead caused by the naive solution discussed above. As intuitively visualized in Figure 2(a) and equations below, CAPABOOST constructs {w,) for free using diverse pruning masks (m:) with original weights
(w O m. x + b w O m. x bb
(EL
(3)
where d is the number of parallel tied models, and m; is the corresponding pruning mask. Unless specified otherwise, we utilize sparsity of 0.5 in each m; by default. These boolean masks (m,) distinct, static, and non-trainable across the training, and should be determined before the training phase. The storage overhead of these boolean masks can be avoided by using a random generator to generate deterministic masks the fly with several scalar seeds per forward and backward pass.
4

---

Published as a conference paper at ICLR 2024
With pruning masks, w Q m, and m, become different matrices and could potentially increase model rank following Theorem 3.1. The additional study shown in Appendix suggests that the superiority of CAPABOOST cannot solely be attributed to the implicit regularization induced by the random masking, which emphasizes the advantage of boosting the model capacity. The CAPABOOST framework not only results in increased model capacity but also allows a further reduction in the number of parameters and even FLOPs. The original parameter w will be pruned by ratio of 1 sd with independently generated masks, where s represents the sparsity ratio of masks. The pruning mask storage can also be avoided by the random generator mentioned above. An illustration is given in Figure 2(a).
Benefits of CAPABOOST.
Benefiting from the sparse matrix, the inference phase of CAPABOOST enjoys both reduced parameter numbers and FLOPs. Though the total FLOPS of CAPABOOST during the training is factor of s x to that of a dense layer, it could remain unchanged or lower than the dense one by appropriately selecting values for and d. Moreover, the computation involving sparse matrices can be accelerated using NVIDIA's sparse tensor core technology for both training and inference (Choquette et al., 2021; Zhang et al., 2022c). In Appendix H, we demonstrate that CAPABOOST can enjoy the hardware acceleration while preserving the performance gains from the increased model capacity.
Therefore, with proper selection of and d, we can implement CAPABOOST nearly for free as can achieve fewer parameter numbers with similar FLOPs compared to original PEFT methods.
Adapting CAPABOOST to PEFT methods. In Figure 2(b), illustrate how to utilize CAPABOOST as a plugin for various existing algorithms to further enhance the model capacity. CAPABOOST-LORA (adapt z x( pro-trained + BA)):
xW prestrained + x E (B @ ma, (A O m..))
(4)
CAPABOOST-Adapter CAPA (adapt z x xW XWrre trained + f linear (x
A):
Ei (Face (ineau(((((xB (x(B ms; (A mc.)).
xW pre truined
(5)
Remark 3.3 (The consistency between intuition and practice). To verify the idea of rank expansion, we measure the rank in LoRA and CAPABOOST-LORA separately, denoted as rank( B A) and
rank (L mb, (A O ma.)). Using RoBERTa-base model as the base model following settings in Zeng et al. (2023), we train LoRA and CAPABOOST-LORA subtask dataset ( (Wang et al., 2018)) and compare their ranks at the final epoch. The results in Table 1 consistently show that the rank of CAPABOOST-LORA is always d times of that in LoRA, regardless of any changes in the inner rank r. This observation confirms our theoretical intuition.
the CoLA (i.e., GLUE
Table CAPABOOST-LORA always has calculated in the RoBERTa-base model after training on the CoLA dataset using three different random seeds. represents ber of parallel tied modules, and r represents the inner dimension of matrices B and A. #Param is factor to that of LoRA (r 8).
4 EXPERIMENTAL SETUP
higher rank than that of LoRA. The rank is
We briefly summarize the experimental setup this sec- tion. More details can be found in Appendix B.
num-
Models and Datasets. We conduct broad range of experiments on various tasks, including (a) Natural Language Understanding (NLU). (b) Question Answering (QA). and (c) Computer Vision (CV). In NLU, we evaluate the fine-tuning performance of RoBERTa-base (Liu et al., 2019) and DeBERTaV3-bass (He et al., 2022a) on GLUE (Wang et al., 2018) benchmark using diverse array of PEFT algorithms. In QA, we evaluate the performance of the proposed PEFT method on SQuADv1.1 (Rajpurkar et al.. 2016) and SQuADv2.0 (Ra-
Rank of method
32
LoRA
#Parmm
1x
2x
4x
8x
CAPA BOoSBOLO--ARACCC (4 =
16
64
128
4Param
0.75k
6x
CAraBoosT-LoRA id 4)
32
128
256
#Parom
0.94k
1.9x
3.75x
7.5x
jpurkar et al., 2018). For the CV task, we use ViT-B/16 (Dosovitskiy et al., 2020) as our tre-trained model and evaluate the efficacy of our method on the VTAB-1k (Zhai et al., 2019) benchmark.
Backbones. For NLU tasks, similar to ProPETL (Zeng et al., 2023) and AdaLoRA (Zhang et al., 2022a), we implement CAPABOOST-LORA and BAOST PAdapterr for fine-tuning RoBERTa- base (Liu al., 2019) and DeBERTaV V3- base (He al., 2022a) GLUE. During fine-tuning, only parameters in incremental modules and the text classification head are trainable and the other model parameters remain frozen. For QA tasks, the publicly available fine-tuning example scripts
5

---

Published as a conference paper at ICLR 2024
(Pfeiffer et al., 2020) library and test D basee (He et al., 2022a) model. For CV tasks, the ViT-B/16 (Dosovitskiy et al., 2020) model is used to evaluate the efficacy of our methods for experiments based the VTAB- Ik benchmark.
from
Baselines. We validate the effectiveness of our CAPABOOST framework by comparing it with the following baselines:
Fully fine-tuning is the common practice for adaptation on downstream tasks. All model parameters trainable during fine-tuning.
BitFit (Zaken et al., 2022) only fine-tunes the bias vector in each layer of the model.
Prefix-Taming (Li & Liang, 2021) concatenates two sets of trainable prefix vectors the keys and values in the attention layer, respectively. Then multi-head attention is computed based on the new prefixed keys and values.
Adapter luning inserts two adapter modules, which are connected consecutively by nonlinear activation function, between transformer blocks. We consider HAdapter (Houlsby et al., 2019) and its popular variant PAdapter (Pfeiffer et al., 2021) as our baselines. The comparison of budget configuration between HAdapter and PAdapter can be found in Appendix D.
LoRA (Hu et al., 2022) parameterizes incremental updates by two low-rank matrices. The number of trainable parameters is solely dependent on the preset rank value r and the depth of the model n. AdaLoRA (Zhang et al., 2022a) emphasizes the varying importance of different weights and innovatively proposes to dynamically allocate the parameter budget among different layers during fine-tuning. It is worth noting that AdaLoRA inserts incremental modules into all weight matrices and achieves better performance compared to only fine-tuning queries and values in LoRA'.
ProPETL (Zeng et al.. 2023) shares single prototype PEFT module across all layers and tasks, and learns diverse binary masks that represent layer-specific and task-specific updates for adaptation. Since PEMN (Bai et al., 2022) shares the similar idea with ProPETI. and the latter is the current oftth-of-the-ar, we only use ProPETL as our baseline in this work.
Setups. To broadly compare with other approaches, replicate the setups used by prior work and reuse reported optimal performance from the original papers as much as possible. We develop CAPA- BoosT-LoRA and CAPABOOST-PAdE
in the codebase provided by ProPETL (Zeng et al., 2023). In Table 2, we follow the evaluation setup as ProPETL (Zeng et al., 2023) on the RoBERTa-base model, which divides the original development set into two parts: evaluation set and test set. Model selection is performed on the evaluation set", and the final performance is reported on the test set. In Table 3, we adopt the evaluation strategy from AdaLoRA (Zhang al., 2022a), which uses the original develop- ment set as the test set and skips model selection. For all baselines, we use the method-specife hyper- parameters reported in prior work and only tune the learning rate to get the best performance. Hyper- parameter tuning details defer to the Appendix. We use NVIDIA RTX-4090 for all experiments.
RESULTS
Our CAPABOOST is able to unleash the potential of PEFT methods across both NLP and CV domains, by increasing the model capacity without introducing additional parameters and computational cost.
5.1 NATURAL LANGUAGE UNDERSTANDING
Fine-tuning RoBERTa-base on GLUE We summarize the performance of CAPABOOST-LORA. We--BoSTT-PAaapeer and other baselines on the GLUE benchmark in Table 2, following the evaluation setup as ProPETL (Zeng et al., 2023) on RoBERTa-base model. Both CAPABoOST-LORA and CAPPAOOST-PAdapter achieve better performance compared to their respective counterparts, while containing even less truinable parameters. Specifically, (1) CAPABOOST-PAdapee significantly outperforms the original PAdapter (86.31 v.s. 84.30) in terms of the average score on the GLUE benchmark, while reducing the trainable parameters by 25.3% (0.71% v.s. 0.95%); (2) CAPABOOST-LORA also improves the score of LoRA by 1.9 with the same extent of parameter reduction
These results reveal that CAPABOOST randomly masking weights in parallel tied modules is capable of enhancing the fine-tuning performance on downstream tasks via increasing the actual rank of incremental weights.
I For the sake of fair comparison with other methods, we append incremental weights to self-attention layers for Adal.oRA in Table 2. We reuse AdaLoRA results from the prior work (Zhang et al., 2022a) in Table 3, which adds incremental weights to all weight matrices.
We modify the AdaL LoRA codebase facilitate evaluation under this setup, ensuring a fair comparison.

---

Published as a conference paper at ICLR 2024
Table 2: Performance of all PEFT methods based on RoBERTa-base the GLUE tasks. We use two parallel tied modules with 0.5 sparsity here CAPABOOST-LLRA and CAPABOOST PAdaate,,, in which the rank of each returned parameter matrices is doubled. The best results on each downstream task are shown in bold. Here % FT Params stands for the percentage of trainable parameters relative to that in fully fine-tuning. We report Matthew's s Correlation for CoLA, average correlation for STS-B, and accuracy for the other tasks, Higher is better for all metrics, "*" indicates the results are reported in prior works. FLOPs are calculated a factor relative to that of
LoRA. Results are averaged over three trials. The details of budget configuration can found in Appendix D.I.I.
CoLAA
SST-2
MRPC
QQP
STS-B
MNLI
QNLI
RTR
All
Methods
FLOPP
Params
M con:.
Acc.
Anc.
Acc.
Corr.
Acc.
Acc.
Acc.
Avg.
Fine-tuniag"
100.00%
60.54
94.04
87.25
91.34
90.83
87.57
92.63
73.14
84.71
Prefix- Tuning
0.96%
63.27
94.42
80.05
88.86
90.43
85.76
91.46
83.38
ProPETI Prefis-Tuins
1.08%
61.81
94.00
87.42
90.48
85.73
91.06
LORAY
0.50%
02.24
98.81
86.76
88.79
90.61
86.39
91.85
67.63
83.54
PROPETL-LRRA"
1.04%
1.02x
62.16
93.62
88.73
87.00
90.88
85.30
91.75
72.66
$4.00
AdaLoRA
1.07%
1.44x
63.86
g4.11g0
98.2 98.24_..1
1.1
90.03
g0.03
91.36
87.08
gou)
92.33
70.49
85.44
CAPABOOST LoRA
0.71%
1x
63.74
g4.19
88.24..c.o
90.50 88 20.3
91.52 2012
87. 60.00.
02, px.2
75.06
85, 25.45
P'ddgser*
0.96%
1x
63.15
86.00
86.03
80.78
90.74
87.10
02:23
70.50
84:30
ProPETL-PAhgger*
1.0%%
1.03x
65.43
96.15
88.24
80.40
91.14
86.53
92.58
76.50
85.50
CAPABOOST PAdapter
0.71%
1x
06.88
94.50....
89.38.:.
91.13
91.00.a01
87.30.oo.
92.59
77.94.2.4
86.35
Table 3: Performance of all PEFT methods based on DeBERTa b-ae on the GLUE tasks. We use two parallel tied modules with 0.5 sparsity here in CAPABOOST- LoRA and CAPABOOST PAdaate,,,,,,r in which the rank of each returned parameter matrices is doubled. The best results on each downstream task are shown in bold. Here % FT Params stands for the percentage of trainable parameters relative to that in fully fine-tuning. We report Matthew's ' Correlation for Col.A, .A, average correlation for STS-B, and accuracy for the other tasks. Higher is better for all metrics. <gg indicates the results are reported in prior works. FI.OPs are calculated factor relative that of LoRA. CAPABOOST-PAR PPddpptr only has bottleneck dimension of 32 to simulate the constrained parameter budget situation, while the other PAdapter variants have bottleneck dimension of 64. Results are averaged over three trials. More details of budget configuration can be found in Appendix D.2.1.
Methods
% FT Params
FLOPS
CoLA
S8T-2
MRPC
QQP
STS-B
MNLI
QNLI
RTE
M
Acc.
Acc.
Con:
Apc.
Aug.
Fine-tuning
100.00%
60.19
65.63
89.46
92:40
81.60
80.90
04.03
53.75
88.25
BirFie
0.05%
66.96
64.84
87.75
88.41
01.35
89.37
02:24
78.70
86,20
LoRA*
0.72%
1x
69.82
94.95
89.95
91.99
91.60
90.65
93.87
85.20
88.50
AdaLoRA*
0.09%%
2. 2.15x
71.45
96.10
90.09
92.23
91.84
90.76
94.55
88.09
89.06
0.78%
1.02x
60. 60.12gg
06 72
90.44.0a
91, 10
g1.52g
00.20
94.28
20
88.46_00
CAMBoosTT-Lok
0.54%
72,22
96.80
91,42 11.0
91.08
91.99 c/3
91.10 00.3
98.83
86.64
89.33
HAdapter*
0.66%
1.05x
68.64
96.55
91.91
91.
90.13
94.11
PMdapter*
0.64%
0.00x
68:77
85.61
89.46
92.04
91.54
90.33
94.29
85,20
88.41
ProPETT- PAddugeer
0.69%
1.02x
70.47.00
85
go
90.444.0
01.74gg
81.48g3.
90.46..0
94.32gn
35.92
88.90_..
CAPABOOST PAdagser
0.24%
0.5x
73.133
s..0
96
22 0.2
92.16
1.2
92.26 00.0
92.
10.2
90.12 no.1
94.12 oo.
87.05
89 09.68... 10.4
We also find the performance of CAPABOOST is competitive compared to state-of-the-art contenders: (i) Trrooo--AAaapterCAAAAABBAOA
surpasses over ProPETL-PAdeee PAEELLLAAdapeeePPPPPELLLLL in out of 8 downstream tasks; (ii) CAPABoOST-LORA can perform just as well as AdaLoRA while using significantly less trainable parameters.
Fine-tuning DeBERTaV3 on GLUE We repeat experiments based on the DeBERTaV3 base model and report results on the GLUE development set in Table 3. We observe that CAPABOOST achieves better or on-par overall performance on the GLUE benchmark compared with existing approaches. We examine the capability of CAPABOOST in situations with constrained parameter budgets by cutting the bottleneck dimensions of CAPABOOST- PAdapter by half, leading to only 0.24% trainable parameters relative to that in the fully fine-tuning. The superior performance of CAPABOOST-PAAaa
despite having only half the bottleneck dimension compared to other baselines, highlights its ability effectively leverage shared parameters in parallel tied modules and learn more powerful features for adaptation.
5.2 QUESTION ANSWERING
To verify CAPABOOST can also work in more difficult tasks, evaluate our method two QA datasets: SQuADv1.1 and SQuADy2.0, under four different budget settings: 0.08%, 0.16%, 0.32%, and 0.65%. The results are presented in Table 4. We find that
7

---

Published as a conference paper at ICLR 2024
Table 4: AAPAPooST-PAdaater results with DeBERTaV3-base on SQuAD v1.1 and SQuADv2.0. We two parallel tied modules with 0.5 sparsity here in CAPABOOST PAdapte.. Here % Params the number of trainable parameters relative to that in full fine-tuning, We report EM/FI for both tasks. The best results in each setting are shown in bold. Higher is better for all metrics. CAPA BOOST--Adappte achieves the best overall performance compared to other baselines, 1.0% /0.7% higher than LoRA, the second best- best-performing method on SQuAD datascts, while containing less trainable parameters.
SQuADvI.1
SQnADv200
Arg
Full
86.0/92.7
85.4/88.4
85.7/90.6
Params
0.16%
0.32%
0.08%
0.65%
HAdageer
84,4/91.5
85.3/92.1
86.19227
86.7/92.9
83.4/96.6
84.3/87.3
84.9/87.9
85.4/88.3
85.1/89.9
PAdapeer
84,4/91.7
85,9992.5
86.29288
86.0/93.0
84.2/87.2
84.5/87.6
84.9/57.8
84.5/87.5
85,29000
LORA
86,492.8
86,69299
86 86.7/9311
86 86.7/3.1
84.7/87.5
83.6/6677
84.3/87.4
05.008.0
85.5
FT Parama
0.06%
0.12%
0.345
0.49%
0.06%
0.129
0.24%
0.49%
Pooteoos-P 87.6/93.5 87.3993.4
87.4/934
87.7/93.5
$5.1/879
N5.5A88.
85888866
85.8/88.6
86.5/9999
Table 5:
for parameter-efficinnt transfer learning on VTAB-1k. CAPABOOST-LORA achieves better overall performance, 1% higher than LoRA, while the parameter numbers remain the same. We follow the experimental settings Zhang et al. (2022b). The base model used VIT-B/ VTT- pre gre-trained on ImageNet-22K, and CAA./CAAABBOOS---CCO
contain 0. 0.29M/0.22M, trainable parameters. Both LoRA and CA- PABOOST-LORA have inner dimension r = 8 and are trained for 100 epochs. The result of VPT-Deep and NOAH are collected from Jia et al. (2022) and Zhang et al. (2022h) respectively. Results are averaged over three trials.
Natural
Specialized
Structured
Aserage
CIRAR-10
DTD
Flower102
Pets
SVHEN
Eurosat
Mesisca5
Clevr-Cont
Clevn DMLaab
dSge-Ort
SMORB-Ai
VPT:Desp
65,50
98.00
88.30
75.10
96.10
53.40
46,50
43.00
10.30
SOAH
70.20
50.10
90.40
86.10
95.40
58.90
R3.31
1.RA
67. 30
69 g3
18-99 95.
90.58
8 81.00g
90.31g
85-95
83.08
67.00
40.88
15.51g0
30 30.80
77.36
69 14.
70-68 1.1
98.91.00
90.61.1
85.01.0
96-11
85.38
8:.00
07.20
40 19
28 2.
12.57
CADABOOST Lukk
65.02.1
73.30
CAPA Boost LaRA (dei)) GR (0
72.18 10 10.0 91.30 21.30
60.78g
85
20.96
82.77
69.11
50.9
15 1l.8
30 08
73.11
CAPABOOST- PAdapter outperforms other baselines, including fully fine-tuning with all model parameters, in all budget settings except for 0.08% parameter budget on SQuADv2. Moreover, it achieves this superiority while being more parameter-efficient, requiring 25% fewer parameters.
The average score improvement compared to the second best- pers-rerforing method, LoRA, is 1.0/0.7.
IIAdapter and PAdapler degrade dramatically when reducing the number of trainable parameters on SQuADv1.1, while our method shows consistent performance under different budget levels. In the most constrained budget setting on SQuADv2, more challenging dataset, CAPABOOST- PAdapter falls short of surpassing fully fine-tuning. We believe that in resource-constraiied scenar- ios for demanding tasks, existing PEFT methods require more efficient parameter allocation across different layers. We leave the adaptive allocation of CAPABOOST over layers for future work.
5.3 VISUAL TASK ADAPTATION BENCHMARK
We further evaluate the efficacy of CAPABOOST on the benchmark with a different modality, VTAB-1k (Zhai et al., 2019), which includes several visual tasks from three categories: natural, specialized, and structured. Experimental results summarized in Table 5 show that our method(d=2)) brings consistent improvements over LoRA on all downstream tasks, leading to average improvement of around 1% on VTAB-Ik, while maintaining the same or less storage and computation efficiency. More parallel tied modules achieve larger performance gain from 73.20 to 73.41 as evidenced in Table 5, which shows vision tasks enjoy more benefits from higher rank updates.
6 DISCUSSION
In this section, we present ablation study results based on the CoLA dataset to validate the effective- ness of our method. Additional ablation studies on the SST-2 dataset can be found in Appendix G.
Different mask sparsity. Figure 3(a) illustrates the results of fine-tuning the RoBERTa-bass model with different pruning mask sparsity on CoLA. Notably, we observe that CAPA reaches peak performance when the density of random masks equals 0.6. This observation highlights the
8

---

Published as a conference paper at ICLR 2024
65
Matthews Correlation
66
60
Matthews Correlation
64
55
50
Same-man mask- Hame-mask-CCpaBaeee CapaBoost-Lokk
Layer- Saayor-Caaa Capa Boost Loost-LoRA
45
Diff mask CapaBrost LoRA
60
layer Capa CopaBrost LnRA
Dropout Capa
3-layer Capa
Fully fine minettningg
Fully finetune
0.2
0.4
0.6 Density
0.8
1.0
10
20
30
40
50
60
Inner Dimension
(a) Mask Density
(b) Inner Dimension Y"
Figure 3: Ablation study for components of CaPA BOoST-LoRA on RoBERTa-base model. (a) Average
performance of CAPA BOOST-LoRA with different pruning masks, same pruning mask, and only Dropout without pruning over different sparsity on CoLA dataset. We use two parallel tied modules with preset LoRA inner dimension of 8. (b) Average performance of CAPABoOST-LORA under different rank values and number of parallel tied modules when density= 0.5 on CoLA dataset. Results are averaged over three trials.
crucial role of random masks in CAPA BOOST, they enforce non-linearity among parallel layers and effectively increase the actual rank of weight matrices, thereby enhancing the performance.
However, observe continuous degradation in the performance of CAPABOOST-LORR when vary the sparsity level in each parallel tied module. We attribute this trend to the diminishing effect of model capacity enhancement when the sparsity level becomes either high or too low. Specifically, when the density is set to 1.0, CAPABoOST-LORP is equivalent to the original LoRA multiplied by coefficient. Conversely, at a density of 0.1, the rank of each parallel weight matrix insufficient to effectively enhance model capacity.
On the influence of mask types. Dropout (Hinton et al., 2012) addresses overfitting by randomly setting parameters activations to zero (Wan et al., 2013; Kingma al., 2015; Gal & Ghahramani. 2016), similar to our masking to some extent. However, our method aims to boost the capacity of dense weights through learning with deterministic binary masks, rather than relying regularization. Figure 3(a) validates the efficacy of our design choice, Diff-mask, by comparing it with two alter- natives: Kamee-makk and Dropout. Our findings reveal that Diff-mask consistently outperforms both alternatives across different sparsity levels, suggesting that different random but deterministic masks in CAPABOOST play vital roles in the performance gain. In the case employing new random weight masking in each iteration like Dropout, it does result in poor performance when the density is low.
Tradeoff between parameter budget and the number of parallel tied modules. discussed in Section 3, the rank of incremental weight matrix bottlenecks the model capacity, especially in the case with a limited parameter budget. We illustrate this point by conducting an ablation study over the inner dimension of LoRA modules shown in Figure 3(b). We find that
The performance of CAPABOOST- -oRA with one single layer on CoLA monotonically increases with the inner dimension r. It is aligned with our intuition in Section 3.1 that the performance is limited by the rank of incremental weight matrices when the model capacity is insufficient. with 2 or 3 parallel layers significantly outperforms the case with only one parallel layer. Specifically. the fine-tunin men-tunin performance of 2-layer-CAt CAPA CAPABOOST BOOST-LoRA with r 32 surpasses that of 1-layer-CAPa BOOST-LORA with 64, where the former one has 25% fewer trainable parameters than that of 1-layer-CAPA BOOST-LLLL
These results demonstrate that the fine-tuning performance bottlenecked by the model capacity and CAPABOOST can alleviate this problem by increasing the model capacity for free. However, we also notice that the performance of CAPABoOST-LORA starts to degrade as we further increase the rank of incremental weights, which might be attributed to potential optimization difficulties e.g. overfitting.
Limitations and future work. Although CAPABOOST has demonstrated its prowess in enhancing model capacity, it becomes susceptible to overfitting when the internal dimension is set to large value. Addressing this issue through adaptive weight allocation across layers remains promising avenue for future research. Furthermore, CAPABOOST remarkable parameter efficiency positions as an ideal solution for tackling demanding applications, like fine-tuning Large Language Models. We also defer exploration of this potential future work.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank anonymous reviewers for their constructive and helpful reviews. We also thank Mr. Shuailong Zhu for his great help in mathematical proof. This work was supported in part by the National Science and Technology Major Project (No. (0222D0111001) the Research Center for Industries of the Future (RCIF) at Westlake University, and the Westlake Education Foundation. Tao Lin and Soumajit Majumder were supported by an Industrial Research Grant from the Huawei Cloud Intelligent Cloud Technologies Initiative when initializing this project.
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 7319-7328, 2021.
Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross- lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1778-17999 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450, 2016.
Yue Bai, Huan Wang. Xu Ma, Yitian Zhang, Zhiqiang Tao, and Yun Fu, Parameter-efficient masking networks. Advances in Neural Information Processing Systems, 2022.
Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, and Quoc V Le. Can weight sharing outperform random architecture search? an investigation with tunas. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 14311-14320. IEEE, 2020.
Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Low-rank bottleneck in multi-head attention models. In International conference on machine learning, pp- 864-873. PMLR, 2020.
Vladimir Igorevich Bogachev and Maria Aparecida Soares Ruas. Measure theory, volume 1. Springer, 2007.
Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parameter-efficient fine-tuning. arXiv preprint vrXiv:2306.07966 2023.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv:2309. 12307, 2023.
Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. Nvidia a100 tensor core gpu: Performance and innovation. IEEE Micro, 41(2):29-35 35, 2021.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning, Electra: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations, 2019.
Raj Dabre and Atsushi Fujita. Recurrent stacking of layers for compact neural machine translation models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6292- 6299, 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https: //openreview.o net/ wwwoommddddyyydiee...
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprins arXiv:2305. 14314, 2023.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint /8.0iv/8100000 2018.
10

---

Published as a conference paper at ICLR 2024
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference machine learning, pp. 647-655. PMLR, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2020.
Luciano Floridi and Massimo Chiriatti. Gpt-3; Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable
neural networks. In International Conference on Learning Representations, 2019. URL https: // //openrevieieee ew net/ / forum? d=r
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? In International Conference on Learning Representations, 2021. URL https: http://ppppeeeeeeeeeeeeeeeeeeeet net taree/forrmme forum fttt:///fooooo....ggg-------11- Vyy---
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation:
Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Mozhdeh Gheini, Xiang Ren, and Jonathan May. Cross-attention is all you need: Adapting pretrained transformers for machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1754 1765, 2021.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter- Earfmmetr-efffcientt transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Itth International Joint Conference on Natural Language Processing ( (Volume I: Long Pupers), pp. 4884 4896, 2021.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirk-arikt and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint 1 2021.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoddig--ehanneed bert with disentangled attention. In International Conference Learning Representations, 2020.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra- style pre-training with pradient-disentangle embedding sharing. In The Eleventh International Conference on Learning Representations, 2022a
Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. Sparseadapter: An easy approach for improving the snrameter-efficienee of adapters. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2184 2190, 2022b.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint 1rXiv:1207.0588, 2012.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp.
PMLR. 2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL ht https: http:////peeeeee//////e/e/ net/ /forum? id=nZeVKeeFYf9.
11

---

Published as a conference paper at ICLR 2024
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405..380 2014.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision, pp. 709-727. Springer, 2022.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Mikhail Khodak, Neil A Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regularization of factorized neural layers. In International Conference on Learning Representations, 2020.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. Advances in neural information processing systems, 28, 2015.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URL https: //wp///eeeewweeeeeeeeeeeeeeeeeee net/ foormm? id-H1eA7AEtwe
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In International conference on machine learning, pp. 6437-6449 PMLR, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4582-4597, 2021.
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky, Stack more layers differently: High-rank training through low-rank updates. arXiv preprint arXiv:2307.055: 2023,
Tao Lin, Sebastian U. Stich, Luis Barba, Danii Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020. URL
https:/ http///peeeeeveeeeeeeeeeeeeeeeee eww net/ peee/foooe fow.mmidddSJee.... d=-JJem8 SFwB.
Kenkun Liu, Rongqi Ding, Zhiming Zou, Le Wang, and Wei Tang. A comprehensive study of weight sharing in graph networks for human pose estimation. In European Conference on Computer Vision, pp. 318-334. Springer, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
George Marsaglia. Bounds for the rank of the sum of two matrices. Mathematical Note, 1964.
Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guaran- tees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint 1rX1v:1906.00332, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, hugh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: framework for adapting transformers. In Proceedings of the 2020 Conference on Empiricul Methods in Natural Language Processing: System Demonstrations, pp. 46 46-54, 2020.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: don-destructive task composition for transfer learning. In 16th Conference of the European Chapter of the Associationfor Computational Linguistics, EACL 2021, pp. 487-503. Association for Computational Linguistics (ACL.), 2021.
12

---

Published as a conference paper at ICLR 2024
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.052550 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint 1806. 2018.
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6655-6659. IEEE, 2013.
Taylor Shin, Yasaman Razeghi, Robert I. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 4222-4235 2020.
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D Lee. Sanity-checking pruning methods: Random tickets can win jackpot. Advances in Neural Information Processing Systems, 33:20390-20401, 2020.
Fan-Keng Sun and Cheng-I I Lai. Conditioned natural language generation using only unconditioned language model: An exploration. arXiv preprint MxXiv:20/1.07344 2020. 2020.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 1124193--4200, 2021.
Sho Takase and Shun Kiyono. Lessons on parameter sharing across layers in transformers. arXiv preprint arXiv:2104.06012 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307. 200.02300,09888, 2023.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pp. 1058-1066. PMLR, 2013.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018
Huan Wang, Can Qin, Yue Bai, and Yun Fu. Rethinking again the value of network pruning dynamical isometry perspective. 2022a. URL https: oppnneevieww net / forummid p41901bJvx.
Huan Wang, Can Qin, Yue Bai, Yulun Zhang, and Yun Fu. Recent advances neural network pruning at initialization. In Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI, Vienna, Austria, pp. 23-29, 2022b.
Sinong Wang, Belinda 7 Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 2020.
Xuesong Wang, Achun Bao, Yuhu Cheng, and Qiang Yu. Weight-sharing multi-stage multi-scale en- semble convolutional neural network. International Journal of Machine Learning and Cyhernetics, 1077):6311-142, 2019.
Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: dixture-of-adapter for parameter-effin efffiiieeeffiiinnt tuning of large language models. arXiv preprint arXiv:2205.1244 1(2),4, 2022c.
13

---

Published as a conference paper at ICLR 2024
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. URL https::////
www ac web. org/ howoooogooooooooooooooooooooooto 2020 emnlp demos
Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. Tied transformers: Neural machine translation with shared encoder and decoder. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5466-5473, 2019.
Tong Xiao, Yinqiao Li, Jingbo Zhu, Zhengtao Yu, and Tongran Liu. Sharing attention weights for fast transformer. arXiv preprint arXiv: 1024, 2019.
Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Lanfei Wang, Zhengsu Chen, An Xiao, Jianlong Chang, Xiaopeng Zhang, et al. Weight-sharing neural architecture search: A battle to shrink the optimization gap. ACM Computing Surveys (CSUR). 54(9):1-37, 2021.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. Unsupervised neural machine translation with weight sharing. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics ( (Volume I: Long Papers), pp. 46 46-55, 2018.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language-models In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, 2022.
Guangtao Zeng, Peiyuan Zhang, and Wei Lu. One network, many masks: Towards more parameter- efficient transfer learning. Proceedings of ACL, 2023.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto. Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arxiv: 10t0:/99//000666 2019.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengeheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning, In The Eleventh International Conference on Learning Representations, 2022a.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv:2206.04600 2022b.
arXiv preprint
Yuge Zhang. Zejun Lin, Junyang Jiang, Quanlu Zhang, Yujing Wang, Hui Xue, Chen Zhang, and Yaming Yang. Deeper insights into weight sharing in neural architecture search, arXiv preprint arXiv:2001.01431. 2020.
Yuxin Zhang, Mingbao Lin, ZhiHang Lin, Yiting Luo, Ke Li, Fei Chao, YONGJIAN WU, and Rongrong Ji. Learning best combination for efficient n.m sparsity. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022c. URL https: openreview net. forum? i d=tbdk YmZ
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking an efficient alternative to finetuning for pretrained language models. arXiv preprint 124400 2020.
14

---

Published as a conference paper at ICLR 2024
CONTENTS OF APPENDIX
A Proof of Theorem 3.1
15
Implementation Details
16
Datasets
16
C.1 GLUE Benchmark
16
C.2 SQuADv1.1 and SQuADv2.0
17
C.3
Vtab-1k
17
Natural Language Understanding
17
D.I RoBERTa-base
17
D.1.1
Budget Configuration
17
D.1.2
Training Details
17
D.2
DeBERTaV3-
18
D.2.1
Budget Configuration
19
D.2.2
Training Details
19
Question Answering
20
E.1 Budget Configuration
20
E.2
Training Details
20
Visual task Adaptation
20
Additional Ablation Study
20
H Hardware Accelaration Utilization
20
I Comparison to regularization-ban baselines
21
J Additional Related Work
21
J.1 Carameter-Efficien Fine-tuning (PEFT)
21
J.2 Low-rank Properties in Deep Neural Networks
................
22
J.3 Weight-Tied Models.
22
J.4 Parameter Pruning for PEFT Methods
22
A PROOF OF THEOREM 3.
We first rewrite the theorem:
Theorem A.I. Assume matrices X and Y randomly generated by X X g and
Y = Y y respectively. Xcol := [x x x.. E Raxr, where column vector basis
{xcm) x2 ...., are sampled from (O, (a). Similarly, X [xx x2" ...., x E
Brxd by sampling row vector basis {xte row ,...,, ...., from N Ia E Badd denotes an identity matrix. For matrices X dRdxd and Y E Rdddd we have
cank(+>X)
+rankK) with probability equal to I almost surely when 2r < d. (6)
We rely on the following two lemmas to finish our proof.
Lemma A.2 (Marsaglia (1964)). Let X and be two matrices of the same size, Ri and R2 are their row spaces, Ci and C2 are their column spaces, O is the linear spuce intersection. The rank of two matrices sum can be bound as:
rank X Y)² Z rank X) + (rank(Y) dim Rí n R2) dim n C2).
Lemma A.3 (Bogaacevv & Ruas, 2007)). I d, then every r-dimensional subspace of R has d-dimensional Lebesgue measure zero.
Proof. Step (1). Fix X, define column space of X
ColumnSpace( = E Rdd x Mex0o ER.
(7)
15

---

Published as a conference paper at ICLR 2024
For X, let X
[11 X2..... xy], then X 2k1 x1 x + T.2 xge
x xx. which is linear
combination of {xfoa x5a ...., x. Therefore, have
ColumnSpace(( X) ColumnSSpac((X*)) for E x= xx===>aa 11x R).
(8)
We want to show that,
Pr (ColummSSpace () ColumnSpace(()) / Ø) =0
(9)
Since rank(X I ColumnSpace( X) is a space built on the r basis vectors and thus a proper subspace of dimension at most r.
We denote the probability measure with Gaussian distribution as p. which is continuous with respect to Lebesgue measure dz.
Pr is linear dependent with x, 200) }}
çu (ColumnSpace(X))
(10)
according to Lemma A.3. Similarly, for vector yj Vjj > 1, since {xcm) ,..,, x vi yj has at most r + - basis vectors, then
Pr (yf linear dependent with (x] x
yi
y
(11)
= μu (ColumnSpace( X) ColumnSpace ((y1
col )-1))) = 0.
Since all y; is linearly dependent with previous vectors with probability 0, can easily show Equation 9.
Step (2). Using the same argument, define the
RowSpace X) = fx E Rd
x= xx===xx Axx"" ERR,
(12)
is easy to prove that,
Pr (RowSpace( (X) n RowSpaceeY"s (Y) + Ø) =0.
(13)
Step (3).
With Pr (ColummSSacce(X) n ColumnSpace YY 2)) Pr (RowSpacee X) RowSpace(Y)) # #0) = 0, we can get Pr (dim R n R2)) = 0 and
and
Pr (dim(C) n C2)) - 0.
By Lemma A. Pr (rank(X X + Y) 2 rank X) + rankkann Y)) Since rank X Y) rank (
(Maarsaaalll,,,, 1964), Pr (rank(X (X + Y) rank x) + rank(Y)) =1
Remark A.4. The condition 2r < d exists because 2r > then Equation 11 doesn't 't hold since r r+) > d when j Physically, since rank( X + Y) dim( Y) impossible that rank(2 X + Y)= = 2r >
B IMPLEMENTATION DETAILS
The implementation of our algorithm builds on the publicly available Pylorch (Paszke et al., 2019) and Transforers (Pfeiffer et al., 2020).
C DATASETS
C.I GLUE BENCHMARK
The General Language Understanding Evaluation (GLUE) is a benchmark of nine sentence- or sentence-pair language understanding tasks (Wang et al., 2018), designed to evaluate and analyze the performance of language model with respect to a broad range of linguistic phenomena found in natural language tasks. Similar to two most recent work (Zeng et al., 2023; Zhang et al., 2022a), we conduct extensive experiments on 8 GLUE tasks, including CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, and RTE. We present the dataset statistics of GLUE (Wang al., 2018) in Table 6.
16

---

Published as a conference paper at ICLR 2024
Table 6: Summary of the GLUE benchmark.
Corpus
Task
#Train
#Dev
#Test
#Label
Metrics
Single-Sentene Classification (GLUE)
CoLA
Acceptability
8.5k
lk
lk
2
Matthews corr
SST-2
Sentiment
67k
872
1.8k
Accuracy
Pairwise Text Classification (GLUE)
MNLI
NLI
393k
20k
20k
Accuracy
RTE
NLI
2.5k
276
3k
2
Accuracy
QQP
Paraphrase
364k
40k
391k
2
Accuracy/F1
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy/F1
QNLI
QA/LII
108k
5.7k
5.7k
2
Accuracy
Text Similarity (GLUE)
STS-B
Similarity
7k
1.5k
1.4k
Pearson/Spearman corr
Table 7: Statistics of the SQuAD dataset.
# Train
# Validation
SQuAD v1.1
87,599 130,319
10,570 11,873
SQuAD v2.0
C.2 SQUADv1.1 AND SQUADv2.0
Stanford Question Answering Dataset (SQuAD) is reading comprehension dataset, consisting of questions collected by crowdworkers on set of Wikipedia articles, where the answer to every question is segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. The statistics of SQuADv1.1 1 and SQuADv2.0 are shown in Table 7.
C.3 VTAB-IK
Visual Task Adaptation Benchmark (VTAB) proposed by Zhai al. (2019), has good representations as those that adapt to diverse, unseen tasks with few examples. It includes CIFAT-1000 DTD, Flower102, Pets, SVHN, Eurosat, Resisc45, Clevr-Count, Clevr-Dist, DMLab, dSpr-Ori and sNORB- Azim datasets and classify them into Natural, Specialized and Structured catagories.
D NATURAL LANGUAGE UNDERSTANDING
D.1 ROBERTA-BASE
RoBERTa (Liu et al., 2019) uses an optimized pre-training recipe based on the one originally proposed in BERT (Devlin al., 2018) and improves the performance on the same tasks with BERT without introducing many more tunable model parameters. It is competitive choice of pre-trained model and broadly used by prior work for comprehensive evaluations (Hu et al., 2022; Zeng et al., 2023). We take the pre-trained RoBERTa-bee et al., 2020) and fine-tune it on the GLUE benchmark (Wang et al., 2018). We initialize the model to the pre-trained model at the beginning of every fine-tuning task.
model from the HuggingFace Transformers library (Wolf
D.1.1 BUDGET CONFIGURATION
We present the budget configuration for experiments on RoBERTa-bass, in Table 8. lr/bb control the parameter budget for percentage of fine-tunable parameters in the model. We use the same peper-paraeeters for all tasks as prior work (Zeng et al., 2023) and show the difference the percentage of trainable parameters relative to the fully fine-tuning.
Luning/LoRR//AAaapteet--tt
respectively. % FT Params denotes the
1.2.2 TRAINING DETAILS
We tune the learning rate from {1 10 10-4.3 x 10
5 10
10 10-4,6 10-x100-44 100 1011x10-3,1.5 x 10 10-3 3.2 x
10-3 3 10-3 x 10-3) and pick the learning rate with the best performance for each dataset. The
17

---

Published as a conference paper at ICLR 2024
Prefix-tuning
PEFT Methods
Prefix
ProPETI. 64 1.03%
# Dimennson ())
64
% FT Params
0.95%
PAdepter-tunring
PEFT Methods
PAdapter
ProPETL 64
CAPABOOST 64
# Dimension (bn)
64
% FT Params
0.95%
104%
0.71%
LoRA
PEFT Methods
LoRA
ProPETL 32 1.04%
CAPABOOST 32 0.71%
AdaLoRA
# Dimension (r)
32 0.95%
32
% FT Params
0.91%
Table 8: Budget configuration for experiments on RoBERTa-base. I denotes the length of prepended prefix vectors in Prefix-tuning. represents the preset rank value of LoRA modules. bn refers the bottleneck dimension of Adapter modules.
details of hyper-parameten shown in Table and Table 10. For fair comparison, only append incremental weights to self-attention layers AdaLoRA as the other baselines do. We set the initial rank of each incremental matrix 48 and the average target rank 32.
Table 9: Hyper-parameter setup of LArABooST-LORA for RoBERTa-base model on GLUE benchmark.
Dataset
learning rate
batch size
# epochs
# tied layers
Density
MNLI
10
128
20
2
0.5
RTE
X 10
128
40
2
0.5
QNLI
5 x 10
128
10
2
0.5
MRPC
x 10
40
QQP
5 x 10
128
20
2
0.5
SST-2
10
20
CoLA
5 X 10
128
40
2
0.5
STS-B
5 10
128
20
2
0.5
Table 10: Hyper-paaee parameter setup CAPABoOST-PAAapte for RoBERTa-base model on GLUE benchmark.
Dataset
learning rate
batch size
# epochs
# tied layers
Density
MNLI
x 10
128
10
2
0.5
RTE
10 3
40
QNLI
2 x 10-3
128
10
2
0.5
MRPC
10
128
30
0.5
QQP
1 x 10-3
128
10
2
0.5
SST-2
X 10
128
10
2
0.5
CoLA
x 10
128
20
2
0.5
STS-B
x 10
128
20
2
0.5
D.2 DEBERTAV3-BD -AERTAAV--BSSS B-BASE
DeBERTa (He et al., 2020) proposes disentangled attention and enhanced mask decoder to enhance model performance and outperforms BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) on a majority NLU tasks with 80GB training data. DeBERTaV3 further improves the training efficiency through using ELECTRA-Style (Clark et al., 2019) pre-training with Gradient Disentangled Embedding Sharing and achieves superior performance on downstream tasks. We take the pre-treee pre-trained DeBERTa ba-e model from the HuggingFace Face Transformers library (Wolf et al., 2020) and fine- tune on the GLUE benchmark (Wang et al., 2018). We initialize the model to the pre- trained model the beginning of every fine-tuning task.
18

---

Published as a conference paper at ICLR 2024
D.2.1 BUDGET CONFIGURATION
We present the budget configuration for experiments on DeBERTaV3-bass in Table 11. r/bn control the parameter budget for LoRAAAAAappeer-tunigg
respectively. % Params denotes the percentage of fine-tunable parameters in the model. We use the same experimental setup for all tasks as prior work (Zhang et al., 2022a) and show the difference
percentage of trainable parameters relative to the fully fine-tuning. Note that AdaLoRA adds LoRA modules to all weight matrices of the model, while other LoRA-related variants only adds incremental modules to self-attention layers. Therefore, A has smaller per-module rank value. We manually smaller bottleneck dimension for CAPA BOOST-PAdapeer here to examine its ability in the constrained budget setting.
Adapter-tuning
PEFT Methods
PAdapter
ProPETL 64 0.69%
CAPABOOST 32
HAdapter
Dimension (bn)
Params
0.64%
0.24%
0.66%
LoRA
PEFT Methods
LoRA
ProPETI. 8 0.78%
CAPABOOST
AdaL.RA
Dimension (r)
Params
0.72%
0.54%
0.69%
Table 11: Budget configuration for experiments DeBERTaV3 stands for the preset rank value of LoRA modules. refers to bottleneck dimension of Adapter modules.
D.2.2 TRAINING DETAILS
We tune the learning rate from (1 10-1,3 x 10 5, 10-1,88 100-188110011111111 10-3, x.5 10 10-3, 3 x 10 3.5 x 10 10-3) 3) and pick the learning rate with the best performance for each dataset. The details of hhper-parameter parameters are shown Table 12 and Table 13.
10
Table 12: Syyeer-arammter setup CAPABoOST-LORA DeBERTaV3-base model on GLUE benchmark.
Dataset
lcarning rate
batch size
# epochs
# tied layers
Density
MNLI
5 X 10
128
20
2
0.5
RTE
10 10
128
50
0.5
QNLI
1 X 10-3
128
15
2
0.5
MRPC
x 10
128
50
2
0.5
QQP
5 x 10
128
20
2
0.5
SST-2
3 10
128
25
2
0.5
CoLA
x 10
15
STS-B
5 x 10
128
25
2
0.5
Table 13: Hyper-parameter setup of CaPABooST-PAdapter for DeBERTaV3-base model on GLUE benchmark.
Dataset
learning rate
hatch size
# epochs
# tied layers
Density
MNLI
5 x 10
128
20
2
0.5
RTE
1,5 10
128
50
0.5
QNLI
5 10-3
128
20
2
0.5
MRPC
5 10
128
50
2
0.5
QQP
x 10
128
30
2
0.5
SST-2
3 x 10
128
10
2
0.5
CoLA
x 10
128
10
2
0.5
STS-B
1 X 10
128
20
2
0.5
19

---

Published as a conference paper at ICLR 2024
E QUESTION ANSWERING
E. BUDGET CONFIGURATION
We present the budget configuration for experiments on DeBERTaV3-base in Table 14.
Table 14: Detailed budget setup for question answering.
Params
Houlsby Adapter
Pfeiffer Adapter
LoRA
SAAABoOST-PAdappee
d
0.65%
32
64
64
0.32%
0.16%
8
0.08%
4
E.2 TRAINING DETAILS
We tune the learning rate from (1 10-1,3 x 10-1,0 10-1,8 100110111111 1 10-3) and pick the learning rate with the best performance for each dataset. The details of in Table 15.
are shown
Table 15: Hyper-parament setup of CAPABOOST for DeBERTaV3-base model question answering tasks.
Dataset
lcarning rate
hatch size
epochs
tied layers
Density
SQuADv1.1
x 10 4
64
15
2
0.5
SQuADv2.0
5 10
64
15
2
0.5
F VISUAL TASK ADAPTATION
For the VTAB-1k(Zha)) al., 2019), we follows its default augmentation settings. All input images resized to 224 x 224 and apply normalization with ImageNet means and standard deviation. The inner dimension LoRA set to 8.
G ADDITIONAL ABLATION STUDY
In this section, we implement additional ablation studies on SST-2-10k dataset, which contains 10k data examples uniformly sampled from original SST-2 dataset, to further analyze the effectiveness of our proposed method. We use the same hyper-parameten in section 6 for all experiments in this section. The results are shown Figure 4.
H HARDWARE ACCELARATION UTILIZATION
The hardware accelaration for sparse matrix operation in CAPABOOST relies NVIDIA's sparse tensor core technology. This requires sparse matrix(e.g, pruning mask in CAPABOOST) a shape of N:M sparsity, for example, 2:4 sparsity. To verify our proposed CAPA BooTB still enjoys performance benefits N:M sparsity matrix, we present GLUE benchmark experimental results Table 16.
The results indicate that CapaBoost performs similarly with both fully random masks and N:M sparsity random masks. Thus, we can directly replace fully random masks with N:M sparsity random masks and benefit from the hardware acceleration provided by the Nvidia sparse tensor core. In our experiment, the fine-tuned weight matrix with N:M sparsity displays the same rank as with fully random masks.
20

---

Published as a conference paper at ICLR 2024
94.5
90
Accuracy
Accuracy aacy
85
94.0
93.5
Same-mask- Lomeemaak--oaaBeettLLAA
1 bayer CapaBoost LoRo
2 2-layer Calayer-CapaBoost CapaBocst-LoRA
Dropout CapaBrost
3 layer CapaBoest LORA
0.2
0.4
0.6
0.8
1.0
93.0
10
30
50
60
Density
Inner Dimension
(a) Mask Density
(b) Inner Dimension
RoBERTa-ben model. (a) Average performance of CAPABOOST-LORA with different masks, mask, and Dropout different sparsity on SST-2- 10k dataset. We use two tied layers with preset LoRA rank value of 8, (b) Average performance CAPABOOST-LoRA under different rank values and number of tied layers when density 0.5 on SST-2-10K dataset. Results are averaged over three trials.
Figure 4: Ablation study for components of CAPABOOST-LoRE
Table 16: Performance of random mask and N:M sparsity mask on the GLUE tasks. This experiment takes RoBERTa-bas pre-trained model with inner dimension=8, and considers two masking strategics: fully random masks and N:M sparsity random pruning masks.
Average
CoLA
RTE
STS-B
MRPC
QNLI
SST-2
MNLI
QQP
Rundom-nasking CAPABoOST-LaRA 85.49 Lo.a 63.20_o. 78.77
90-50
89.54ga7
92. 15. 94.72 86.861o.
80.19.0..
Sparsity CAPAROOST-LoRk
45.51g0n
64.27.0a
76.93
90.52gm: 80-62
92.11 94.531o.
86.00
90.23
Table 17: Detailed comparison results with regularization--aaed baselines.
This experiment takes
RoBERTa-hase pre-trained model with inner dimension=8. The underlying PEFT mechanism for all baselines PAdapter.
Average
CoLA
RTE
STS-B
MRPC
QNLI
SST-2
MNLI
QQP
AdaMix
85 g0.53
63. 19
78.22 go
90.51 go.0
90.11
92.83
94.31
86. go.
89 00.91_.0
Same-mask-CApA BooSS-PoPo-
86 go
02.77.0s
27.98
90.16g00
89.4620.a
92.66
94.00
86 26.00.oo.
90.41
Tippuu-CAPAaoosST PAddapter
85.06 16 so.2
59.92
77.98
90.33
ao.
89.13
92.66
94.82
86.51
80.09-m.1
CupsBuosTPAdaapte
86.33 20.2
65.25
2o.
79.42
a1.2
90.51
20.0 90.36
94.61 92.90.c.1
87.00
90.61.00
I COMPARISON TO REGULARIZATION-BA BASED BASELINES
One potential question arising from our CAPABOOST approach whether the performance gain is primarily due the implicit regularization effect induced by the random masking. We address this concern via comparing CAPABOOST with three new baselines: AdaMix (Wang et al., 2022c), Dropout (Hinton et al., 2012), and same masking. In particular, AdaMix proposes randomly route training examples from a batch of inputs via stochastically chosed PEFT modules and adopt consistency regularization loss. The performance improvements of Dropout and same masking compared to the underlying PEFT modules are purely coming from regularization effects. Experiment results shown in Table 17. Note that we use PAdapter the underlying PEFT mechanism for all baselines in this experiment.
ADDITIONAL RELATED WORK
J.1 PARAMETER-EAFICCEEE FINE-TUNING (PEFT)
The first line of work PEFT picks up subset the original model parameters to update, such as top layers (Donahue et al., 2014). specific model layers (Gheini et al., 2021), and internal modules (Zaken et al., 2022). These brute-fort force selection approaches are effective in reducing trainable parameters, but only leading to sub-optimal performance. Thus, various scoring functions importance each parameter (Sung et al., 2021; Ansell et al., 2022; Guo et al., 2021). However, these scoring functions usually task-specific and require additional computation.
used to measure the
21

---

Published as a conference paper at ICLR 2024
Another line of work proposes to share the pre-trained network and insert task-specific trainable modules to steer the behavior of neural networks, which greatly reduces storage costs (Zhao et al., 2020). In particular, HAdapter proposed in Houlsby et al. (2019) places adapter modules after each feed-forward and attention layer. For better efficiency, PAdapter (Pfeiffer et al., 2021) suggests using adapters after FFN and LayerNorm modules (Ba al., 2016). Inspired by textual prompting methods (Sun & Lai, 2020; Liu et al., 2019; Jiang et al., 2020; Shin et al., 2020), Prefix-Tuning (Li & Liang, 2021) prepends additional prefix vectors to the keys and values in the attention layer. He et al. (2021) systematically examine existing progress and further proposes diverse array of Adapter variants by transferring design elements from Trefix-Tuming
To remove inference latency introduced by incremental parameters, Hu et al. (2022) use a similar bottleneck structure with low-rank constraint, in which learned weights can be merged into the pre-trained network. However, the upper bound of the rank of trainable parameter matrices in these methods is largely affected by the preset rank values, leading to constrained model capacity. Our work aims to break such capacity constraints and enhance parameter efficiency in fine-tuning.
J.2 LOW-RANK PROPERTIES IN DEEP NEURAL NETWORKS
In the over wver-parameterized regime, it has been demonstrated in many deep learning tasks that neural networks enjoy low-rank properties after training (Oymak et al., 2019). Inspired by this insight, some works (Jaderberg et al., 2014; Sainath al., 2013; Khodak et al., 2020) propose reinforce neural network training performance by explicitly injecting the low-rank constraints and achieve great success in CNNs. Similarly, LoRA (Hu et al., 2022) and number of follow-up works (Dettmers et al., 2023; Zhang et al., 2022a; Chavan et al., 2023; Chen et al., 2023) borrow this idea and suggest applying low-rank updates to frozen pre-trained network for fine-tuning on downstream tasks. While state-of-the-art model architectures like transformers have been shown to present a low-rank dimensionality and representations (Aghajanyan et al., 2021: Wang et al., 2020), Bhojanapalli et al. (2020) point out that the low-rank of key and query projections in the multi-head attention modules bottlenecks the performance of transformers. Experiments in Lialin et al. (2023) also demonstrate that transformers with low-rank updates perform significantly worse than full-rank baseline in the training stage.
J.3 WEIGHT-TIED MODELS
Weight- tied models, also known as weight-sharing or weight-tying models, are type of parameter- efficient neural network architecture, in which the same set of weights is used across different layers or parts of the input (Dehghani et al., 2019; Dabre & Fujita, 2019; Xia et al., 2019; Lan et al., 2020; Li et al., 2021; Takase & Kiyono, 2021). This architecture, as the backbone of most implicit models, has been widely studied in recent years for various tasks (Wang et al., 2019; Liu et al., 2020; Yang et al., 2018; Lan et al., 2020; Takase & Kiyono, 2021; Zhang et al., 2020; Bender et al., 2020; Xie et al., 2021; Li et al., 2021). For instance, the Universal Transformer proposed in Dehghani et al. (2019) ties the parameters through one Transformer layer, such idea was later employed in Dabre & Fujita (2019); Lan et al. (2020). In addition, Xia et al. (2019) introduce an encoder-decoder architecture that shares parameters between the encoder and decoder parts. Xiao al. (2019) propose a method to share attention weights to speed up the computation of Transformers. Takase & Kiyono (2021) take this idea further and proposes three strategies to tie the parameters of different layers (with various combinations), instead of just sharing the parameters of one layer with all layers.
The aforementioned studies focus on proposing strategies to tie different layers and do not introduce sparse masks to a shared layer like ours. As a separate line of research, weight-sharing in Neural Architecture Search (NAS) (Zhang et al., 2020; Bender et al., 2020; Xie et al., 2021) reduces the computational overhead by sampling distinct neural architectures from super net using sparse masks, where exponentially many architectures share weights in the same super-network and the costly training procedure is performed only once.
J.4 PARAMETER PRUNING FOR PEFT METHODS
Pruning is a widely used strategy to improve the efficiency of neural networks by detecting and removing redundant parameters. To demonstrate the efficacy of parameter pruning. Frankle & Carbin (2018) propose the Lottery Ticket Hypothesis and show that one can always find a sparse sub-network in dense model that, when trained in isolation, achieve comparable performance to the original dense model. This conclusion also holds for randomly initialized dense models. There exist three directions of pruning, namely i) pruning initialization, ii) dynamic pruning during training, and iii)
22

---

Published as a conference paper at ICLR 2024
pruning after training. The latter two normally prune model weights relying either importance- based or regularization based criteria. The idea pruning at initialization initially also relies on the magnitude-basee metric (Frankle & Carbin, 2019); however, it is debated in several follow-up studies (Su et al., 2020; Frankle et al., 2021; Wang et al., 2022b; He et al., 2022b) that random sampled and only fixes the pruning ratio per layer-aud aree--areeae equally effective the previous lottery ticket (Frankle & Carbin, 2019). Note that the research of model pruning primarily aims to develop improved pruning criteria or suggest enhanced optimization strategies/objective functions. To the best of our knowledge, our idea of introducing various random masks to parallel weight-tied model novel.
The most relevant work to our method might Bai al. (2022), which extends the codebook idea and learns masks on top of a fixed random weight vector to represent diverse dense layers. Specifically, masks are used to select values from random vector (i.e., codebook) and thus form layers with distinct structures. Zeng et al. (2023) concurrently propose ProPETL, which also possesses number of shared weights, on which different masks are learned to encode layer-specific and task-specific knowledge. Our work, in contrast to these approaches, learns shared weights with deterministic random binary masks, leading to stronger model capability.
Another scoring- braing-based approach AdaLoRA (Zhang et al., 2022a) dynamically distributes the parame- ter budget among different layers by iteratively pruning singular values of incremental parameters in correspondence to the importance metric. In addition to extra computation. AdaLoRA also requires higher budget of trainable parameters the beginning of training, which does not apply some low-resource scenarios.
23


Reviews:
[{"review_id": "9Vqxe6SEoB", "reviewer": "Reviewer_ZJAd", "summary": "In this paper, CAPABOOST is introduced as a versatile framework with the aim of amplifying model capacity in a range of Parameter Efficient Fine-Tuning (PEFT) methods, including those like Adapters and LoRA. This is achieved by integrating techniques such as model pruning and weight sharing. CAPABOOST's primary objective is to expand model capacity without adding extra computational overhead, leading to enhanced performance compared to conventional PEFT approaches. The paper provides empirical evidence of CAPABOOST's effectiveness by highlighting significant performance enhancements, reduced parameter counts, and equivalent or reduced floating-point operations (FLOPs).", "questions": "Kindly refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The central concern addressed in this paper pertains to a key issue in existing PEFT methods: while they effectively reduce the parameter count, they often impose limitations on the model's rank.\n2) Through extensive empirical results, it is evident that CAPABOOST surpasses current benchmarks by achieving substantial reductions in parameter count while preserving the same or fewer FLOPs.\n3) The authors commit to providing both the code and benchmarks, which holds the potential to enhance the reproducibility of their research.", "weaknesses": "Regarding the computer vision (CV) task experiments, the paper only presents results for LoRA and LoRA+CAPABOOST. It would be advantageous to incorporate the results of other baseline methods, such as ProPETL[1], to provide a more comprehensive comparison.\n\n[1] One network, many masks: Towards more parameter efficient transfer learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eCivthqLEm", "reviewer": "Reviewer_7oiV", "summary": "This paper provides an approach to improving the performance of PEFT methods. They propose to \"expand\" the effective rank of the learned  low-rank matrices by generating different masks for masking the same base low-rank matrix. \nThey show through ablations and experiments that their approach is complementary with other PEFT methods.", "questions": "1. How do you measure/compute the flops in your paper ? You only mention that its \"relative to lora\"\n2. I am surprised that full finetuning from Figure 4b performs substantially worse that the low rank methods. What hyper-parameters were used ? Was there regularization when training this model ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "In general, I think the method is a clever idea. I believe this idea of generating \"pseudo-independent\" parallel matrices whilst saving storage by applying masks to a base matrix is novel. In the presence of specialized hardware, the sparse matrix form and the use of random-seed to mark masks could lead to considerable memory and compute savings.", "weaknesses": "## Weaknesses\n* Flop improvement from method only exist in the presence of specialized hardware for sparse matrices. \n* Were any of the non-capaboost methods regularized ?  One possible explanation for the improvements of capaboost -- is that the random masking creates implicit regularization. It would be good to compare the performance to other methods, once they also enjoy some level of regularization.\n* I find it hard to understand / believe Table 1 for multiple reasons. First, the claim is that you generally get a d - 1 increase over LoRA (remark 3.3) but  you consistently report dx instead (d - 1)x increase. Second, the basis for the intuition for the dx expansion, is theorem 3.1 which assumes that X and Y are generated independently from Gaussians. In general, this would be the upper bound on the rank of the sum. The matrices you instantiate are not independent (since they are generated from the same B,A combination -- though masked) ... but somehow your results (from Table 1) suggest that you are always achieving this upper bound which is quite surprising. And what about the case where the rank of A, B = 1 ?\n\nThere seem to be quite a few inaccurate statements peppered through the paper : \n1. \"Fine-tuning large pre-trained foundation models, such as the 175B GPT-3, has become the prevailing approach for downstream tasks.\" -- I don't think this is true -- the defacto approach for using models at these scales is prompting and not fine-tuning.\n\n2. \"..... replacing a single layer’s heavyweight matrix w ∈ Rd1×d2 with two consecutive layers possessing much smaller inner dimension r, such as B ∈ Rd1 ×r , A ∈ Rr×d2 . This approach results in a significant reduction in parameter count (evident in Adapters and LoRA), achieving performance comparable to full-size fine-tuning while retaining only 1% of trainable parameters\" -- the original matrices are not \"replaced\" by these smaller ones but this statement makes it seem like the full parameter versions are swapped for the low-rank version\n\n\nIn general, I am not sure I am convinced that the method works for the reasons mention in the paper. I am skeptical of the proposed rank expansion framing -- and I am more convinced that there might be some effective regularization occurring due to the random masks.\n\nNote : I am willing to raise my score if my concerns / questions  are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rk40QKmAC6", "reviewer": "Reviewer_LzY5", "summary": "The paper presents parallel smaller rank multiple low-rank metrices to approximate the frozen model's performance via low rank approximation, during the fine-tuning phase. To do this, the authors propose CAPABOOST, a sparse low-rank approximation with random sparse mask that remain frozen throughout the fine-tuning phase. The authors show results with CAPABOOST to perform better than LoRA, pre-fix tuning and adapter based approaches.", "questions": "1. When the author says training phase with the random generated frozen mask, does he/she mean fine-tuning phase?\n\n2. The author should not claim as \"for free\", as with sparsity = 0.5 would often need storage that can be more than that with no sparsity. Also, additional tensor sparse operation  is needed making the latency to grow a little up, as it is not really N:M sparsity. So, please tone down on your claim, title, and writing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The results are impressive.\n\n2. The paper is well written with related works being presented thoroughly to motivate and differentiate the proposed solution from the existing ones.\n\n3. The baselines and comparison points cover multiple variants of low rank approximation of PEFT.", "weaknesses": "1. This sentence is not clear, \"Furthermore, experiments conducted in Lialin et al. (2023) reveal that transformers with low-rank updates significantly underperform compared to full-rank baselines during pre-training.\".. does it mean the Low rank adaptation during fine-tuning perform poorly? If so, how the LoRA or similar models perform close to baseline full-model fine-tuning? I suppose event the foundation of this paper, assumes that low rank adaptation is useful, and then tries to improve their rank. So, please clarify here.\n\n2. Generation of mask randomly does increase storage cost, as we need to store the index location of the mask while performing forward and backward pass. So, it being storage free is not really a correct statement.\n\n3. The authors claims of speeding up the sparse operation via Nvidia sparse tensor core, however, that is only applicable for N:M sparsity, which the current sparse mask generation process does not satisfy.\n\n4. Can you explain with an example how with sparsity of 0.5 you can have more low rank matrices? And at what point it surpasses the params for a scenario when the sparsity is 1.0 (meaning that of Eq. 2)\n\n5. Isn't capaboost with d>2 would require more fine-tuning params with s=0.5?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)