VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks
Authors: Yang Li
Shaobo Han
Shihao Ji
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter-efficient fine-tuning
Low-rank adaptation
Transfer learning
Transformer
Top-k admixture module
Primary area: deep_learning_architectures
TL;DR: An extreme parameter-efficient fine-tuning method based on vector banks

Abstract: As the adoption of large language models increases and the need for per-user or per-task model customization grows, the parameter-efficient fine-tuning (PEFT) methods, such as low-rank adaptation (LoRA) and its variants, incur substantial storage and transmission costs. To further reduce stored parameters, we introduce a "divide-and-share" paradigm that breaks the barriers of low-rank decomposition across matrix dimensions, modules, and layers by sharing  parameters globally via a vector bank. As an instantiation of the paradigm to LoRA, our proposed VB-LoRA composites all the low-rank matrices of LoRA from a shared vector bank with a differentiable top-$k$ admixture module. VB-LoRA achieves extreme parameter efficiency while maintaining comparable or better performance compared to state-of-the-art PEFT methods. Extensive experiments demonstrate the effectiveness of VB-LoRA on natural language understanding, natural language generation, instruction tuning, and mathematical reasoning tasks. When fine-tuning the Llama2-13B model, VB-LoRA only uses 0.4% of LoRA's stored parameters, yet achieves superior results. Our source code is available at https://github.com/leo-yangli/VB-LoRA. This method has been merged into the Hugging Face PEFT package.

VB-LoRA: Extreme Parameter Efficient Fine- Tuning with Vector Banks
Yang
Shaobo Han
Shihao Ji
Dept. of Computer Science Georgia State University Atlanta, GA 30303 y1193@student. www.:nddetttuuueeee...ee gsu.edu
Optical Networking and Sensing NEC Laboratories America Princeton, NJ 08540 shaobo@nec-labs.. com
School of Computing University of Connecticut Storts, CT 06269 shihao. eeuunnnnnn... edu
Abstract
As the adoption of large language models increases and the need for per-user per- task model customization grows, the parameter-effcient fine-tuning (PEFT) meth- ods, such as low-rank adaptation (LoRA) and its variants, incur substantial storage and transmission costs. further reduce stored parameters, we introduce a "divide- and-share" paradigm that breaks the barriers of low-rank decomposition across matrix dimensions, modules, and layers by sharing parameters globally via a vector bank. As an instantiation of the paradigm to LoRÁ, our proposed VB-LoRA com- posites all the low-rank matrices of LoRA from a shared vector bank with a differ- entiable top-k admixture module. VB-LoRA achieves extreme parameter efficiency while maintaining comparable or better performance compared to state-of- stae-oof-the-art PEFT methods. Extensive experiments demonstrate the effectiveness of VB-LoRA on natural language understanding, natural language generation, instruction tuning, and mathematical reasoning tasks. When fine-tuning the Llamm2-133B model, VB- LoRA only uses 0.4% of LoRA's stored parameters, yet achieves superior results. Our source code is available https: htttt:://githubb gith.u........ cww. leo -aaoo-yagllii /angli/VB-LoRl
This method has been merged into the Hugging Face PEFT packag
Introduction
Parameter-efficient fine-tuning (PEFT) casts a new paradigm that leverages strong prior knowledge built in foundation mod-
69
Matthew's correlation
els and adapts them to a wide range of downstream tasks by
68
updating a small amount of trainable parameters (He al.
67
2021]. Compared to prefix/prompt tuning [Li and Liang 2202
Lester al. 2021] or in-context learning [Brown al. 2020
66
VE-LEAA 1Ound
fine-tuning large-scale pre-trained model yields better domain
65
specialization dictated by high-quality datasets [Brown et al.
2020 Liu et al.) 2022, (Zhoo et al. 2023). This process can be re- peated to suit the needs of ever-changing deployment scenarios and personalizations. However, the sheer volume of param- eter space across a multitude of instantiations (Sheng al. 2023] poses challenges for storage, transmission, and computa- tion, especially for low-resource hardware and consumer- gradd networks [Borzunov et al. 2024].
10a
10F
# of stored parameters
Figure 1: Comparison of the PEFT methods on RoBERTa-Large Our VB-LoRA achieves higher scores with significantly smaller number of stored parameters.
To mitigate these challenges, various PEFT methods have been proposed by adding or adapting small amount of trainable parameters per task without sacrificing performance [Houlsby et al. 2019
Part of the work done while the author was affiliated with Georgia State University. https://huggingtaae
co/docs/ ho/bbcs/ppet//ee//ccaeee/ / weteerenne/vblorra
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Sub-vectors
Tile
Add & Norm
W
LoRA
Feed-Forwward
Waaus
Modules
Vector Bank
Tesinable panerattera
panamaterx
Setect and pool
Stored parometors
Pool
Layers +
+
Tap Sartrax
Salect
Multi- Haatt-HHdd Attention
Lawer2
Logts
Vector Bank
The model parameters can be represented composition of vectors from a vector bank, which is shared across sub-vectors, modules and layers. Right: Architecture of VB-LoRA. We use a top-k softmax function select vectors from the vector bank. The selected vectors are then pooled into sub-vector, which is arranged at desired position, forming the parameters of LoRA.
Figure 2: Left:
Karimi Mahabadi et al. 2021 Ding et al. 2023 These methods exploit the dependencies among model parameters to reduce the redundancy. For example, [Hu al. [2021] propose the low-rank adaptation (LoRA) to approximate the accumulated gradient update for self-attention modules, and induces the intra-matrix parameter coupling. Renduchintala et al. [024] further study the options of allowing the inter-matrix parameter sharing via weight tying across all the layers. In both cases, the number of trainable parameters is reduced significantly. These two methods stand the two extremes of spectrum in deciding the range of model components reuse (locally or across-layers) and designating which low-rank matrices needs to be shared and updated. However, as model size increases and the demand for user-customized models across various services rises, the expense of storing and transmitting the customizations
each combination escalates and emerges as a critical issue. Hence, investigating PEFT methods with significantly smaller number of trainable parameters has attracted a flurry of research interests [Kopickko et al. 2024 Renduchintala et al. 2024).
This paper introduces VB-LoRA, extreme parameter-efficient fine- tuning with vector banks based on a simple yet effective "divide-and-shar" paradigm. We push the limits of LoRA parameter efficiency by breaking the two barriers of low-rank decomposition: (1) locally within each module and each layer, and (2) only across the two original matrix dimensions (without division; see Sec. [3.2] for details). We argue that the parameters across different modules and layers can be shared, and thus the redundancy parameters can further reduced. In addition, by partitioning rank-one component vectors into sub-vectors, we introduce "virtual" dimensions such that deep structure in the parameter space can be represented by a highly compressed matrix factorization.
VB-LoRA draws inspirations from previous line of work on quantized tensor networks (Oseledets 2010 Cichocki] 2014] in breaking the constraint of physical dimension for extreme parameter compression. Specifically, VB-LoRA reparameterizes LoRA's low-rank adaptation by rank-one decomposition and then divides the resulting vectors into sub-vectors the same size. A global sharing mechanism is then learnt based on sparse top-k admixture module. The same sized sub- vectors allows parameters to be shared across modules and layers compared to the post-hoc matrix compression methods (Oseledets) 2010, Khoromskij [011], VB- LoRA end-to-end differentiable, and therefore the fine-tuning process is aware the compressed form, enabling task-oriented compression. Figure T iilluttrates the parameter efficiency of VB-LORA as compared with state-of tfate-of-the-att the-art PEFT methods. Our contributions are summarized as follows:
sub-vector level. Moreover,
We introduce a 'ddivde--and-share" paradigm that breaks the barriers of low-rank decomposition across matrix dimensions, modules, and layers by sharing parameters globally via vector bank.
2. We reparameterize LoRA's low-rank decomposition by rank-one decomposition, and divide the resulting vectors further into sub-vectors of the same size, enabling extreme parameter efficiency at the sub-vector level.

---

3. We propose a sparse top-k module based on the admixture model to learn global sharing mechanism, making our framework end-to-end differentiable and compression-ae
4. Our method achieves extreme parameter efficiency while maintaining comparable better empirical performance compared to the state-of-the-art PEFT methods on natural language understanding, natural language generation, instruction tuning, and mathematical reasoning tasks.
2 Related Work
Exploit Global Redundancy for Enhanced Parameter Efficiency The parameters of deep neural networks (DNNs) can be naturally divided by layers, heads, or types (MHA FFN). While LoRA (Hu et al. 2021] only exploits the intra-matrix dependency, Tied-LoRA [Renduchintala et al. [024] employs simple weight tying scheme on the low-rank matrices A and B across layers to reduce the inter-matrix redundancy. When A and B are randomly initialized, frozen, and shared across all layers, Tied-LoRA degenerates to VeRA Kopiezko et al., 2024], which only requires two scaling vectors to be updated, leading to impressive parameter efficiency. A concurrent work, LoRA-XS (Balazy et al., 2024), further improves the parameter efficiency of LoRA by introducing small trainable matrices between frozen LoRA projection matrices, which are initialized using Singular Value Decomposition (SVD) of the pretrained module weights. Our VB-LoRA pushes the limits of LoRA parameter efficiency by sharing parameters globally across modules and layers at the sub-vector level.
On the low-dimensional reparameteriization, Aghajanyan et al. [020] empirically show that there exists a low-dimensional reparameterization that is as effective for fine-tuning as the full parameter space. The actualization of the random projection is achieved through the Fastfood transform (Le] et al. 2013 for large-scale pre-trained language models. To make it structure-aware a set of layer- wise scaling parameters are included as part of the training parameters. Following this intuition, study the lightweight fine-tuning within LoRA based on the customized reparameterization that arises from the rank-one matrix decomposition.
Moreover, tensor decomposition has been leveraged for PEFT in ViT models [ie and Deng 2023 based classical formats, such censor-train or Tucker [Kolda and Bader 2009). We find that forcing multilinear decomposition across multiple modes results in a higher rank number, which is detrimental to the objective of parameter compression. An indirect comparison of VB-LORA to Tie and Deng 2023 can be conducted by referring the compression rate to LoRA. From this perspective, VB-LoRA be viewed as customized tensor format endowed with convex geometry structure, which is enabled by the sparse top-k admixture model we proposed.
Compared to the deep fusion approach Mazzawi et al. 2024 where LLM parameters are split and initialized using pre-trained smaller networks under a designed network growth mechanism, our parameter division operates on the rank-one component vectors. Sub-vector division allows for similar extensions to leverage pre-trained vector bank initializations from smaller models and distributed training using model parallelism.
Parameter Modeling based on Sparse Admixture Models Admixture models have been widely
used in population genetics [Pritchard er al.) 2000), topic modeling |Reisinger et al. 2010 Inouye
al. 2014]. and hyperspectral unmixing [L and Bioucas-Dias 2008 Fu al. 2015 extract
archetypal (or endmember) components from observed data. The archetypal components can be relaxed have mixed sign [Ding et al., 2008 with identifiability guarantees [Lin et al.) 2015
Conventionally, parameters estimation are conducted based on linear programming [Chan et al. 2009] or combinatorial algorithms [Arora et al. 2013). However, an involved integer programming problem arises when incorporating an extra top- k constraint into the mixing weights that is especially challenging for the large-scale language models. In this work, we propose learning archetypal vector banks not from observed data but from model parameters of LLMs. By modifying the sparse top-k module [Shazeer et al., 2016) commonly used Mixture-of-Exper models [Jiang et al.) 2024), the mixing weights and vector banks are optimized by back-propagation under the objective of downstream fine-tuning tasks. The proposed top-k admixture model model-agnostic the sense that it can be readily integrated into any neural network parameters accumulated gradient updates.
3

---

3 Proposed Method
3.1 Preliminaries: Transtormer Architecture and LoRA Adapters
The transformer architecture [Vawwani al. 2017 consists of L layers, each containing two types of blocks: Multi-Head Attention (MHA) and Feed-Forward Network (FFN). We denote the query, key, value, and output matrices of MHA at layer & as W; (Wi I E {q, ki 0,0), where W E Rdddd and N₁, is the number of heads. Given (FN( = W Wanaal (Wwu (ua with a E R viewing FFN as a multi-head operation, we further divide W E Roaxd and W down E Rdxed into c [ matrices of size d x d, denoted by W {Wipp Jim1 and W
WW2d downJi=1- c= 4.
down
constrains the weight increments R RMxr A E R are trainable param- AW as a low-rank decomposition AW = -A. where E
Given pre-trained matrix Wo E R LoRA Huu et al.
2021
eters, with < min(m, n.). VeRA [Kopiczko et al. 2024 further limits the trainable parameters to two scaling vectors and d, which form the diagonal elements two diagonal matrices As and Ag. Hence, VeRA can be expressed as AW- A,BA, where B and A are randomly initialized, frozen and shared across layers.
Collectively, we denote the model parameters of transformer (W up- W do E R 12 In the sequel, we propose global reparameterization on the weight increments W E Z based on the LoRA decomposition AW BA. we will show how extreme parameter efficiency can be achieved by (1) parameter sharing across matrix dimensions of and based on a rank-one decomposition and sub-vector partitions (Sec. 3.2), and (2) across modules and layers regardless of the index matrix type (Sec. 3.3)
{(W2 4¹ W W W
3.2 Divide-and-Shaie New Paradigm for Parameter Sharing
The low rank decomposition of LoRA can be equivalently expressed in rank-one form as follows:
AW = BA = Σ by a
k
(5)
(1)
bk
uk
(2)
ak.
(3.1)
where & denotes the outer product operator and 2% (i)
vector of size d..
Divide Based on the rank-one decomposition above, we further represent each component vector % as concatenation of a set of sub-vectors,
concat u k,1 4,2"
(6)
ER', jE (I......,,d'
(3.2)
where (d))=1,2 represents the size of the matrix dimension of AW. In general, 1=122 equal across A and B, and we choose b as a common factor of d, such that de = d and de e Z.
Share To facilitate parameter sharing across model dimensions, we assume each sub-vector uk, as top-k admixture of basic elements from vector bank B -.r,,..., where o; E R& for i E {I,...,h) ,1,, and is defined as follows (with the subscripts omitted for clarity):
u = -'- w.(o)). w(a) Softmax(TopK((r k)),
(3.3)
where TopK (a,, k); o; o; is among the top-k of and TopK (a, k), -o otherwise. For each sub-vector u. introduce logits E R² as its learnable parameters. We call the model expressed in Eq. 3.3 as the top-k admixture module (TKAM), which is differentiable. This design enables the joint learning of vector bank B and logits or in end-to-end manner, which is amenable for model fine-tuning to the downstream tasks.
The TKAM module promotes sparsity by selecting & vectors of the largest logits from the vector bank. By setting & h, we restrict the sub-vector u be sparse. That is, each iteration, the updates to the vector bank remain locally dominated - with at most & basis vectors a E B affected by the backpropagation through u in the hope that the learnt vectors can be more specialized and the knowledge encapsulated in the vector bank can be activated and updated sparsely.

---

Noise-free Top-k module The Noisy Top-k Gating module [Shazeer et al. 2016 has been widely used to replace the fully connected layers with the Mixture of Experts (MoE) layers in large language models [Jiang et al. 2024]. In contrast, we use Eq. 3.3 to learn the selective sharing scheme across the rank-one component vectors without changing the original model. Due to the decomposition, we find that the cumulative gradient parameter updates are more sensitive than the original model parameters during the training process. This may be related to the training instability issues observed in hypernetworks [Ortiz et al. 2024), where parameters are generated by another parameterized model as well. Therefore, keeping zero noise in the gating function can help make the learning more efficient and stable. An ablation study of different vector selection methods, including Gumbel-softmax, is provided in Sec. 4.5
3.3 Breaking Boundaries of LoRA for Global Parameter Sharing
While LoRA only applies the low rank decomposition each individual weight increment, the boundary can be broken by the divide-and-shen scheme we proposed in Sec. 3.2 Our divide- and-share approach can be interpreted as hierarchical and constrained tensor decomposition, which facilitates efficient global parameter sharing that goes beyond LoRA's low-rank representation of matrices.
The divide operator was first introduced in Quantized Tensor Train (QTT) for super compression of large-scale matrices [Oseledets] 2010 Cichocki 2014). For example, dyadic division reshapes a vector length L = 2 into a p-dimensional array which facilitates the efficient Tensor Train decomposition to be used. Our divide operator instead applies to the rank-one component vectors 0k and the resulting hierarchical tensorial representation of AW can be viewed as Canonical Polyadic Decomposition (CPD) Kolda and Bader 2009| with component vectors Uk
(4) folded into 2-dimensional arrays with sub-vectors "k.j as columns. Each sub-vector u4 is composed from a globally shared vector bank B via TKAM, where = [j, v] is multi-index including physical indices j. such as module, layer, head, and left/right decomposed matrix, and virtual indices v (created from vector partition).
(2)
The share operator (TKAM module) be viewed as factor model with simplex constraints on the mixing weight (e.g., k = 2, the sub-vector u lies on the edges of the simplex) and common factors stored in B. Let u E I&' and u En e=1 E W.,W', where a, is the s-th factor, and w is the factor score for the sub-vector u. We consider the following options for w: (1) Admixture (convex
combination): w E [0, 11ah and Σ W. = 1, which is commonly used in various communities. (2) Sparse Admixture (TKAM): E [0, 11ah and Σ.= W, 1 with only k < h non-zero elements allowed. It's worth mentioning that adding the multi-index information to the vector selection mechanism can make the TKAM model structure-aware, potentially yielding additional benefits. One possibility is to make the logits of vector selection conditional on the embeddings of the layer, module, and matrix type, which be implemented through hypernetwork |Mahabadi al.) 2021 However, we leave this for future work.
In summary, LoRA provides local low-rank factorization for each di x x2 matrix A AW independently. In contrast, our VB-LoRA introduces a global low-rank factorization on a x {i]] matrix composed of partitioned rank-one vectors, where {ii denotes the cardinality of the index set including both physical and virtual indices. As we will see below, this differentiation can better leverage the redundancy in the cumulative gradients, leading to extreme parameter efficiency.
Figure 2 overviews our method. The left section demonstrates the high-level idea of VB-LoRA: the vector bank is shared across sub-vectors, modules, and layers. The right section details its architecture. To form each vector,,or, we use a top-k softmax function to select k vectors from the vector bank, which are then pooled into sub-vector. These sub-vectors are arranged in the desired positions, forming the parameters for LoRA with negligible computational overhead. Algorithm | provides the Py -orrch-like pseudocode for VB-LoRA, which can be seamlessly integrated into the PyTorch framework.
3.4 Parameter Count
In full fine-tuning, the number of trainable parameters equal the model size, i.e., LMd where L is the number of layers, M is the number of finc-tuned modules, and d is hidden dimension.
5

---

Algorithm Pseudocode of VB-LoRA in a PyTorch-like style
d: hidden dimension; b: length of sub-vectors; r: rank; h: size of vector bank
# k: number of selected vectors used in the top-k admixture module
# logits: Each linear layer has two trainable parameters: logits_A and logits_B. Both parameters have shape of (d/b)+r*h.
pector_bank: The shared vector bank with shape h*b.
# x and W: input and the original weight.
def get low_low_rrr wow_rank,,eg matrix vector, _ank, x):
topk_logita, topk_indices logits.topkkk, dim=-1) copk_weighta
wof.max matrix = (topk_weights vector_bank bankr_bank[tt bankktook_indicettt s-m(
dim=-1)
return matrix
def @LLoRA_forwar((x vector_bank, logits_A, logits_ k):
r Logits_A.shapes
B weloo_rrnn__atrixxaeeeee
A
vector_ bank, k) transpose (0, 1) (rehhape -1)
vector_bank, k) transpose (1, 2) reeshape (-1, r)
For memory efficiency, avoid explictly computing \delta i
return x a W (x
LoRA reduces this number 2LMdr, while VeRA further reduces it to LM + r). The trainable parameters of LoRA and VeRA are the same as the parameters they need to store.
In VB-LoRA, the trainable parameters consist of two parts: the parameters of the vector bank B and the parameters of logits or. However, at the end of training. the logit parameters can be discarded and only the k selected indices and the top-k admixture weights need to be stored. Therefore, the stored parameters can be represented by a triplet = {B, I, V). where B E igxb is a vector bank containing à vectors of O-dimensional, in B for all sub-vectons and composite the sub-vectors from the bank. It is worth noting that the top-k admixture weights have only k degrees of freedom since they must be summed to Additionally, depending on the size of the vector bank h. the indices I can be efficiently stored unsigned integers (e.g., uint8 when < 256), and hence, we count the number of parameters as the float32-cquivalem size for fair comparison. When we use k = 2 and uint8 for indices, the number of stored parameters of VB-LoRA is hb + 3LMr(d/b). Unlike LoRA and VeRA, the number of parameters VB-LoRA does not increase linearly with the model size (determined by L and d) the number of fine-tuned modules, i.e., M. While the second term VB-LoRA's parameters a linear function LMd, the coefficient 3r /b, which is typically very small. For example, in experiments, the typical values are r = and = 256, leading to a coefficient of 0.04, whereas the coefficient 2r for LoRA and for VeRA. Most of the parameters in VB-LoRA reside within the shared vector bank, whose size does not increase linearly with the model size or number fine-tuned modules.
(2x is the top-k indices of the vectors is the top-k admixture weights used to
4 Experiments
In this section, we conduct comprehensive evaluation of our method through a series of experiments. We begin by comparing VB- to the state-of-the-ar PEFT methods: LoRA, VeRA, and Tied- LoRA on the GLUE benchmark. Next, we extend our analysis to natural language generation tasks using GPT-2, instruction tuning tasks the Llama2, well as mathematical reasoning tasks Mistral and Gemma models. All our experiments were conducted on a server equipped with NVIDIA A100 GPUs. For reproducibility, provide detailed hyperparameters and specifications of computing resources for each experiment in the appendix. The source code is available at http:///gtttubb..cmmeeeeaaaaa///
4.1 Natural Language Understanding
We adopt the General Language Understanding Evaluation (GLUE) benchmari Wang al. 2018
to assess the performance of VB-LoRA across various natural language understanding tasks, inc including
https://gllebbnncmark.coom

---

Table 1: Results with RoBERTa and RoBERTatrge on the GLUE benchmark. The best results in
each group are shown in bold. We report Matthew's ' correlation for CoLA, Pearson correlation for STS-B, and aceuracy for all other datasets. Results for AgR and VeR are sourced from their respective original papers, while the other results are based on our implementations. We report the median performance from runs using different random seeds.
Method
Params
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg.
FT
125M
94.8
90.2
63.6
92.8
78.7
91.2
85.2
LoRAge
0.295M
95.1
89.720.7
63.421.2
93.320-3
86.620.7
91.520.2
86.6
VeRAg
0.043M
94.6.1.1
89,5
65.6_0.
91.8102
78.7.0.7
90.7.10.
85.2
BASE
Tied-LoRAq
0.043M
94,4400.5
88.5
61.9-1.
92.0+0.1
76.2=110
89.8
83.8
VB-LoRApp (Ours)
0.023M
94,420.2
89.5g0.5
63.3
92.2g0.2
82.321.3
90.8
85.4
VeRA,1
0.157M
95.1+0.4
88.7+0.s
64.5=1.0
92.3+0.2
81.9=14
90.2
85.5
Tied-LoRAai
0.109M
94.7:1..2
88.540.8
64.7 =0.7
92.4
76.5+1.
90.3
84.5
VB-LoRAn (Ours)
0.027M
95.0+0.2
89.7+0.2
64.3-1.4
92.3+0.2
82.3+0.9
90.7
85.7
LoRA,
0.786M
96.2+0.5
90.2
68.2-1.9
94.820.8
85.2
92.3
87.8
VeRAg
0.061M
96.1
90.920.7
68.0_o.8
94.420.2
85.920.7
91.220.8
87.8
LARGE
Tied-LoRAq
0.066M
94.810.
89.7a1.0
64.7=1.2
94.110.1
81.2 10.1
90.8 10.3
85.9
VB-LoRAgr (Ours)
0.024M
96.1i0.2
91.440.6
68.3
94.7
86.621.
91.8
88.2
VeRAui
0,258M
96.6+0.5
90.9+o.s
68.5
94.4
85,9=1.2
92.2++022
88.1
Tied-LoRAal
0.239M
94.8+0..
90.0+0.4
66.8
94.1
82.3+2.0
91.6+0.2
86.6
VB-LoRAm (Ours)
0.033M
96.310.2
91.9
69.3
94.4
87.4
91.8
88.5
similarity, paraphrase, and inference tasks. Following Kopiczko et al. [2024] we focus on six tasks
from GLUE:: CoLA (Warstadt et al., [019] (linguistic acceptability), SST-2 (Socher et al. 2013) (sentiment analysis), MRPC [Dolan and Brockett [2005] (paraphrase detection), STS-B [Cer et al. 2017) (semantic textual similarity), QNLI [Rajpurkar et al. 2018) (inference), and RTE (inference).
Our experiments are performed with RoBERTapws and RoBERTa [Liu et al. 2019 While LoRA and VeRA only finetune the query and value modules, we explore two finettuning strategies: query and value only (VB-LoRAge), and all linear modules (V (VB-LoRAaI). including W Wk. W., W. Wap. and W down- We create a vector bank of 90 vectors of length of 256, initialized with a uniform distribution U 4(-0.02, 0.02). The logits are initialized with a normal distribution (0, 0.01). The learning rates for the vector bank and logit parameters are set 0.001 and 0.01, respectively. We set the rank to 4 and h = 2 for all our experiments.
Table D reveals that VB-LoRA achieves competitive or superior performance compared to VeRA and Tied-LoRA, while being more parameter efficient. For example, when fine-tuning the query and value modules on the RoBERTaaage model, our method reduces the stored parameters to less than 40% of those required by VeRA or Tied-LoRA, while outperforming them across all tasks. These results suggest that model performance depends not only on the quantity of trainable parameters but also on how they are composed.
Moreover, the results consistently indicate that fine-tuning all modules, beyond just the query and value modules, enhances performance for all the methods. However, LoRA, VeRA and Tied-LoRA requires 2-4 times of the parameters in this case because their parameter counts increase linearly with the number of fine-tuned modules. In contrast, our method uses only 37.5% additional parameters as we maintain the same vector bank size but add additional parameters for indices and top-k weights. Thus, with only 12.8% of the parameters compared to VeRA (4% compared to LoRAgu), our method achieves the best average performance.
4.2 Natural Language Generation
For natural language generation experiments, we fine-tune the GPT-2 Medium and Large mod- els [Radford al. 2019) on the E2E d Novikova al. 2017 which contains approximately 42,000 training examples, 4,600 validation examples, and 4,600 test examples from the restaurant domain. We use vector bank of size 256 for GPT-2 Medium and 350 for GPT-2 Large. The vector length is set to 256 and the rank is set to 4 for both models. To achieve the best performance, we fine-tune all attention layers and FFN layers. As shown in Table our approach achieves competitive performance compared to VeRA, while requiring about 20% less stored parameters for both models.
"Licensed under CC BY-SA 4.0. URL: https: http:://gggattteo//////////caaaa
7

---

Table Results with GPT-2 Medium and GPT-2 Large on the E2E benchmark. The results for FT and LoRA are taken from Hu et al. [202T], and the results for VeRA are taken from Kopiczko et al. We report the mean of 3 runs using different random seeds.
2024
Method
# Params
BLEU
NIST
METEOR
ROUGE-L
CIDEr
MEDIUM
FT
354.92M
68.2
8.62
46.2
71.0
2.47
LoRA
0.35M
68.9
8.69
46.4
71.3
2.51
VeRA
0.098M
70.1
8.81
46.6
71.5
2.50
VB LoRA (Ours)
0.076M
70.0
8.81
46.6
71.5
2.52
FT
774.03M
68.5
8.78
46.0
69.9
2.45
LARGE
LoRA
0.77M
70.1
8.80
46.7
71.9
2.52
VeRA
0.17M
70.3
8.85
46.9
71.6
2.54
VB-LoRA (Ours)
0.13M
70.3
8.86
46.7
72.2
2.54
4.3 Instruction Tuning
Instruction tuming is process of fine-tuning model with set of instructions or prompts to enhance its performance on specific instructions [Ouyang et al. [2022] We first experiment on a general instruction tuning dateset. We use the Cleaned Alpaca Dataset which improves the data quality of the original Alpaca dataset [Taori et al. 2023]. We evaluate the fine-tuned models on the MT. Bench Zheng et al. 2024). which contains 80 multi-turn questions.
Following Kopiczko al. 2024 we fine-tune the Llama2 model [Touvron et al. 2023 within
the QLoRA [Dettmers et al. 2023 framework which aims to reduce memory usage when fine- tniin large language models on single GPU. We utilize the quantization strategy provided by QLoRA, including 4-bit NormalFloat for storage data, BF for computation parameters, double quantization and paged optimizers to train on a single GPU. Our fine-tuned models generate responses to these questions, and subsequently, GPT-4 is employed to review and evaluate the generated answers, assigning a quantitative score on a scale of 10. Note that aligning with VeRA, report the score of the first turn of the conversation. Following Kopiczko et al. [2024]. apply VB-LoRA to all linear layers except the top one. For Llama2 7B, we use a vector bank of 2,048 vectors, each with length of 256, and the rank is set 4, resulting in a total of 0.8M stored parameters. For Llama2 13B, we use the same-sized vector bank but increase the rank to 6, leading 1.IM stored parameters. For all the experiments, train for epoch.
The results are reported in Table Notably, we report two sets of LoRA results for each experi- ment: one from our implementation and the other from Kopiczko et al. 2024), due to noticeable discrepancy between the scores. Since we closely follow the experimental settings of Kopiczko et al.) 2024]. speculate that the difference is due to changes in the GPT-4 model over time. However, comparing the relative improvements of VeRA and VB-LoRA with their respective implementations of LoRA remains fair. VB-LoRA achieves higher scores than LoRA while using only 0.5% (Llama2 7B) and 0.4% (Llama2 13B) of the stored parameters. While VeRA can reach similar scores with their implementation of LoRA, it requires more than twice of parameters compared to VB-LoRA.
4.4 Mathematical Reasoning
To evaluate mathematical reasoning capabilities, we munetune the Mistral-7B- -0.1 and Gemma- 7B models the MetaMathOA [| Yu et al. 2023 dataset and test them GSM8K Cobbe al. 2021 and MATH 10 HHendrycks et al. 2021 datasets. We compare our results with the concurrent work
LoRA-XS
Batazy [0224 following its experimental configuration. The result is shown in
Table Our method outperforms all baselines on GSM8K, with Mistral-7B utilizing only 0.4% of
'The original and cleaned Alpaca datasets are licensed under CC BY-NC 4.0. URLs: https://huggingtte htt///aaaett/ttttaalaaa///
https://huggingfacee cttei/aacec.o///aaaaeeeeeeeeeeee co/datasess
"Licensed under CC BY 4.0. URL: https: http/////wc///oooot.c/////////ss
judmeets
attps://githin com/ htt.//atiioo/////oooo..
Licensed under MIT. http://///ps.hg.ooe....////////a
9 Licensed under MIT. URL: htgpaa/eiooaoo..o..o/////ooona/n htt.hp/..//ngggggee.....//////ep
10 "iceensed under MIT. URL: http://p://ggghhibom////t/eree//

---

Table 3: Results with Llama2 on MT-Bench, scored by GPT-4 out of 10. LoRAi and VeRA are sourced from Kopiczko et al.] [2024]. imple-
Table 4: Results with Mistral-7B and Gemma-78 models on the GSM8K and MATH Benchmarks. Specifically, in VB-LoRA, we use vector bank size of 2,048 with b = 256, set the rank to 4, and train with a batch size of 128 for epochs. The warm-up ratio is 0.02, and training uses cosine learning rate scheduler, with an initial learning rate of 0.001 for the vector bank and 0.01 for the logits. The bascline
LoRA3 and VB-LoRA are from mentations. The discrepancy between LoRA and LoRA+ may be due to changes in the GPT-
4 model over time.
Model
Method
Parameters
Score
w/o
4.79
results are taken from Batazy et al. 2024
LoRA
159.9M
5.19
LLAMA2
VeRA
1.6M
5.08
Model
Method
Parameters
GSM&K
MATH
LORA
159.9M
5.63
VB-LORA (Ours)
0.8M
5,71
Full-FT
LoRA
724IM
168M
67.02
67.70
19.68
18.60
w/o FT
5.38
MISTRAL-7B
LaRA-XS
0.92M
68.01
17.86
VR-LaRA (Ours)
0.65M
69.22
17:90
LoRA
250.3M
5.77
LLAMA2 13B
VeRA
2.4M
5,93
Full-FT
8538M
71.34
22.74
LoRA
200M
74.90
31.28
LoRA"
250.3M
6.13
GEMMA-7B
LoRA-XS
0.80M
27.62
VB-LoRA (Ours)
1.1M
6.31
VB-LoRA (Ourz)
0.67M
75.96
28.90
the parameters compared to LoRA, and Gemma-7B using just 0.3%. Compared with LoRA-XS, our method outperforms on both evaluation datasets while using 70% (Mistral-7BB and 83% (Gemma-7B) of LoRA-XS parameters.
4.5 Ablation Study
We conduct an ablation study to examine the impact of each individual component of VB-LoRA. The experiments are performed RoBERTa-large, fine-tuning only the query and value modules.
Vector Selection Methods Besides the top-k admixture module (abbreviated Top-k below), there exist several commonly used discrete optimization methods for vector selection, including Noisy Top-k [Shazeer al. 2016). Gumbel-Softmax (GS), and Straight-Through Gunbel-Softmax (Jang ctt al. 2017 Maddison et al. 2016). For Top-k and Noisy Top-k, we evaluate the impact of different to the performances on the CoLA dataset. For GS and Straight-Through GS, we set the temperature T 1/3 during training and use Top-1 and Top-2 Softmax for inference. Additionally, we explore "Select All", special case of Top-k with i equals the vector bank size h. As shown in Table Noisy Top-k. GS, and Straigh-TThrouge GS significantly underperform Top-k and "Select All". We hypothesize that random noise injected by these methods likely disrupts the parameters of vector bank, leading to instability in the learning process.
We further investigate the impact of l to the training dynamics and performance of VB-LoRA. As discussed Sec. 3.4 the choice of l affects not only the model's performance but also the number of parameters be stored. Hence, smaller & is generally preferred for improved parameter efficiency. Table shows that & = 2 yields the best result on CoLA, whereas A = 1 performs significantly worse. To explain this, we delve into the training dynamics of VB-LoRA. As shown in 3 when k == 1, the selected vectors remain largely unchanged during training. In contrast, when k > 1, the model actively explore the vector bank as illustrated in Figure (b) and (c), i.e., different vectors are selected and updated actively during the training process. Additionally, we observed that this vector exploration primarily occurs in the early stages of training, with updates becoming progressively sparser in later stages, as shown in Figure in the appendix. This suggests that the vectors become increasingly specialized for specific sub-vectors as training progresses.
Sub-vector Length b VB-LORA introduces new virtual dimension that divides the original dimensions of LoRA matrices into sub-vectors of length b. Note that b must be common factor of all hidden dimensions to ensure compatibility across the entire model. However, the optimal value of b is task-specifie and requires tuning as hyperparameter. Theoretically, with a fixed vector bank budget, larger reduces the number of vectors in the vector bank, potentially making each vector less specialized. On the other hand, smaller bs increases the number of trainable parameters and complicates the vector selection process. As shown in Table a moderate b 256 yields the best performance on the CoLA task.

---

Table 5: Ablation study of different vector selec- tion methods. S.: Softmax, GS Gumbel-Softmant
13
23
ST-GS: Straight Through Gumbel-Softmax.
Method
Training
Inference
CoLA
Select All
S.
67.5
Top 1 S.
Top S. 66.920.5
65
60
Top 2
Top 2 S.
68.3
Top-k
(a) When À:
(b) When & =
Top 3 S.
Top 3 S.
68. 68.1
Top 6 S.
6.
67.1
6:0-
Noisy Top 1 S.
Top 1 S.
45.3
20
200
Noisy Top-k
Noisy Top 2 S.
Top S.
62.610.2
GS (r=1/3)
Top S. 57.1
200
GS
GS (r=1/3)
Top 2 S. 57.3
ST-GS (r=1/3)
Top 1 S.
55.611.6
ST-GS
20
40
55
22
45
00
ST-GS (r=1/3)
Top 2 S. 54.7
(c) When l: =
(d) Noisy Top-2
Table 6: Ablation study of subtvector length.
Figure 3: VB-LoRA's vector selection foot- prints during training. The x-axis represents the 96 sub-vectors formed by the vectors from bank of 90 vectors, while the y-axis repre- sents the indices of selected vectors from the bank. The blue blocks indicate the selection footprint during training.
Length
Vector Bank Size
CoLA
128
240
67.0=a.8
256
120
68.7
512
60
67.8
1024
30
67.3a111
5
Conclusion
This paper introduces a "divide-and-shee paradigm and a differentiable top-k admixture module for extreme parametter-efficient fine-tuning with vector banks. Our proposed VB-LoRA achieves the competitive or higher accuracy while using significantly smaller number of stored parameters compared to the state-of-the-arr PEFT methods, including LoRA, VeRA, and Tied-LoRA. In addition,
VB-LoRA is model-agnostic and applicable to other PEFT methods (Ding et al.) 2023 including inserted adapters [Karimi Mahabadi al. 2021], prompt tuning [Qin et al. 2021], and BitFit (Ben Za
ken et al. 2022]. Although VB-LoRA focuses on reducing the storage and transmission costs for LLM tuninn, we believe the proposed scheme be extended to memory-efficient fine-tuning and parameter-efficient pre-training. We leave these for future exploration.
Fine-tuning pre-tree pre-trained model requires making design choices about which layers of the model should be frozen or updated. Multitask fine-tuning adds extra complexity about which parameters should be shared or task-specific. Along this line of work, Polytropon (Ponti et al. 2022 jointly learns a small inventory of LoRA adapters and a routing function that selects a variable- sized subset of adapters for few-shot adaptation, [Ccciia et al. (2023) emphasize the importance of routing granularity and further propose a finer- grrined mixing across multiple heads. Following these works, it would be interesting to explore finer-graie -rainng parameter transfer across tasks, heads, types, and layers the sub-vector level for multitask fine-tuning.
I Limitations and broader impacts Our experiments are limited to monomodal (text-based), monolin- gual (English), and LoRA-only settings. Additionally, our exploration of the vector bank is somewhat limited, as only examine a small range of configurations for bank size and vector length. In terms of broader impacts, VB-LoRA reduces the storage and transmission costs of LLM adapters and demonstrates improved memory vremory-efficiency, making customized LLMs more accessible. We do not foresee any negative societal impact beyond those generally associated with LLMs.
Acknowledgments
We would like to thank the anonymous reviewers for their comments and suggestions, which helped improve the quality of this paper.
10

---

References
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness language model fine-tuning, arXiv preprint arXiv:2012.13255, 2020.
Sanjeev Arora, Rong Ge, Yonatan Halpern, David Mimno, Ankur Moitra, David Sontag, Yichen Wu, and Michael Zhu. A practical algorithm for topic modeling with provable guarantees. In International Conference on Machine Learning, pages 280-288. PMLR, 2013.
Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. LoRA-XS: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405. 2024.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine- Smaranda Muresan, Preslav Nakov,
tuning for transformer-basee masked language-modes and Aline Villavicencio, editors, Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1 1-9, Dublin, Ireland, May 2022.
Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin A Raffel. Distributed inference and fine-tuning of large language models over the internet. Advances in Neural Information Processing Systems, 36, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni. Multi-head adapter routing for cross-task generalization. In Advances in Neural Information Processing Systems, 2023.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens, editors, Proceedings of the 11th International Workshop Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada, August 2017.
Tsung- Han Chan, Chong- Chong-Yung Chi, Yu-Min Huang, and Wing-Kin Ma. A convex analysis-based minimum-volume enclosing simplex algorithm for hyperspectral unmixing. IEEE Transactions on Signal Processing. 57(11))4418 4432, 2009.
Andrzej Cichocki. Era of big data processing: A new approach via tensor networks and tensor decompositions. arXiv preprint arXiv:/40.3 2048, 2014.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14.1668 2021.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 10088- 10115, 2023.
Chris HQ Ding, Tao Li, and Michael I Jordan. Convex and neni-nonnegative matrix factorizations. IEEE transactions pattern analysis and machine intelligence, 32(1):45-55 2008
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-233. 2023.
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop Paraphrasing (ZWP2005), 2005.
11

---

Xiao Fu, Wing-Kin Kin Ma. Kejun Huang, and Nicholas D Sidiropoulos. Blind separation of quasi- stationary sources: Exploiting convex geometry in covariance domain. IEEE Transactions on Signal Processing, 63(9):2306-23200 2015.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrickh and Graham Neubig. Towards unified view of parameter-efficien transfer learning. In International Conference on Learning Representations, 2021.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pages 2790-2799 PMLR, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
David Inouye, Pradeep Ravikumar, and Inderjit Dhillon. Admixture of Poisson MRFs: topic model with word dependencies. In International Conference on Machine Learning, pages 683-691. PMLR, 2014.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrizzaioon with Gumble-softmax. In International Conference on Learning Representations, 2017.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.04088, 2024.
Shibo Jie and Zhi-Hong Deng. Fact: Pactor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1060-1068, 2023.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 14:1022-, 2021.
Boris N Khoromskij. O (d log n)-quantics approximation of n-d tensors in high-dimensional numerical modeling. Constructive Approximation, 34:257-288 2011.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51()): 455-500, 2009.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In International Conference on Learning Representations, 2024.
Quoe Le, Tamás Sar1ós, Alex Smola, et al. Fastfood-approximmatng kernel expansions in loglinear time. In International Conference on Machine Learning, volume 85, 2013.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
Jun Li and José M Bioucas-Dias Minimum volume simplex analysis: A fast algorithm to unmix hyperspectral data. In IGARSS 2008-2008 IEEE International Geoscience and Remote Sensing Symposium, volume 3. pages III-250 IEEE, 2008
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582-4597, 2021.
12

---

Chia-Hsiame Lin, Wing-Kin Kin Ma, Wei-Chiang Li, Chong Yung Chi, and ArulMurugan Ambikapathi. Identifiability of the simplex volume minimization criterion for blind hyperspectral unmixing: The no-pure-pixee case. IEEE Transactions on Geoscience and Remote Sensing, 53(10):5530-554 2015.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-effiient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35: 15:1990-1965, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprini arXiv: /907. 1692, 2019.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, 2016.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypemeeworkss. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 565-576, 2021.
Hanna Mazzawi, Javier Gonzalvo, Michael Wunder, Sammy Jerome, and Benoit Dherin. Deep fusion: Efficient network training via pre-trate pre-trained initializations. In Forty-first International Conference on Machine Learning, 2024.
Jekaterina Novikova, Ondiej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to- end generation. In Kristinna Jokinen, Manfred Stede, David Vault, and Annie Louis, editors, Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 201-206, Saarbrücken, Germany, August 2017.
Jose Javier Gonzalez Ortiz, John Guttag, and Adrian Dalca. Magnitude invariant parametriza- tions improve hypernetwork learning. In The Twelfth International Conference on Learning Representations, 2024.
Ivan V Oseledets.
Approximation of 2d x 2d matrices using tensor decomposition. SIAM Journal on Matrix Analysis and Applications, 31(4):2130-211 2010.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Edoardo M Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining modular skills in multitask learning. arXiv preprint arXiv:2202. 13914, 2022.
Jonathan K Pritchard, Matthew Stephens, and Peter Donnelly. Inference of population structure using multilocus genotype data. Genetics, 155(2):945-95) 959, 2000.
Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, al. Exploring universal intrinsic task subspace via prompt tuning. arXiv preprint arv: /0.07867, 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 784. 789, Melbourne, Australia, July 2018
Joseph Reisinger, Austin Waters, Bryan Silverthorn, and Raymond J Mooney. Spherical topic models. In International Conference on Machine Learning, pages 903-910. Citeseer, 2010.
13

---

Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-LoRA: Enhancing parameter efficiency of LoRA with weight tying. In Proceedings of the 2024 Conference of the North Ameri- can Chupter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pages
2024.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gae -ggttdd mixture-of-experts layer. In International Conference on Learning Representations, 2016.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, al. S-LoRA: Serving thousands of concurrent LoRA adapters. arXiv preprint arXiv:231 1.03111.0255 2023.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA, October 2013.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. Stanford alpaca: An instruction -orllottoonfolooiingg Ilama model. https://github www.tatsuulaa///ccooocaaooaaaaet wttpaaaassaafffdddaaaaaaaa alpaca 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288 2023.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9866) 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Information Processing Systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, November 2018
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. /2284, 2023.
Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Li Yun, Hejie Cui, Zhang Xuchao, Tianjiao Zhao, et al. Domain specialization as the key to make large language models disruptive: A comprehensive survey. arXiv preprint 2857:23305.18703, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judde with MT-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
14

---

A Appendix
A.1 Hyperparameters and Computing Resources
The hyperparameters used for the natural language understanding, natural language generation and instruction tuning are provided in Table 7 8 and All experiments were conducted server equipped with NVIDIA A100 80GB GPUs.
Computation overhead The proposed factorization in VB-LoRA A is simple implement in modern deep learning frameworks such as PyTorch, allowing us fully leverage GPU acceleration. However, the use of subvector decomposition does introduce some computational overhead. This additional overhead is limited to the training phase and does not affect inference, as both LoRA A and VB-LoRA merge their parameters back into the original model parameters during this stage.
Memory efficiency Despite the training time overhead, the reduced number of trainable parameters in VB-LoRA results in lower memory consumption. During LoRA fine-tuning, the forward pass Ax. H Bz, without the need to materialize AW. This memory-saving technique can be seamlessly incorporated in VB-LoRA and has been implemented in our source code. Table shows that VB-LoRA requires approximately 15%-20% more training time than LoRA, while it consumes less memory than LoRA in both the LLaMA2 7B model and LLaMA2 13B models.
Table Hyperparameters and computing resources for natural language understanding experiments on the GLUE benchmark. Training time and GPU memory are reported as "query and value only" "all linear modules". h: hour, m: minute.
Model
Hyperparameter
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Optimizer
Adam
Warmup Ratio
0.06
LR Schedule
Linear
Init. of the Vector Bank
24( (0000 0.02)
Init. of the Logits
(N 0.01)
GPUs
Epochs
60
30
25
160
Learning Rate (Head)
4E-3
4E-3
2E-2
1E-2
2B-2
2E-2
Learning Rate (Logits)
IE-2
Learning Rate (Vector Bank)
IE-3
BASE
Vector Bank Size
90
Vector Length Rank
256
Max Seq. Len.
$12
Batch Size Per GPU
32
Training Time
8h/ 10h
27m 40m
80m 100m
5h 6.5h
50m /
1h 80m
GPU Memory
24,552 MiB / 28,120 MiB
GPUs
1
Epochs
20
40
40
20
40
40
Learning Rate (Head)
3E-3
3E-3
3E-3
2E-3
2B-3
6E-3
Learning Rate (Logits)
1E-2
Learning Rate (Vector Bank)
IE-3
LARGE
Vector Bank Size
90
Vector Length
256
Rank
4
Max Seq. Len.
128
Batch Size Per GPU
32
Training Time
/
12m
30m 45m
3h 4.5h
10m
20m 30m
GPU Memory
MiB
12,170 MiB
A.2 Visualization of the Vector Selection
For visualization, we conducted experiments the CoLA dataset using 24-layer RoBERTa-large model with vector bank of 30 vectors. We fine-tuned the query and value modules, setting the rank to and the vector length to 1024, resulting in 192 sub-vectors.
15

---

Table Hyperparameters and computing resourees on natural language generation experiments on the E2E dataset. Training time and GPU memory are reported as "query and value only" / "all linear modules". hour, m: minute.
Hyperparameter
Medium
Large
# GPUs
Optimizer
AdamW Linear
Learning Rate Schedule
Weight Decay
0.01
Batch Size
Epechs
5
Warmup Steps
500
Label Smooth
Rank
Vector Length
256
Vector Bank Size
256
350
Learning Rate (Vector Bank)
1E-3
IE-3
Learning Rate (Logits)
1E-2
1E-2
Training Time
3h
3h
GPU Memory
29,061 MiB
29,282 MiB
Table 9: Hyperparameters and computing resources on instruction tuning on the Cleaned Alpaca
Dataset. h: hour. 7B: Llama2 7B, 13B: Llama2 13B.
Hyperparameter
LoRA. 7B LoRA,
VB-LoRA, VB-LoRA, 13B
GPUs
Optimizer
AdamW
Warmup Ratio
0.1
Batch Size
4
Accumulation Steps
4
Epochs
LR Schedule
Linear
Vector Length
N/A
N/A
256
256
Rank
64
64
4
6
Vector Bank Size
N/A
N/A
2048
2048
Learning Rate (Vector bank)
N/A
1E-3
1E-3
Learning Rate (Logits)
N/A
N/A
IE-2
IE-2
Learning Rate (LoRA)
4e-4
4e-4
N/A
N/A
Training Time
2h
2.6h
2.5h
3h
GPU Memory
8,467 MiB
11,624 MiB
6,872 MiB
11,486 MiB
Figure displays the vectors selected by sub-vectors at the initialization (red) and at the end of training (blue), respectively. As we can see, most of the final selections differ from the initial selections, demonstrating the training dynamics of the vector selection process.
In Figure 5 plot the footprint at different training periods. This visualization demonstrates that vector exploration predominantly occurs the early stages of training, and the updates become progressively sparser in the later stages of training.
(igurr/Jillusraaal the sum of the top-k weights for each vector, grouped by the first, middle, and last 8 layers. It shows that certain vectors are favored by deeper layers, such vectors #1 and #29, while some are favored by shallower layers, such as vectors #20 and #26.
We then group the same data with respect to query and value modules, well as matrices A and B, shown in Figure 7 As can see, some vectors are predominantly utilized by specific module or matrix types. For instance, vector #23 is heavily utilized in the formation of matrix A, while vector #29 is predominantly used the formation of Query modules.
Load balancing To demonstrate that the vector selection free from load balancing issue, we present the vector usage in a Gemma-7B model trained the MetaMathQA dataset, shown in
16

---

Initially selected Hinally selected Overlapped
100
125
175
Sub vecter
Figure 4: The x-axis represents the 192 sub-vectors formed by the vectors in the vector bank, while the y-axis represents the 30 vectors in the vector bank. The vectors initially selected by each sub-vector shown in red, the vectors finally selected shown in blue, and the overlapping vectors shown in green.
Vector the vector bank
Vector # the vector bank
Vector in the vector bank
20
40
40
40
60
50
25
75
50
vebtor # (a) Epoch 0-4
Sub-vector #
Sub-vector # (c) Epoch 10-14
(b) Epoch 5-9
Vector in the vector bank
Vector # in the vector bank
Vector in the vector bank
20
20
60
60
60
80
80
80
25
50
75
25
50
75
25
50
75
Sub-vector
Sub-vector (e) Epoch 20-24
Sub-vector Epoch 25-29
Epoch 15-19
Figure VB-LoRA's vector selection footprints during training, The x-axis represents the 96 sub-vectors formed by the vectors from bank of 90 vectors, while the y-axis represents the indices of selected vectors from the bank. The blue blocks indicate the selection footprint during training.
Figure 8 The vector bank contains 2048 vectors. The distribution of vector usage follows roughly normal distribution, with most vectors being selected between 40 to 55 times.
A.3
Visualization of the Vector Bank and the Sub-vectors
Figure illustrates the positioning of the sub- vectorstors along the edge of the simplex spanned by the vector bank. The vector bank is projected into a 2-D space using T-SNE [van der Maaten and Hinton 2008 for visualization.
17

---

First layers Middle B layers
Sum of the top-k weights
6-
5-
3-
Vector
Figure 6: The sum of the top-k weights for each vector, grouped by the first, middle, and last 8 layers. The vectors in B are sorted their norms.
V2
Sum the top-k weights
6-
Vector
Figure 7: The sum of the top-k weights for each vector, grouped by query (Q) and value (V) modules, and matrices A and B. The vectors in B are sorted by their norms.
18

---

100
Number of vectors
80
60
40
20
30
60
70
Frequency usage
Figure 8: Histogram of vector usage frequency. The frequency ranges from 29 to 70, with most vectors being selected between 40 and 55 times. The distribution vector usage follows an approximately normal pattern.
19

---

Vector Bank Sub-vectors
Figure 9: Visualization of the learned vector bank and sub-vectors.
20

---

A.4 Instruction-tuning Examples
Tables I0 and II provide examples of the responses generated by the finetuned Llama2 13B model with LoRA and VB-LoRA.
Table 10: Example #1 of the responses generated by the finetuned Llama2 13B model with LoRA and VB-LoRA.
LoRA
VB-LoRA
Draft professional email seeking your supervisor's 's feedback the 'Quarterly Financial Report' you prepared. Ask specifically about the data analysis, presentation style, and the clarity of conclusions drawn. Keep the email short and to the point.
ANSWER
Dear [Supervisor's Name],
Dear Supervisor.
am writing to request your feedback on the quarterly financial report prepared. I would appreciate your input on the data analysis, pre- sentation style, and the clarity the conclusions drawn.
I am writing to request your feedback on the 'Quarterly Financial Report' prepared. Specifi- cally, would like to know your thoughts on the data analysis, presentation style, and clarity of conclusions drawn.
have attached the report for your review. Please let know if you require any additional infor- mation.
As reminder, used the most recent data avail- able analyze the company's financial perfor- mance provide insights into its overall health. I used charts and graphs to present the data in an easy-to- to-understand manner. Finally. I summa- rized the key findings and provided recommen- dations for future action.
Thank you for taking the time to review my work. am eager to receive your feedback and learn from your insights.
look forward to your feedback and appreciate your time and consideration. Thank you.
Your Name
REVIEW
The assistant's response very professional, concise, and the point. clearly communi- cates the user's request for feedhack on specific aspects of the report, namely the data analysis, presentation style, and clarity of conclusions. The assistant also shows respect for the supervi- sor's time expresses a willingness to learn from their feedback. The response is relevant, accurate, and detailed enough for the context. However, it lacks bit of creativity, but given the professional context, this is not major is-
The assistant's response highly relevant, rate, and helpful. It provides professional and concise email draft that meets the user's require- ments. The assistant has included all the neces- sary elements such as the specific areas where feedback is needed (data analysis, presentation style, and clarity of conclusions). brief descrip- tion the work done, and a polite request the supervisor's 's time and feedback. The response is also creative and detailed, providing clear context for the supervisor. Therefore, I would rate highly.
sue.
SCORE
10
21

---

Table 11: Example #2 of the responses generated by the finetuned Llama2 13B model with LoRA and VB-LoRA. Parts of answers are omitted due to their length.
LoRA
VB-LoRA
Compose engaging travel blog post about recent trip to Hawaii, highlighting cultural
experiences and must-see attractions.
ANSWER
Sure! Here is travel blog post about recent trip to Hawaii:
Dear reader, I recently had the opportunity to travel to the beautiful island of Hawaii, and am thrilled to share my experience with you. This trip was full of cultural immersion, adventure, and breathtaking scenery. Here is glimpse of my journey:
Hawaii is paradise on earth, with its heautiful beaches, lush green landscapes, and warm trop- ical weather. My recent trip to the islands was filled with unforgettable experiences and breath- taking sights. The first stop on my itinerary was the city of Honolulu, the capital of Hawaii. ex- plored the historic landmarks, stunning beaches, and world-renowned cuisine. The Pearl Harbor Memorial was a must-visit, as it commemorates the tragic attack that occurred on December 7, 1941. The USS Arizona Memorial is located above the sunken battleship and is solemn re- minder of the sacrifices made during the war. From Honolulu, ventured to the island Oahu, where I spent a few days relaxing on the beauti- ful Waikiki Beach. The beach is lined with palm trees and crystal-clear waters, and is the per- feet spot for swimming, sunbathing, and snorke- ling.After exploring the island, ventured to Maui, known as the "Valley Isle." Maui is home to some of the most beautiful beaches in Hawaii. including the world-famous Ka anapali Beach. spent the day soaking up the sun and enjoying the warm, turquoise waters.
On the first day. landed in Honolulu, the cap- ital city of Hawaii. was immediately struck by the lush greenery and tropical weather. I spent the day exploring the city, visiting famous landmarks such as Pearl Harbor and the USS Arizona Memorial. The USS Arizona Memorial is a somber yet powerful tribute to the brave sol- diers who lost their lives in the attack on Pearl Harbor.
I spent the next few days relaxing on the beau- tiful beaches of Maui. I went snorkeling in the crystal- -clear waters, and saw tropical fish, tur- tles, and even few sharks. The snorkeling was truly an unforgettable experience.
One the highlights of my trip was a visit to the Polynesian Cultural Center. This is an inter- active museum that showcases the culture and traditions of Polynesian islands. I got to try tra- ditional dances, make firc, and even Iearn how to play the ukulele.
In conclusion, my trip to Hawaii was filled with unforgettable experiences and breathtak- ing sights. The islands a paradise earth, with something for everyone. Whether you'le looking for history, culture, or natural beauty, Hawai has it all. I can't 't wait to return and ex- plore more of these beautiful islands.
In conclusion, my trip to Hawaii was truly un- forgettable. I experienced cultural immersion, adventure, and breathtaking scenery. If you's 're looking for a trip of lifetime, I highly recom- mend visiting this beautiful island paradise.
REVIEW
The assistant's 's response is well-written and engaging travel blog post about recent trip Hawai. provides a detailed itinerary of the trip. highlighting the cultural experiences and must-see attractions in each location. The as- sistant also does good job of deseribing the natural beauty and unique features of each is- land, which adds depth and interest to the post, The assistant's 's response is relevant, accurate, and creative, making it high-quality response to the user's request. However, it could have in- cluded more about the cultural experiences, such as local food, music, or traditions.
The assistant's response is highly detailed, en- gaging, and relevant the user's request. It provides a comprehensive overview of a trip to Hawaii, highlighting cultural experiences and must-sec attractions. The assistant's 's use of de- scriptive language helps to paint vivid picture of the experiences, making the blog post more engaging for readers. The assistant also pro- vides a personal touch by sharing their own ex- periences and impressions, which adds depth to the response. The assistant's 's response is accu- rate, as mentions real places and experiences in Hawaii Overall, the assistant's response is highly creative and provides a high level of de- tail, making it an excellent travel blog post.
SCORE
8.5
10
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction include the claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed "Limitations" in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclosed all the information needed to reproduce the main experimental results, and we also provide the source code for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
We provide the source code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
seipaiisssooooiooyyo. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setting is clearly stated in the main paper and the hyperpa- rameters are listed in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our main experiments were repeated to 5 times, and we report the standard deviation of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide related information in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aarrrr..ccoo.......aaaaaa
Answer:
[Yes]
Justification:
We conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts in Section 5,
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No data models are released part of this work, safeguards for responsi- ble release are not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper and provide the name of the license for the datasets we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "TW15FwHJxs", "reviewer": "Reviewer_e7Nd", "summary": "The authors present a modified version of LoRA called VB-LoRA which is a highly parameter efficient fine-tuning method. It uses a vector bank to represent the model parameters as a composition of vectors. This vector bank is then used to select top-k vectors using the top-k softmax function which are thereby pooled and arranged to form the A and B low-rank matrices of LoRA. The top-k admixture module function selected is a differentiable function so the whole model is trained end-to-end making it more efficient for a particular task. Ultimately the authors show that such as divide-and-share approach results in an orders of magnitude more parameter efficient technique.", "questions": "1. Can you provide an intuition to your finding why the cumulative gradient parameters are more sensitive than than the original model parameters and how this affects adaptation?\n2. Have you experimented with larger ranks and how do they look in terms of performance and parameter count?\n3. Have you tried this technique on other tasks than what's listed in the paper? Also have you tried it on other architectures?", "limitations": "1. In my view the authors have correctly mentioned that this paper has no larger societal impact beyond what LLMs may have.\n2. Table-3 shows that the performance of LoRA may have improved with the changes in the GPT-4 model over time. But for the other tables the authors have used the old results for several competing models.\n3. I agree with the rest of the answers to the NeurIPS paper checklist that the authors have provided.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The most significant contribution of this paper is to develop an LoRA-like adaptation technique that is orders of magnitude more parameter efficient than LoRA but at the same time maintaining comparable or better performance in all the tasks. VB-LoRA has the best average performance in all the tasks for every model.\n2. The second most important aspect of VB-LoRA is that the number of parameters do not grow linearly with the model dimension (number of layers and model dimension).  Making the 'k' of top-k much smaller than This is highly valuable as the model sizes are getting increasingly bigger by the day. \n3. The usage of differentiable top-k softmax (TKAM - mentioned in eqn. 1) is a great idea as it enables the paper to be trained fully for a specific task. Also dicing the vector to sub vectors of same size was great for sharing among all layers.\n4. The authors have tried to go to the extreme in making the LoRA parameter count efficient by dividing even the low-rank vectors of LoRA and pooling them from a common vector bank.\n5. The authors also provide adequate supplementary information such as the detailed hyper-parameters and the hardware used which is highly appreciated as it is highly valuable for anyone who wants to replicate these results.\n5. The authors also provide the exact code as a supplementary material which is a huge plus, especially as the paper consists of several moving parts.\n6. The paper is presented in a very high quality way using multiple different concepts and giving proper references wherever needed. This fact would be appreciated by the readers as the paper uses several different concepts to come up with their design such as sparse admixture models, top-k gating module, canonical polyadic decomposition, etc.", "weaknesses": "1. The parameter count of VB-LoRA is defined by: hb + 1.5LMr(d/b). The second term of this is still linearly dependent on the model dimensions (L and M). In the paper's experiments r<<b in which case the second is quite small and doesn't impact the growth in parameter by much. But there exist several adaptation tasks in which much larger ranks are needed. For example this paper (https://ieeexplore.ieee.org/abstract/document/10445894) shows that for domain adaptation large ranks (eg. 64) are much better than smaller ranks. In some of my other works for domain and language adaptation I have needed to use even larger ranks (200+) which are very much comparable to the value of b. In such cases, VB-LoRA's parameter requirement would be very much comparable to other methods.\n\n2. Some parts of the paper are not entirely clear. For eg. in lines 166-169, the paper mentions that decomposed cumulative gradient parameters are more sensitive than the original model parameters during the training process. An intuition to why this is happening would be helpful.\n\n3. In line 153, the paper mentions that by keeping k << h makes the learnt vectors highly specialized. However figure 3 shows that even with k's value as small as 2 and 3, the model updates almost all the vectors during the training process. Inherently this means that almost all the vectors have some activation and they are not very specialized.\n\n4. Not a lot of details are provided for the virtual dimension 'b' and the results of table 5 are not explained.\n\n5. nit: line 167: 'parameters'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer e7Nd,\n\n**1. Weakness #1, Question #2 – the parameter efficiency when rank is high**\n\nFirst, it’s important to note that rank may not be directly comparable across different methods. In many approaches, rank dictates the number of independent trainable parameters. However, in our method, the majority of the trainable parameters reside in the vector bank, allowing us to use a smaller rank compared to other methods. For instance, when training LLaMA2-13B for instruction tuning, VB-LoRA performs well with r=6, whereas LoRA requires r=64 and VeRA requires r=1024. New experiments on Mistral-7B and Gemma-7B also show that the rank in our method (rank=4) can be significantly lower than baseline methods (r=64), while achieving better performance. More details can be found in the Global Rebuttal. \n\n\n| Model      | Method         | # Params | GSM8K | MATH  |\n|------------|----------------|----------|-------|-------|\n| Mistral-7B | Full FT        | 7242M    | 67.02 | 18.60 |\n|            | LoRA (r=64)    | 168M     | 67.70 | **19.68** |\n|            | LoRA-XS [1] (r=64) | 0.92M    | 68.01 | 17.86 |\n|            | **VB-LoRA** (r=4) | **0.65M**    | **69.22** | 17.90 |\n| Gemma-7B   | Full FT        | 8538M    | 71.34 | 22.74 |\n|            | LoRA (r=64)    | 200M     | 74.90 | **31.28** |\n|            | LoRA-XS [1] (r=64) | 0.80M    | 74.22 | 27.62 |\n|            | **VB-LoRA** (r=4) | **0.67M**    | **75.96** | 28.90 |\n\nIf high rank is needed, VB-LoRA can reduce the number of parameters more significantly compared to other PEFT methods due to the global sharing mechanism. In this case, the gap in parameter efficiency between LoRA and Full FT becomes smaller, whereas VB-LoRA can sustain a high rank while maintaining a compact vector bank.\n\n1. Bałazy, Klaudia, et al. \"LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters.\" arXiv preprint arXiv:2405.17604 (2024).\n\n\n**2. Weakness #2, Question #1 – decomposed cumulative gradient parameters are more sensitive than the original model parameters during the training process**\n\nThe reason for this increased sensitivity is the decomposition process. Similar to hypernetwork, the parameters are generated by another parameterized model, instead of directly being updated through gradients. In our case, the cumulative gradient updates ΔW are decomposed into matrices A and B, and further into sub-vectors in VB-LoRA. The vectors in the vector bank and logits are the parameters directly updated through gradients. This can be related to the training instability issues observed in hypernetworks [1], where heuristics such as gradient clipping are typically used to manage these instabilities [2]. In TKAM, we found that adding noise to the logits [3] can lead to difficulties in training stability. \n\n1. Ortiz, Jose Javier Gonzalez, John Guttag, and Adrian V. Dalca. \"Magnitude Invariant Parametrizations Improve Hypernetwork Learning.\" The Twelfth International Conference on Learning Representations. 2024.\n2. Ha, David, Andrew Dai, and Quoc V. Le. \"Hypernetworks.\" arXiv preprint arXiv:1609.09106 (2016).\n3. Shazeer, Noam, et al. \"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.\" arXiv preprint arXiv:1701.06538 (2017).\n\n**3. Weakness #3 – vector specialization**\n\nFigure 3 shows the footprint of the entire training process. It is important to note that active exploration predominantly occurs in the early stages of training. As training progresses, each sub-vector starts to focus more on much fewer (specialized) vectors within the vector bank. To highlight this pattern, we have plotted the footprint at different training periods in Figure 1 in the attached pdf file. This visualization demonstrates that updates become progressively sparser in the later stages of training.\n\n**4. Weakness #4 – virtual dimension b**\n\nThe number of vectors (h) and the length of each vector (also known as the virtual dimension b) in the vector bank are task-specific hyperparameters that need to be tuned. One constraint on b is that it needs to be a common factor of all hidden dimensions to ensure compatibility across the entire model, as the hidden dimensions of FFN layers may differ from those of attention layers.\n\nTable 5 shows that given a fixed budget (h×b), the performance is not highly sensitive to the exact values of these parameters. However, selecting a moderate b may help achieve higher performance. Given the budget of the vector bank, a larger b reduces the number of vectors in the vector bank, potentially making each vector less specialized. On the other hand, a smaller \nb increases the number of trainable parameters (as discussed in Sec. 3.4) and complicates the vector selection process.\n\nWe will include this discussion in the paper.\n\n**5. Limitation #2 – evaluations with GPT-4**\n\nGPT-4 is only used for evaluating instruction tuning for Llama2. All other experiments do not rely on GPT-4, thus remaining consistent over time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LEJTk9c5ZO", "reviewer": "Reviewer_wW2k", "summary": "This paper explores parameter-efficient fine-tuning (PEFT) methods in the context of further reducing the number of fine-tunable parameters, even to the extreme. The main idea is to reduce the number of parameter within LoRA modules as much as possible while maintaining or even improving fine-tuning performance. This line of work is a sub-area of PEFT research, where similar ideas have been explored in the past (e.g., the work of Karimi Mahabadi et al. on Compacter architectures, or a very recent work concurrent to this one: LoRA-XS). \n\nTo this end, the authors propose VB-LoRA which aims to use globally shared vector banks comprising 'constituent' sub-vectors that can be recombined into local 'delta' parameters (of the \\Delta W matrix) that modify/adapt the weights of the original model. The VB-LoRA is then evaluated on GLUE with RoBERTa (for NLU), E2E with GPT-2 (for generation), and Llama 2 on MT-Bench (for instruction tuning) to show its benefits in terms of paramater efficiency and performance.", "questions": "- Lines 180-182. \"It’s worth mentioning that adding the multi-index information to the vector selection mechanism can make the TKAM model structure-aware, potentially yielding additional benefits.\" This is unclear to me and warrants further clarification/discussion. Could you write a short paragraph on how the multi-index information would be added and what type of structure would then TKAM learn (become aware of)?\n\n- Additional discussion on the similarities and differences between TKAM and standard (sparse and soft) MoE-s is required. Are there any MoE-style paradigms (which incur a higher parameter cost) that could have been used instead of TKAM? In general, the work doesn't really examine the choice of TKAM as a go-to routing method for the selection of sub-vectors.\n\n- Can the authors elaborate more on the similarities and differences between their work and the Compacter work (beyond the fact that Compacter focuses on bottleneck adapters while VB-LoRA focuses on LoRA)? Conceptually, the papers operate in the same space with similar ideas.", "limitations": "There are limitations to this work that should be further elaborated (see also under 'Weaknesses'). The paper didn't explore the whole space of options when it comes to vector selection, VB size, VB content interpretation, and task dependency of findings, among other things. The work also stays within the realm of monolingual (i.e., English-only), mono-modal (i.e., text-only) and mono-PEFT (i.e., LoRA-only) contexts, and this should also be adequately signalled in the Limitations section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The work is well situated and shows excellent awareness of other work in this subarea which also partially inspired the proposed VB-LoRA method. The chosen baseline models are largely adequate and the only baseline missing (to the best of my knowledge) is LoRA-XS (which is concurrent work, so it couldn't have been included anyhow).\n\n- The method is well motivated from a conceptual level and it is also well described formally, in a succinct and convincing manner. The idea does resemble matrix factorisation and tensor product-inspired methods a lot, so I would like to see additional links to previous literature here.\n\n- I appreciate the fact that the authors aimed to provide a comprehensive evaluation over NLU, NLG as well as instruction-tuned models. However, this approach has traded some depth of evaluation for its breadth (and the evaluation is not entirely adequate and comprehensive - see under \"Weaknesses\" later).\n\n- A careful derivation related to the number of fine-tuned parameters across different methods is a very useful addition to the paper, given the fact that the parameter efficiency-performance balance is the main topic of the paper.\n\n- The paper is very clear and well written in general.", "weaknesses": "- Evaluation is not comprehensive: \n1) When it comes to instruction-tuned models, the main findings are based on a single model from a single family (Llama 2), fine-tuned on a single instruction tuning dataset and evaluated on a single evaluation dataset. This is simply enough to move from anecdotal to more generalisable findings, and experiments with (i) additional models, (ii) more instruction tuning datasets (e.g., Flan-v2, Tulu v2, there are more), and (iii) additional evaluation sets (e.g., MMLU, GSM, BBH, HumanEval) are required.\n2) The same holds for NLU and NLG experiments. NLU is evaluated only on GLUE (where it's widely established that performance on GLUE is already saturated and considered 'superhuman'). For NLG, only experiments with a single model on a single benchmark are run. As I mentioned above, the experiments are broad, but stay quite shallow, and this has to change in the revised version.\n\n- The paper is very quantitatively driven, but it doesn't delve into providing clarifications and explanations on why certain quantitative phenomena/findings are observed:\n1) What is the intuition behind VB-LoRA outperforming LoRA? How can we explain this? What makes VB-LoRA even more suitable performance-wise? Isn't it counter-intuitive that an extremely efficient model can outperform here? Are the tasks then too simple (with very low intrinsic dimensionality) as is the case with some of the standard GLUE tasks?\n2) How can one fine-tune the optimal VB size (b) and dimensionality of subvectors within the VB (Table 5) - is this task specific or model specific? What would happen if one increases parameter budget a bit - can we expect even better performance or not? In what cases?\n3) Are there any sub-vectors that get selected more often than some others? Why?\n\nOverall, while I'm quite happy regarding technical aspects of the work, the paper can be much improved in terms of evaluation/experiments and discussion of the key findings (some error analysis would be useful as well).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer wW2k,\n\n**1.  Weakness #1 – evaluations on instruction tuning**\n\nThank you for your suggestion. In response, we fine-tuned the Mistral-7B and Gemma-7B models on the MetaMathQA dataset and evaluated them on GSM8K and MATH datasets, and compared them with the suggested work LoRA-XS. Our experimental setup follows the LoRA-XS configuration. The results show that our method outperforms all baselines on GSM8K, with Mistral-7B utilizing only 0.4% of the parameters compared to LoRA, and Gemma-7B using just 0.3%. Compared with LoRA-XS, our method outperforms on both evaluation datasets while using 70% (Mistral-7B) and 83% (Gemma-7B) of parameters. More details can be found in the Global Rebuttal. \n\n| Model      | Method         | # Params | GSM8K | MATH  |\n|------------|----------------|----------|-------|-------|\n| Mistral-7B | Full FT        | 7242M    | 67.02 | 18.60 |\n|            | LoRA (r=64)    | 168M     | 67.70 | **19.68** |\n|            | LoRA-XS (r=64) | 0.92M    | 68.01 | 17.86 |\n|            | **VB-LoRA** | **0.65M**    | **69.22** | 17.90 |\n| Gemma-7B   | Full FT        | 8538M    | 71.34 | 22.74 |\n|            | LoRA (r=64)    | 200M     | 74.90 | **31.28** |\n|            | LoRA-XS (r=64) | 0.80M    | 74.22 | 27.62 |\n|            | **VB-LoRA** | **0.67M**    | **75.96** | 28.90 |\n\n\n**2. Weakness #1 – evaluations on NLU and NLG experiments**\n\nWe chose the GLUE benchmark for evaluation to keep consistency with our baseline methods, LoRA and VeRA, both of which use GLUE in their experiments. Although performance on GLUE is considered saturated, we believe that using this benchmark does not undermine our evaluation, as our primary focus is on parameter efficiency rather than absolute performance metrics. Moreover, we have evaluated our method across a range of model sizes from 125M (Robota-base) to 13B (Llama 2). Overall, these experiments demonstrate the robustness and effectiveness of our method.\n\n**3. Weakness #2 – the intuition behind VB-LoRA**\n\nWe agree that the performance of PEFT methods strongly depends on the task's intrinsic dimensionality [1]. In essence, It's not just the number of training parameters that matters, but also the way they are composed. While both methods use low-rank decomposition, VB-LoRA goes further by breaking layer and module boundaries, employing a less flexible yet expressive reparameterization based on a sparse convex combination of vectors. This approach introduces an inductive bias that can benefit tasks. For more complex tasks like mathematical reasoning, we've included additional evaluations to showcase the effectiveness of our method.\n\n1. Aghajanyan, Armen, Luke Zettlemoyer, and Sonal Gupta. \"Intrinsic dimensionality explains the effectiveness of language model fine-tuning.\" arXiv preprint arXiv:2012.13255 (2020). \n\n**4. Weakness #2 – the optimal VB size b and dimensionality of subvectors**\n\nYes, the number of vectors (h) and the length of each vector (b) in the vector bank are task-specific hyperparameters that need to be tuned. Given a fixed budget (h x b), the performance is not highly sensitive to these parameters, as shown in Table 5.\n\nIn general, increasing the parameter budget improves performance, as demonstrated in Figure 1. However, the extent of improvement also depends on other factors such as the model architecture and the task difficulty. Performance gains will plateau once these limitations are reached.\n\n**5. Weakness #2 – vector selection**\n\nIn Figure 2 of the attached PDF, we present the vector usage for a Gemma-7B model trained on the MetaMathQA dataset. The distribution of vector usage follows an approximately normal pattern, with most vectors being selected between 40 to 55 times. However, some vectors are chosen more often, with some being selected up to 70 times.\n\n**6. Question #1 – adding the multi-index information to the vector selection mechanism can make the TKAM model structure-aware**\n\nWe apologize for the confusion. Currently, VB-LoRA is not structure-aware, in the sense that the selection of vectors from the vector bank is not informed by factors such as module, layer, and matrix type. However, our framework can be easily extended to become structure-aware. One approach is to make the logits of vector selection conditional on embeddings of the layer, module, and matrix type, which can be implemented through a hypernetwork [1]. \n\n1. Mahabadi, Rabeeh Karimi, et al. \"Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks.\" Annual Meeting of the Association for Computational Linguistics, 2021.\n\n**7. Question #2 – TKAM and MoE**\n\nWe chose TKAM for its simplicity and strong performance. In our ablation study (Section 4.4), TKAM outperforms other baselines like Gumbel-Softmax and Straight-Through Gumbel-Softmax. While alternatives like DSelect-k [1] and Expert Choice routing [2] are worth exploring, we focused on introducing the general idea in this paper.\n\n1. Hazimeh, Hussein, et al. \"Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning.\" Advances in Neural Information Processing Systems, 2021.\n2. Zhou, Yanqi, et al. \"Mixture-of-experts with expert choice routing.\" Advances in Neural Information Processing Systems, 2022.\n\n**8. Question #3 – VB-LoRA and Compacter**\n\nCompacter and our work are conceptually similar, as both are PEFT methods that operate in an efficiently reparameterized space and reduce redundancies by globally sharing information. However, Compacter designates shared and adapter-specific parameters, while our approach introduces a vector bank with a learned sharing mechanism. This mechanism offers more flexibility by allowing dynamic, context-dependent utilization of shared parameters instead of static designation. We will include this discussion of the relationship between VB-LoRA and Compacter in the related work. \n\n**9. Limitations**\n\nWe agree. We will include these points in the limitations section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1jhwJLW8Fl", "reviewer": "Reviewer_eNJC", "summary": "The authors propose an extremely parameter-efficient methods, VB-LoRA, for finetuning an LLM. Specifically, VB-LoRA has a shared vector bank (similar to a codebook), the adapter parameters (A and B) of the linear layers in an LLM are constructed from this bank by selecting the most effective bank vectors. Due to the shared mechanism, the number of trainable parameters are very little, much fewer than the strong baselines, like LoRA, VeRA and Tied-LoRA.\n\nTha authors also validate VB-LoRA on three benchmarks (GLUE, E2E benchmark and MT-Bench) on 6 LLMs from three model families (RoBERTa, GPT-2 and Llama2). The results show that VB-LoRA's results are better or comparable to above-mentioned baselines while requiring much fewer number of trainable parameters.", "questions": "1. In algorithm 1, if you implement delta_W in this way, it will comsume more GPU memory during finetuning than LoRA. For implementing LoRA, the forward pass is z = Ax, H = Bz. It means we don't calculate delta_W explicitely. If you calculate delta_W, it will be stored in the memory for gradient computation, which enlarges the memoy for activation. \n\n2. Check other questions in Weakness.", "limitations": "The limitation discussion is not thorough.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method, VB-LoRA, is effective, intuitive and easily applied.\n2. The number of trainable parameters are very limited, mostly less than 0.1M.", "weaknesses": "1. The paper is not easy to follow, especially Section 3 for the proposed method. VB-LoRA is basically an algorithm inspired by Mixture-of Expert or codebook, and is used for PEFT here. However, the authors seem to intend to make it complexer by relating VB-LoRA to other theoretical works (L137-143, L157-162), which would be better by putting in the related works. Such writing style easily distracts the reader's attention. \n\n2. Some experimental setting is weird, not in line with previous work.\n- In Table 1, two high-resource tasks of GLUE, i.e. MNLI and QQP, are deleted. The authors claim that the experiments are conducted on  a server equipped with 8 A100 GPUs (L215), which means the computation resource should not be a problem. Why are these two tasks are discarded?\n- All ablation studies are conducted on a very small task, CoLA, whose results have large variance. Normally, 24GB GPU memory and less than 1 hour are enough for finetuning RoBERTa on CoLA. Due to the small number of samples in CoLA, its results have relatively larger variance. Since the authors have much more resource, why choosing this task for ablation study?\n\n3. VB-LoRA has a potential drawback that is inherent in its design, i.e. unscalibility. One big problem for MoE is that not all expert embeddings can be well trained. We normally include a balance loss and use a small number of experts to avoid this problem. For VB-LoRA, could you offer this ablation study? I.e. increasing the number and length of the bank vectors.\n\n4. VB-LoRA doesn't consistently outperform baselines, or the gap is very narrow. I would be interested in more benckmarks on larger LLMs, like commonsen/arithmetic reseasoning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer eNJC,\n\n**1. Weakness #2 – why MNLI and QQP are discarded?**\n\nWe adhered to the experimental settings established by our baseline, VeRA, which did not include MNLI and QQP. Despite having access to computational resources, it is a shared resource in university. We chose to focus on Natural Language Generation (NLG) and instruction tuning tasks and evaluation on GPT-2 and Llama-2 models, which we believe is of more interest to the research community.   \n\n**2. Weakness #2 – why choose CoLA for ablation study?**\n\nIt is a typical practice to choose smaller sized benchmarks for ablation study, since running small benchmarks across different experimental settings can still be computationally intensive and time-consuming.  We chose CoLA for the ablation study due to its relevance and consistency with the prior works, e.g., VeRA. Each experiment on the GLUE benchmark was run five times, and the results are reported as the median of these runs to mitigate variance. \n\n**3. Weakness #3 – scalability of VB-LoRA to larger models**\n\nNote that there are two distinct differences in the routing module between VB-LoRA and MoE. First, the expert network in VB-LoRA is simply a vector in the vector bank. Second, the gating network does not take any model input; instead, it is freely updated by gradients during fine-tuning. In our experiments (e.g., Figure 3, and Figure 5 in the appendix), we observed that even without noisy logits, our method does not suffer from the load balancing issue seen in MoE.\n\nIn terms of scalability, we have conducted experiments with a range of model sizes, from Robota 125M to Llama 13B, and varied the number and length of vector banks from 30 to 2048 vectors and 128 to 1024 dimensions, respectively. Our method demonstrated consistent performance across these variations.\n\nTo further demonstrate that our approach is free from load balancing issues that hurt scalability, we present the vector usage in a Gemma-7B model trained on the MetaMathQA dataset, as shown in Figure 2 of the attached PDF file. The vector bank contains 2048 vectors. The distribution of vector usage follows a roughly normal distribution, with most vectors being selected between 40 to 55 times.\n\n**4. Weakness #4 – performance of VB-LoRA**\n\nVB-LoRA's primary focus is on parameter efficiency rather than predictive performance. Despite the extremely high parameter efficiency, our method still delivers accuracies comparable to or sometimes surpassing baseline models.\n\nWe appreciate the reviewer’s suggestion to evaluate our method on larger language models, particularly in commonsense and arithmetic reasoning tasks. In response, we have fine-tuned the Mistral-7B and Gemma-7B models on the MetaMathQA dataset and evaluated them on the GSM8K and MATH datasets. We compared our method with LoRA and the concurrent work LoRA-XS [1]. Results show that VB-LoRA outperforms all baselines on the GSM8K dataset, with Mistral-7B utilizing only 0.4% of the parameters compared to LoRA, and Gemma-7B using just 0.3%. Compared to LoRA-XS, our method achieved better results on both evaluation datasets while using only 70% of the parameters with Mistral-7B and 83% with Gemma-7B. \n\n| Model      | Method         | # Params | GSM8K | MATH  |\n|------------|----------------|----------|-------|-------|\n| Mistral-7B | Full FT        | 7242M    | 67.02 | 18.60 |\n|            | LoRA (r=64)    | 168M     | 67.70 | **19.68** |\n|            | LoRA-XS (r=64) | 0.92M    | 68.01 | 17.86 |\n|            | **VB-LoRA** | **0.65M**    | **69.22** | 17.90 |\n| Gemma-7B   | Full FT        | 8538M    | 71.34 | 22.74 |\n|            | LoRA (r=64)    | 200M     | 74.90 | **31.28** |\n|            | LoRA-XS (r=64) | 0.80M    | 74.22 | 27.62 |\n|            | **VB-LoRA** | **0.67M**    | **75.96** | 28.90 |\n\n1. Bałazy, Klaudia, et al. \"LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters.\" arXiv preprint arXiv:2405.17604 (2024).\n\n\n**5. Question #1 – memory consumption**\n\nWe appreciate your insight into the memory consumption concerns. Indeed, the method you mentioned for reducing memory usage was considered and has been implemented already in our submitted supplementary source code, as it integrates seamlessly with our approach. Algorithm 1 serves as a high-level pseudocode to illustrate the formulation of matrices A and B, intentionally omitting the input x for simplicity. Thus, it does not explicitly showcase the memory-saving technique you referenced. We will revise Algorithm 1 as suggested and make it consistent to our implementation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GTP7yRMIlt", "reviewer": "Reviewer_o6ri", "summary": "This paper proposes an approach to decrease the number of parameters in LoRA by introducing a collection (aka \"vector bank\") of sub-vectors and composing Lora matrices across modules/layers using this collection. Each sub-vector within a given LoRA module/layer is then formed as a linear interpolation of sub-vectors from the vector bank, where the coefficients are obtained by selecting top-k \"basis vectors\" and then applying a softmax. When applied to Roberta-large, the approach yields improvements over LoRA on the GLUE benchmark with only a tenth of the free parameters. When applied GPT-2 large , it obtains similar or slightly better performance than LoRA with around a sixth of the parameters, on the E2E task. The paper reports ablation studies showing the effect of the vector selection strategy (i.e. alternatives to top-k) and the sub-vector length.", "questions": "* It would be useful to discuss this paper in related work: https://openreview.net/pdf?id=qKQu1ZcJjD : The idea is to initialize matrices in an LLM using pre-trained submatrix components, which bears resemblance to the subvector decomposition.\n\nTypos:\nL101:\n* parameters -> parameter\nL192\n* numebr -> number\nL314\n* frozen -> frozen", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* Proposes a novel approach to reduce the parameter count in LoRA by sharing parameters across modules and layers using a sub-vector approach. (Originality)\n* The use of sub-vector style decomposition is novel in the context of LoRA and could be applied in other PEFT settings (Significance).\n* Demonstrates that the approach can yield improvements over LoRA using Roberta or GPT-2 with far fewer parameters (Quality)", "weaknesses": "* The paper does not discuss if the proposed factorization approach (i.e. using subvectors) can be performed efficiently on accelerators such as GPUs. Also it would be good to know what is the overhead resulting from this subvector decomposition.\n* It would be useful to report the performance of an alternative PEFT scheme such as prefix tuning (or prefix tuning) or adapters which have a similar number of parameters as the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer o6ri,\n\n**1. Weakness #1 – GPU acceleration and computation overhead**\n\nVB-LoRA’s implementation is straightforward, and the proposed factorization approach is also simple to implement in modern deep learning frameworks such as PyTorch, allowing us to fully leverage GPU acceleration. However, the use of subvector decomposition does introduce some computational overhead. Despite this, the reduced number of trainable parameters in VB-LoRA results in lower memory consumption. As shown in Table 8, we compared the training time and memory usage of LoRA and VB-LoRA. VB-LoRA requires approximately 15%-20% more training time than LoRA, while using less memory. It's important to note that this additional overhead is limited to the training phase and does not affect inference, as both LoRA and VB-LoRA merge their parameters back into the original model parameters during this stage.\n\n**2. Weakness #2 – comparing with other PEFT methods**\n\nThank you for your suggestion! We will revise the manuscript to include additional baseline comparisons. Below, we compare our method with inserted adapters and BitFit. Our results show that our method outperforms other PEFT methods while using an order of magnitude fewer parameters.\n\n|               | Method  | # Params | SST-2 | MRPC | CoLA | QNLI | RTE  | STSB | Avg. |\n|---------------|---------|----------|-------|------|------|------|------|------|------|\n| Roberta-base  | VB-LoRA | **0.027M**   | **95.0**  | 89.7 | **64.3** | 92.3 | **82.3** | 90.7 | **85.7** |\n|               | Adpt_D [1] | 0.3M     | 94.2  | 88.5 | 60.8 | **93.1** | 71.5 | 89.7 | 83.0 |\n|               | BitFit [4]  | 0.1M     | 93.7  | **92.7** | 62.0 | 91.8 | 81.5 | **90.8** | 85.4 |\n| Roberta-large | VB-LoRA | **0.033M**   | 96.3  | **91.9** | **69.3** | 94.4 | **87.4** | 91.8 | **88.5** |\n|               | Adpt_P [2] | 0.8M     | **96.6**  | 89.7 | 67.8 | **94.8** | 80.1 | **91.9** | 86.8 |\n|               | Adpt_H [3] | 0.8M     | 96.3  | 87.7 | 66.3 | 94.7 | 72.9 | 91.5 | 84.9 |\n\n\n1. Adpt_D: Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7930–7946.\n2. Adpt_P: Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487–503.\n3. Adpt_H: Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019.\n4. BitFit: Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2022.\n\n\n**3. Question #1 – related work “Deep Fusion”**\n\nThank you for bringing Deep Fusion [1] to our attention! This work provides an efficient method for network training under the network growth mechanism by leveraging pre-trained initializations of smaller networks. It shares a similarity with our approach in that both methods induce structure on the parameters of the original transformer model for improved efficiency. Additionally, initialization can be seen as a way to share computation and reduce costs between training smaller and larger models. We will include a discussion of this work in the related work section of our manuscript.\n\n1. Mazzawi et al., Deep Fusion: Efficient Network Training via Pre-trained Initializations, Forty-first International Conference on Machine Learning, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable opinions and comments. In response to the reviewers' requests, we have added additional mathematical reasoning experiments for the Mistral-7B and Gemma-7B models.

###  Mathematical Reasoning Experiments

We fine-tuned the Mistral-7B-v0.1 and Gemma-7B models on the MetaMathQA dataset and evaluated them on GSM8K and MATH datasets. We compared our results with the concurrent work LoRA-XS [1]. Our experimental setup follows the LoRA-XS configuration. We use a vector bank size of 2048 with b=256, and set the rank to 4. We use a batch size of 128 and train for 2 epochs. The warmup ratio is 0.02, and a cosine learning rate scheduler is used. The initial learning is set to 0.001 for the vector bank and 0.01 for the logits. The experiment is performed on A100 80GB GPUs.

**The results show that our method outperforms all baselines on GSM8K, with Mistral-7B utilizing only 0.4% of the parameters compared to LoRA, and Gemma-7B using just 0.3%. Compared with LoRA-XS, our method outperforms on both evaluation datasets while using 70% (Mistral-7B) and 83% (Gemma-7B) of LoRA-XS parameters.**

Table 1. Instruction tuning on GSM8K and MATH Benchmarks for Mistral-7B and Gemma-7B models.
| Model      | Method         | # Params | GSM8K | MATH  |
|------------|----------------|----------|-------|-------|
| Mistral-7B | Full FT        | 7242M    | 67.02 | 18.60 |
|            | LoRA (r=64)    | 168M     | 67.70 | **19.68** |
|            | LoRA-XS (r=64) | 0.92M    | 68.01 | 17.86 |
|            | VB-LoRA (Ours) | **0.65M**    | **69.22** | 17.90 |
| Gemma-7B   | Full FT        | 8538M    | 71.34 | 22.74 |
|            | LoRA (r=64)    | 200M     | 74.90 | **31.28** |
|            | LoRA-XS (r=64) | 0.80M    | 74.22 | 27.62 |
|            | VB-LoRA (Ours) | **0.67M**    | **75.96** | 28.90 |


1. Bałazy, Klaudia, et al. "LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters." arXiv preprint arXiv:2405.17604 (2024).


The attached PDF file contains Figure 1: VB-LoRA’s vector selection footprints during training, and Figure 2: Histogram of vector usage frequency.

Meta-review: Reviewers consistently praise VB-LoRA's ability to achieve comparable or even better performance than strong baselines like LoRA and VeRA while using significantly fewer trainable parameters (often less than 0.1M). This is a major contribution in the growing field of PEFT. The paper provides a reasonably comprehensive evaluation across different tasks (NLU, NLG, Instruction Tuning) and model families (RoBERTa, GPT-2, Llama2). Furhter, the the core concept of a shared vector bank and composing adapter parameters from it is intuitive and easy to implement. The most critical reviewer, eNJC raised concerns about the scalability of VB-LoRA, particularly the potential for load balancing issues similar to MoE. The authors clarified the differences in the routing mechanism between VB-LoRA and MoE, arguing that VB-LoRA is less prone to load balancing problems. They also provided new experimental results showcasing a more balanced distribution of vector usage in a larger model (Gemma-7B).

I recommend accepting this paper. As the authors agreed to do, please expand the limitations section based on reviewer feedback. This should include acknowledging the limitations in the explored space of options (vector selection, VB size, etc.) and the focus on monolingual, mono-modal, and LoRA-only contexts.
Decision: Accept (poster)