Title: VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks
Venue: NeurIPS 2024
Accepted: yes
Keywords: Parameter-efficient fine-tuning

Low-rank adaptation

Transfer learning

Transformer

Top-k admixture module
Abstract: As the adoption of large language models increases and the need for per-user or per-task model customization grows, the parameter-efficient fine-tuning (PEFT) methods, such as low-rank adaptation (LoRA) and its variants, incur substantial storage and transmission costs. To further reduce stored parameters, we introduce a "divide-and-share" paradigm that breaks the barriers of low-rank decomposition across matrix dimensions, modules, and layers by sharing  parameters globally via a vector bank. As an instantiation of the paradigm to LoRA, our proposed VB-LoRA composites all the low-rank matrices of LoRA from a shared vector bank with a differentiable top-$k$ admixture module. VB-LoRA achieves extreme parameter efficiency while maintaining comparable or better performance compared to state-of-the-art PEFT methods. Extensive experiments demonstrate the effectiveness of VB-LoRA on natural language understanding, natural language generation, instruction tuning, and mathematical reasoning tasks. When fine-tuning the Llama2-13B model, VB-LoRA only uses 0.4% of LoRA's stored parameters, yet achieves superior results. Our source code is available at https://github.com/leo-yangli/VB-LoRA. This method has been merged into the Hugging Face PEFT package.

VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks

Abstract

As the adoption of large language models increases and the need for per-user or pertask model customization grows, the parameter-efficient fine-tuning (PEFT) methods, such as low-rank adaptation (LoRA) and its variants, incur substantial storage and transmission costs. To further reduce stored parameters, we introduce a "divideand-share" paradigm that breaks the barriers of low-rank decomposition across matrix dimensions, modules, and layers by sharing parameters globally via a vector bank . As an instantiation of the paradigm to LoRA, our proposed VB-LoRA composites all the low-rank matrices of LoRA from a shared vector bank with a differentiable topk admixture module. VB-LoRA achieves extreme parameter efficiency while maintaining comparable or better performance compared to state-of-the-art PEFT methods. Extensive experiments demonstrate the effectiveness of VB-LoRA on natural language understanding, natural language generation, instruction tuning, and mathematical reasoning tasks. When fine-tuning the Llama2-13B model, VBLoRA only uses 0.4% of LoRA's stored parameters, yet achieves superior results. Our source code is available at https://github.com/leo-yangli/VB-LoRA . This method has been merged into the Hugging Face PEFT package 2 .

1 Introduction

Parameter-efficient fine-tuning (PEFT) casts a new paradigm that leverages strong prior knowledge built in foundation models and adapts them to a wide range of downstream tasks by updating a small amount of trainable parameters [He et al., 2021]. Compared to prefix/prompt tuning [Li and Liang, 2021, Lester et al., 2021] or in-context learning [Brown et al., 2020], fine-tuning a large-scale pre-trained model yields better domain specialization dictated by high-quality datasets [Brown et al., 2020, Liu et al., 2022, Zhao et al., 2023]. This process can be repeated to suit the needs of ever-changing deployment scenarios and personalizations. However, the sheer volume of parameter space across a multitude of instantiations [Sheng et al., 2023] poses challenges for storage, transmission, and computation, especially for low-resource hardware and consumer-grade networks [Borzunov et al., 2024].

10 4

10 5

10 6

# of stored parameters

65

66

67

68

69

VB-LoRA (Ours)

VeRA

Tied-LoRA

LoRA

To mitigate these challenges, various PEFT methods have been proposed by adding or adapting a small amount of trainable parameters per task without sacrificing performance [Houlsby et al., 2019,

Shihao Ji ∗

sub-vector

#2

Logits

Topk-softmax

Multi-Head

Attention

Q

K

V

Feed-Forward

O

Add & Norm

Add & Norm

W

down

W

up

Vector Bank

Logits

for sub-vector #1

Top-K Softmax

+

W

pretrained

LoRA

A

B

⨂

⨂

sub-vector

#1

⊕

Select

Tile

🔥

🔥

❄

...

💾

💾

sub-vector

#8

💾

Select and pool

🔥

Trainable parameters

❄

Frozen parameters

💾

Stored parameters

Modules

Q

K

V

O

W

down

W

up

Vector Bank

Layers

Layer 1

Layer 2

...

Pool

Sub-vectors

A

B

Karimi Mahabadi et al., 2021, Ding et al., 2023]. These methods exploit the dependencies among model parameters to reduce the redundancy. For example, Hu et al. [2021] propose the low-rank adaptation (LoRA) to approximate the accumulated gradient update for self-attention modules, and induces the intra-matrix parameter coupling. Renduchintala et al. [2024] further study the options of allowing the inter-matrix parameter sharing via weight tying across all the layers. In both cases, the number of trainable parameters is reduced significantly. These two methods stand at the two extremes of spectrum in deciding the range of model components reuse (locally or across-layers) and designating which low-rank matrices needs to be shared and updated. However, as the model size increases and the demand for user-customized models across various services rises, the expense of storing and transmitting the customizations for each combination escalates and emerges as a critical issue. Hence, investigating PEFT methods with significantly smaller number of trainable parameters has attracted a flurry of research interests [Kopiczko et al., 2024, Renduchintala et al., 2024].

This paper introduces VB-LoRA, extreme parameter-efficient fine-tuning with vector banks based on a simple yet effective "divide-and-share" paradigm. We push the limits of LoRA parameter efficiency by breaking the two barriers of low-rank decomposition: (1) locally within each module and each layer, and (2) only across the two original matrix dimensions (without division; see Sec. 3.2 for details). We argue that the parameters across different modules and layers can be shared, and thus the redundancy in parameters can be further reduced. In addition, by partitioning rank-one component vectors into sub-vectors, we introduce "virtual" dimensions such that deep structure in the parameter space can be represented by a highly compressed matrix factorization.

VB-LoRA draws inspirations from previous line of work on quantized tensor networks [Oseledets, 2010, Cichocki, 2014] in breaking the constraint of physical dimension for extreme parameter compression. Specifically, VB-LoRA reparameterizes LoRA's low-rank adaptation by a rank-one decomposition and then divides the resulting vectors into sub-vectors of the same size. A global sharing mechanism is then learnt based on a sparse topk admixture module. The same sized subvectors allows parameters to be shared across modules and layers at the sub-vector level. Moreover, compared to the post-hoc matrix compression methods [Oseledets, 2010, Khoromskij, 2011], VBLoRA is end-to-end differentiable, and therefore the fine-tuning process is aware of the compressed form, enabling task-oriented compression. Figure 1 illustrates the parameter efficiency of VB-LoRA as compared with state-of-the-art PEFT methods. Our contributions are summarized as follows:

We introduce a "divide-and-share" paradigm that breaks the barriers of low-rank decomposition across matrix dimensions, modules, and layers by sharing parameters globally via a vector bank.

We reparameterize LoRA's low-rank decomposition by a rank-one decomposition, and divide the resulting vectors further into sub-vectors of the same size, enabling extreme parameter efficiency at the sub-vector level.

We propose a sparse topk module based on the admixture model to learn a global sharing mechanism, making our framework end-to-end differentiable and compression-aware.

Our method achieves extreme parameter efficiency while maintaining comparable or better empirical performance compared to the state-of-the-art PEFT methods on natural language understanding, natural language generation, instruction tuning, and mathematical reasoning tasks.

2 Related Work

Exploit Global Redundancy for Enhanced Parameter Efficiency The parameters of deep neural networks (DNNs) can be naturally divided by layers, heads, or types (MHA or FFN). While LoRA [Hu et al., 2021] only exploits the intra-matrix dependency, Tied-LoRA [Renduchintala et al., 2024] employs a simple weight tying scheme on the low-rank matrices A and B across layers to reduce the inter-matrix redundancy. When A and B are randomly initialized, frozen, and shared across all layers, Tied-LoRA degenerates to VeRA [Kopiczko et al., 2024], which only requires two scaling vectors to be updated, leading to impressive parameter efficiency. A concurrent work, LoRA-XS [Bałazy et al., 2024], further improves the parameter efficiency of LoRA by introducing small trainable matrices between frozen LoRA projection matrices, which are initialized using Singular Value Decomposition (SVD) of the pretrained module weights. Our VB-LoRA pushes the limits of LoRA parameter efficiency by sharing parameters globally across modules and layers at the sub-vector level.

On the low-dimensional reparameterization, Aghajanyan et al. [2020] empirically show that there exists a low-dimensional reparameterization that is as effective for fine-tuning as the full parameter space. The actualization of the random projection is achieved through the Fastfood transform [Le et al., 2013] for large-scale pre-trained language models. To make it structure-aware, a set of layerwise scaling parameters are included as part of the training parameters. Following this intuition, we study the lightweight fine-tuning within LoRA based on the customized reparameterization that arises from the rank-one matrix decomposition.

Moreover, tensor decomposition has been leveraged for PEFT in ViT models [Jie and Deng, 2023] based on classical formats, such as tensor-train or Tucker [Kolda and Bader, 2009]. We find that forcing multilinear decomposition across multiple modes results in a higher rank number, which is detrimental to the objective of parameter compression. An indirect comparison of VB-LoRA to Jie and Deng [2023] can be conducted by referring the compression rate to LoRA. From this perspective, our VB-LoRA can be viewed as a customized tensor format endowed with a convex geometry structure, which is enabled by the sparse topk admixture model we proposed.

Compared to the deep fusion approach [Mazzawi et al., 2024] where LLM parameters are split and initialized using pre-trained smaller networks under a designed network growth mechanism, our parameter division operates on the rank-one component vectors. Sub-vector division allows for similar extensions to leverage pre-trained vector bank initializations from smaller models and distributed training using model parallelism.

Parameter Modeling based on Sparse Admixture Models Admixture models have been widely used in population genetics [Pritchard et al., 2000], topic modeling [Reisinger et al., 2010, Inouye et al., 2014], and hyperspectral unmixing [Li and Bioucas-Dias, 2008, Fu et al., 2015] to extract archetypal (or endmember) components from observed data. The archetypal components can be relaxed to have mixed sign [Ding et al., 2008] with identifiability guarantees [Lin et al., 2015]. Conventionally, parameters estimation are conducted based on linear programming [Chan et al., 2009] or combinatorial algorithms [Arora et al., 2013]. However, an involved integer programming problem arises when incorporating an extra topk constraint into the mixing weights that is especially challenging for the large-scale language models. In this work, we propose learning archetypal vector banks not from observed data but from model parameters of LLMs. By modifying the sparse topk module [Shazeer et al., 2016] commonly used in Mixture-of-Expert models [Jiang et al., 2024], the mixing weights and vector banks are optimized by back-propagation under the objective of downstream fine-tuning tasks. The proposed topk admixture model is model-agnostic in the sense that it can be readily integrated into any neural network parameters or accumulated gradient updates.

3 Proposed Method

3.1 Preliminaries: Transformer Architecture and LoRA Adapters

The transformer architecture [Vaswani et al., 2017] consists of L layers, each containing two types of blocks: Multi-Head Attention (MHA) and Feed-Forward Network (FFN). We denote the query, key, value, and output matrices of MHA at layer ℓ as W ℓ t = { W i t } N h i =1 , t ∈ { q, k, v, o } , where W i t ∈ R d × d , and N h is the number of heads. Given FFN ( x ) = W downReLU ( W up x ) with x ∈ R d , viewing FFN as a multi-head operation, we further divide W up ∈ R cd × d and W down ∈ R d × cd into c matrices of size d × d , denoted by W ℓ up = { W ℓ,i up } c i =1 and W ℓ down = { W ℓ,i down } c i =1 . c = 4 .

Given a pre-trained matrix W 0 ∈ R m × n , LoRA [Hu et al., 2021] constrains the weight increments ∆ W as a low-rank decomposition ∆ W = BA , where B ∈ R m × r , A ∈ R r × n are trainable parameters, with r ≪ min( m,n ) . VeRA [Kopiczko et al., 2024] further limits the trainable parameters to two scaling vectors b and d , which form the diagonal elements of two diagonal matrices Λ b and Λ d . Hence, VeRA can be expressed as ∆ W = Λ b B Λ d A , where B and A are randomly initialized, frozen and shared across layers.

Collectively, we denote the model parameters of transformer as Ω = {{ W ℓ q , W ℓ k , W ℓ v , W ℓ o } ∪ { W ℓ up , W ℓ down }} L ℓ =1 ∈ R 12 L × d × d . In the sequel, we propose a global reparameterization on the weight increments of W ∈ Ω based on the LoRA decomposition ∆ W = BA . we will show how extreme parameter efficiency can be achieved by (1) parameter sharing across matrix dimensions of A and B based on a rank-one decomposition and sub-vector partitions (Sec. 3.2), and (2) across modules and layers regardless of the index or matrix type (Sec. 3.3).

3.2 Divide-and-Share: a New Paradigm for Parameter Sharing

The low rank decomposition of LoRA can be equivalently expressed in a rank-one form as follows:

where ⊗ denotes the outer product operator and v ( i ) k is a vector of size d i .

Divide Based on the rank-one decomposition above, we further represent each component vector v ( i ) k as a concatenation of a set of sub-vectors,

where { d i } i =1 , 2 represents the size of the matrix dimension of ∆ W . In general, { d i } i =1 , 2 are not equal across A and B , and we choose b as a common factor of d i such that d ′ i = d i /b and d ′ i ∈ Z .

Share To facilitate parameter sharing across model dimensions, we assume each sub-vector u ( i ) k,j as a topk admixture of basic elements from vector bank B = { α 1 , . . . , α h } , where α i ∈ R b for i ∈ { 1 , . . . , h } , and is defined as follows (with the subscripts omitted for clarity):

where TopK( σ , k ) i = σ i if σ i is among the topk of σ and TopK( σ , k ) i = -∞ otherwise. For each sub-vector u , we introduce logits σ ∈ R h as its learnable parameters. We call the model expressed in Eq. 3.3 as the topk admixture module (TKAM), which is differentiable. This design enables the joint learning of vector bank B and logits σ in an end-to-end manner, which is amenable for model fine-tuning to the downstream tasks.

The TKAM module promotes sparsity by selecting k vectors of the largest logits from the vector bank. By setting k ≪ h , we restrict the sub-vector u to be sparse. That is, in each iteration, the updates to the vector bank remain locally dominated - with at most k basis vectors α ∈ B affected by the backpropagation through u - in the hope that the learnt vectors can be more specialized and the knowledge encapsulated in the vector bank can be activated and updated sparsely.

Noise-free Topk module The Noisy Topk Gating module [Shazeer et al., 2016] has been widely used to replace the fully connected layers with the Mixture of Experts (MoE) layers in large language models [Jiang et al., 2024]. In contrast, we use Eq. 3.3 to learn the selective sharing scheme across the rank-one component vectors without changing the original model. Due to the decomposition, we find that the cumulative gradient parameter updates are more sensitive than the original model parameters during the training process. This may be related to the training instability issues observed in hypernetworks [Ortiz et al., 2024], where parameters are generated by another parameterized model as well. Therefore, keeping zero noise in the gating function can help make the learning more efficient and stable. An ablation study of different vector selection methods, including Gumbel-softmax, is provided in Sec. 4.5.

3.3 Breaking Boundaries of LoRA for Global Parameter Sharing

While LoRA only applies the low rank decomposition to each individual weight increment, the boundary can be broken by the divide-and-share scheme we proposed in Sec. 3.2. Our divideand-share approach can be interpreted as hierarchical and constrained tensor decomposition, which facilitates efficient global parameter sharing that goes beyond LoRA's low-rank representation of matrices.

The divide operator was first introduced in Quantized Tensor Train (QTT) for super compression of large-scale matrices [Oseledets, 2010, Cichocki, 2014]. For example, dyadic division reshapes a vector of length L = 2 p into a p -dimensional array which facilitates the efficient Tensor Train decomposition to be used. Our divide operator instead applies to the rank-one component vectors v ( i ) k , and the resulting hierarchical tensorial representation of ∆ W can be viewed as a Canonical Polyadic Decomposition (CPD) [Kolda and Bader, 2009] with component vectors v ( i ) k folded into 2 -dimensional arrays with sub-vectors u ( i ) k,j as columns. Each sub-vector u i is composed from a globally shared vector bank B via TKAM, where i = [ j , v ] is a multi-index including physical indices j , such as module, layer, head, and left/right decomposed matrix, and virtual indices v (created from vector partition).

The share operator (TKAM module) can be viewed as a factor model with simplex constraints on the mixing weight (e.g., k = 2 , the sub-vector u lies on the edges of the simplex) and common factors stored in B . Let u ∈ R b and u = ∑ h s =1 α s w s , where α s is the s -th factor, and w is the factor score for the sub-vector u . We consider the following options for w : (1) Admixture (convex combination): w ∈ [0 , 1] h and ∑ h s =1 w s = 1 , which is commonly used in various communities. (2) Sparse Admixture (TKAM): w ∈ [0 , 1] h and ∑ h s =1 w s = 1 with only k ≪ h non-zero elements allowed. It's worth mentioning that adding the multi-index information to the vector selection mechanism can make the TKAM model structure-aware, potentially yielding additional benefits. One possibility is to make the logits of vector selection conditional on the embeddings of the layer, module, and matrix type, which can be implemented through a hypernetwork [Mahabadi et al., 2021]. However, we leave this for future work.

In summary, LoRA provides a local low-rank factorization for each d 1 × d 2 matrix ∆ W independently. In contrast, our VB-LoRA introduces a global low-rank factorization on a b ×|{ i }| matrix composed of partitioned rank-one vectors, where |{ i }| denotes the cardinality of the index set including both physical and virtual indices. As we will see below, this differentiation can better leverage the redundancy in the cumulative gradients, leading to extreme parameter efficiency.

Figure 2 overviews our method. The left section demonstrates the high-level idea of VB-LoRA: the vector bank is shared across sub-vectors, modules, and layers. The right section details its architecture. To form each sub-vector, we use a topk softmax function to select k vectors from the vector bank, which are then pooled into a sub-vector. These sub-vectors are arranged in the desired positions, forming the parameters for LoRA with negligible computational overhead. Algorithm 1 provides the PyTorch-like pseudocode for VB-LoRA, which can be seamlessly integrated into the PyTorch framework.

3.4 Parameter Count

In full fine-tuning, the number of trainable parameters is equal to the model size, i.e., LMd 2 , where L is the number of layers, M is the number of fine-tuned modules, and d is hidden dimension.

Algorithm 1 Pseudocode of VB-LoRA in a PyTorch-like style

LoRA reduces this number to 2 LMdr , while VeRA further reduces it to LM ( d + r ) . The trainable parameters of LoRA and VeRA are the same as the parameters they need to store.

In VB-LoRA, the trainable parameters consist of two parts: the parameters of the vector bank B and the parameters of logits σ . However, at the end of training, the logit parameters can be discarded and only the k selected indices and the topk admixture weights need to be stored. Therefore, the stored parameters can be represented by a triplet Θ = {B , I , V} , where B ∈ R h × b is a vector bank containing h vectors of b -dimensional, I ∈ R 2 × L × M × r × ( d/b ) × k is the topk indices of the vectors in B for all sub-vectors, and V ∈ R 2 × L × M × r × ( d/b ) × ( k -1) is the topk admixture weights used to composite the sub-vectors from the bank. It is worth noting that the topk admixture weights have only k -1 degrees of freedom since they must be summed to 1. Additionally, depending on the size of the vector bank h , the indices I can be efficiently stored as unsigned integers (e.g., uint8 when h ≤ 256 ), and hence, we count the number of parameters as the float32-equivalent size for a fair comparison. When we use k = 2 and uint8 for indices, the number of stored parameters of VB-LoRA is hb +3 LMr ( d/b ) . Unlike LoRA and VeRA, the number of parameters in VB-LoRA does not increase linearly with the model size (determined by L and d ) or the number of fine-tuned modules, i.e., M . While the second term of VB-LoRA's parameters is a linear function of LMd , the coefficient is 3 r/b , which is typically very small. For example, in our experiments, the typical values are r = 4 and b = 256 , leading to a coefficient of 0.04, whereas the coefficient is 2 r for LoRA and 1 for VeRA. Most of the parameters in VB-LoRA reside within the shared vector bank, whose size does not increase linearly with the model size or number of fine-tuned modules.

4 Experiments

In this section, we conduct a comprehensive evaluation of our method through a series of experiments. We begin by comparing VB-LoRA to the state-of-the-art PEFT methods: LoRA, VeRA, and TiedLoRA on the GLUE benchmark. Next, we extend our analysis to natural language generation tasks using GPT-2, instruction tuning tasks on the Llama2, as well as mathematical reasoning tasks on Mistral and Gemma models. All our experiments were conducted on a server equipped with 8 NVIDIA A100 GPUs. For reproducibility, we provide detailed hyperparameters and specifications of computing resources for each experiment in the appendix. The source code is available at https://github.com/leo-yangli/VB-LoRA .

4.1 Natural Language Understanding

We adopt the General Language Understanding Evaluation (GLUE) benchmark 3 [Wang et al., 2018] to assess the performance of VB-LoRA across various natural language understanding tasks, including

similarity, paraphrase, and inference tasks. Following Kopiczko et al. [2024], we focus on six tasks from GLUE: CoLA [Warstadt et al., 2019] (linguistic acceptability), SST-2 [Socher et al., 2013] (sentiment analysis), MRPC [Dolan and Brockett, 2005] (paraphrase detection), STS-B [Cer et al., 2017] (semantic textual similarity), QNLI [Rajpurkar et al., 2018] (inference), and RTE (inference).

Our experiments are performed with RoBERTabase and RoBERTalarge [Liu et al., 2019]. While LoRA and VeRA only finetune the query and value modules, we explore two fine-tuning strategies: query and value only (VB-LoRAqv), and all linear modules (VB-LoRAall), including W q , W k , W v , W o , W up, and W down. We create a vector bank of 90 vectors of a length of 256, initialized with a uniform distribution U ( -0 . 02 , 0 . 02) . The logits are initialized with a normal distribution N (0 , 0 . 01) . The learning rates for the vector bank and logit parameters are set to 0.001 and 0.01, respectively. We set the rank to 4 and k = 2 for all our experiments.

Table 1 reveals that VB-LoRA achieves competitive or superior performance compared to VeRA and Tied-LoRA, while being more parameter efficient. For example, when fine-tuning the query and value modules on the RoBERTalarge model, our method reduces the stored parameters to less than 40% of those required by VeRA or Tied-LoRA, while outperforming them across all tasks. These results suggest that model performance depends not only on the quantity of trainable parameters but also on how they are composed.

Moreover, the results consistently indicate that fine-tuning all modules, beyond just the query and value modules, enhances performance for all the methods. However, LoRA, VeRA and Tied-LoRA requires 2-4 times of the parameters in this case because their parameter counts increase linearly with the number of fine-tuned modules. In contrast, our method uses only 37.5% additional parameters as we maintain the same vector bank size but add additional parameters for indices and topk weights. Thus, with only 12.8% of the parameters compared to VeRAall (4% compared to LoRAqv), our method achieves the best average performance.

4.2 Natural Language Generation

For natural language generation experiments, we fine-tune the GPT-2 Medium and Large models [Radford et al., 2019] on the E2E dataset 4 [Novikova et al., 2017], which contains approximately 42,000 training examples, 4,600 validation examples, and 4,600 test examples from the restaurant domain. We use a vector bank of size 256 for GPT-2 Medium and 350 for GPT-2 Large. The vector length is set to 256 and the rank is set to 4 for both models. To achieve the best performance, we fine-tune all attention layers and FFN layers. As shown in Table 2, our approach achieves competitive performance compared to VeRA, while requiring about 20% less stored parameters for both models.

4.3 Instruction Tuning

Instruction tuning is a process of fine-tuning model with a set of instructions or prompts to enhance its performance on specific instructions [Ouyang et al., 2022]. We first experiment on a general instruction tuning dateset. We use the Cleaned Alpaca Dataset 5 , which improves the data quality of the original Alpaca dataset [Taori et al., 2023]. We evaluate the fine-tuned models on the MTBench 6 [Zheng et al., 2024], which contains 80 multi-turn questions.

Following Kopiczko et al. [2024], we fine-tune the Llama2 model [Touvron et al., 2023] within the QLoRA [Dettmers et al., 2023] framework 7 , which aims to reduce memory usage when finetuning large language models on a single GPU. We utilize the quantization strategy provided by QLoRA, including 4-bit NormalFloat for storage data, BFloat16 for computation parameters, double quantization and paged optimizers to train it on a single GPU. Our fine-tuned models generate responses to these questions, and subsequently, GPT-4 is employed to review and evaluate the generated answers, assigning a quantitative score on a scale of 10. Note that aligning with VeRA, we report the score of the first turn of the conversation. Following Kopiczko et al. [2024], we apply VB-LoRA to all linear layers except the top one. For Llama2 7B, we use a vector bank of 2,048 vectors, each with a length of 256, and the rank is set to 4, resulting in a total of 0.8M stored parameters. For Llama2 13B, we use the same-sized vector bank but increase the rank to 6, leading to 1.1M stored parameters. For all the experiments, we train for one epoch.

The results are reported in Table 3. Notably, we report two sets of LoRA results for each experiment: one from our implementation and the other from Kopiczko et al. [2024], due to a noticeable discrepancy between the scores. Since we closely follow the experimental settings of Kopiczko et al. [2024], we speculate that the difference is due to changes in the GPT-4 model over time. However, comparing the relative improvements of VeRA and VB-LoRA with their respective implementations of LoRA remains fair. VB-LoRA achieves higher scores than LoRA while using only 0.5% (Llama2 7B) and 0.4% (Llama2 13B) of the stored parameters. While VeRA can reach similar scores with their implementation of LoRA, it requires more than twice of parameters compared to VB-LoRA.

4.4 Mathematical Reasoning

To evaluate mathematical reasoning capabilities, we fine-tune the Mistral-7B-v0.1 and Gemma-7B models on the MetaMathQA 8 [Yu et al., 2023] dataset and test them on GSM8K 9 [Cobbe et al., 2021] and MATH 10 [Hendrycks et al., 2021] datasets. We compare our results with the concurrent work LoRA-XS [Bałazy et al., 2024], following its experimental configuration. The result is shown in Table 4. Our method outperforms all baselines on GSM8K, with Mistral-7B utilizing only 0.4% of

Table 4: Results with Mistral-7B and Gemma-7B models on the GSM8K and MATH Benchmarks. Specifically, in VB-LoRA, we use a vector bank size of 2,048 with b = 256 , set the rank to 4, and train with a batch size of 128 for 2 epochs. The warm-up ratio is 0.02, and training uses a cosine learning rate scheduler, with an initial learning rate of 0.001 for the vector bank and 0.01 for the logits. The baseline results are taken from Bałazy et al. [2024].

the parameters compared to LoRA, and Gemma-7B using just 0.3%. Compared with LoRA-XS, our method outperforms on both evaluation datasets while using 70% (Mistral-7B) and 83% (Gemma-7B) of LoRA-XS parameters.

4.5 Ablation Study

We conduct an ablation study to examine the impact of each individual component of VB-LoRA. The experiments are performed on RoBERTa-large, fine-tuning only the query and value modules.

Vector Selection Methods Besides the topk admixture module (abbreviated as Topk below), there exist several commonly used discrete optimization methods for vector selection, including Noisy Topk [Shazeer et al., 2016], Gumbel-Softmax (GS), and Straight-Through Gumbel-Softmax [Jang et al., 2017, Maddison et al., 2016]. For Topk and Noisy Topk , we evaluate the impact of different k to the performances on the CoLA dataset. For GS and Straight-Through GS, we set the temperature τ = 1 / 3 during training and use Top-1 and Top-2 Softmax for inference. Additionally, we explore "Select All", a special case of Topk with k equals to the vector bank size h . As shown in Table 5, Noisy Topk , GS, and Straight-Through GS significantly underperform Topk and "Select All". We hypothesize that random noise injected by these methods likely disrupts the parameters of vector bank, leading to instability in the learning process.

We further investigate the impact of k to the training dynamics and performance of VB-LoRA. As discussed in Sec. 3.4, the choice of k affects not only the model's performance but also the number of parameters to be stored. Hence, a smaller k is generally preferred for improved parameter efficiency. Table 5 shows that k = 2 yields the best result on CoLA, whereas k = 1 performs significantly worse. To explain this, we delve into the training dynamics of VB-LoRA. As shown in Figure 3 (a), when k = 1 , the selected vectors remain largely unchanged during training. In contrast, when k > 1 , the model actively explore the vector bank as illustrated in Figure 3 (b) and (c), i.e., different vectors are selected and updated actively during the training process. Additionally, we observed that this vector exploration primarily occurs in the early stages of training, with updates becoming progressively sparser in later stages, as shown in Figure 5 in the appendix. This suggests that the vectors become increasingly specialized for specific sub-vectors as training progresses.

Sub-vector Length b VB-LoRA introduces a new virtual dimension that divides the original dimensions of LoRA matrices into sub-vectors of length b . Note that b must be a common factor of all hidden dimensions to ensure compatibility across the entire model. However, the optimal value of b is task-specific and requires tuning as a hyperparameter. Theoretically, with a fixed vector bank budget, a larger b reduces the number of vectors in the vector bank, potentially making each vector less specialized. On the other hand, a smaller b increases the number of trainable parameters and complicates the vector selection process. As shown in Table 6, a moderate b = 256 yields the best performance on the CoLA task.

5 Conclusion

This paper introduces a "divide-and-share" paradigm and a differentiable topk admixture module for extreme parameter-efficient fine-tuning with vector banks. Our proposed VB-LoRA achieves the competitive or higher accuracy while using significantly smaller number of stored parameters compared to the state-of-the-art PEFT methods, including LoRA, VeRA, and Tied-LoRA. In addition, VB-LoRA is model-agnostic and applicable to other PEFT methods [Ding et al., 2023], including inserted adapters [Karimi Mahabadi et al., 2021], prompt tuning [Qin et al., 2021], and BitFit [Ben Zaken et al., 2022]. Although VB-LoRA focuses on reducing the storage and transmission costs for LLM fine-tuning, we believe the proposed scheme can be extended to memory-efficient fine-tuning and parameter-efficient pre-training. We leave these for future exploration.

Fine-tuning a pre-trained model requires making design choices about which layers of the model should be frozen or updated. Multitask fine-tuning adds extra complexity about which parameters should be shared or task-specific. Along this line of work, Polytropon [Ponti et al., 2022] jointly learns a small inventory of LoRA adapters and a routing function that selects a variable-sized subset of adapters for few-shot adaptation. Caccia et al. [2023] emphasize the importance of routing granularity and further propose a finer-grained mixing across multiple heads. Following these works, it would be interesting to explore a finer-grained parameter transfer across tasks, heads, types, and layers at the sub-vector level for multitask fine-tuning.

Limitations and broader impacts Our experiments are limited to monomodal (text-based), monolingual (English), and LoRA-only settings. Additionally, our exploration of the vector bank is somewhat limited, as we only examine a small range of configurations for bank size and vector length. In terms of broader impacts, VB-LoRA reduces the storage and transmission costs of LLM adapters and demonstrates improved memory-efficiency, making customized LLMs more accessible. We do not foresee any negative societal impact beyond those generally associated with LLMs.

Acknowledgments

We would like to thank the anonymous reviewers for their comments and suggestions, which helped improve the quality of this paper.

0

20

40

60

80

Sub-vector #

0

10

20

30

40

50

60

70

80

Vector # in the vector bank

(a) When

k

= 1

0

20

40

60

80

Sub-vector #

0

10

20

30

40

50

60

70

80

Vector # in the vector bank

(b) When

k

= 2

0

20

40

60

80

Sub-vector #

0

10

20

30

40

50

60

70

80

Vector # in the vector bank

(c) When

k

= 3

0

20

40

60

80

Sub-vector #

0

10

20

30

40

50

60

70

80

Vector # in the vector bank

(d) Noisy Top-2

References

Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255 , 2020.

Sanjeev Arora, Rong Ge, Yonatan Halpern, David Mimno, Ankur Moitra, David Sontag, Yichen Wu, and Michael Zhu. A practical algorithm for topic modeling with provable guarantees. In International Conference on Machine Learning , pages 280-288. PMLR, 2013.

Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. LoRA-XS: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604 , 2024.

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient finetuning for transformer-based masked language-models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 1-9, Dublin, Ireland, May 2022.

Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin A Raffel. Distributed inference and fine-tuning of large language models over the internet. Advances in Neural Information Processing Systems , 36, 2024.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 33:1877-1901, 2020.

Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni. Multi-head adapter routing for cross-task generalization. In Advances in Neural Information Processing Systems , 2023.

Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens, editors, Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) , pages 1-14, Vancouver, Canada, August 2017.

Tsung-Han Chan, Chong-Yung Chi, Yu-Min Huang, and Wing-Kin Ma. A convex analysis-based minimum-volume enclosing simplex algorithm for hyperspectral unmixing. IEEE Transactions on Signal Processing , 57(11):4418-4432, 2009.

Andrzej Cichocki. Era of big data processing: A new approach via tensor networks and tensor decompositions. arXiv preprint arXiv:1403.2048 , 2014.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 10088-10115, 2023.

Chris HQ Ding, Tao Li, and Michael I Jordan. Convex and semi-nonnegative matrix factorizations. IEEE transactions on pattern analysis and machine intelligence , 32(1):45-55, 2008.

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence , 5(3):220-235, 2023.

William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005) , 2005.

Xiao Fu, Wing-Kin Ma, Kejun Huang, and Nicholas D Sidiropoulos. Blind separation of quasistationary sources: Exploiting convex geometry in covariance domain. IEEE Transactions on Signal Processing , 63(9):2306-2320, 2015.

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations , 2021.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS , 2021.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning , pages 2790-2799. PMLR, 2019.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2021.

David Inouye, Pradeep Ravikumar, and Inderjit Dhillon. Admixture of Poisson MRFs: A topic model with word dependencies. In International Conference on Machine Learning , pages 683-691. PMLR, 2014.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with Gumble-softmax. In International Conference on Learning Representations , 2017.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024.

Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 1060-1068, 2023.

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems , 34:1022-1035, 2021.

Boris N Khoromskij. O (d log n)-quantics approximation of n-d tensors in high-dimensional numerical modeling. Constructive Approximation , 34:257-280, 2011.

Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review , 51(3): 455-500, 2009.

Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In International Conference on Learning Representations , 2024.

Quoc Le, Tamás Sarlós, Alex Smola, et al. Fastfood-approximating kernel expansions in loglinear time. In International Conference on Machine Learning , volume 85, 2013.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 2021.

Jun Li and José M Bioucas-Dias. Minimum volume simplex analysis: A fast algorithm to unmix hyperspectral data. In IGARSS 2008-2008 IEEE International Geoscience and Remote Sensing Symposium , volume 3, pages III-250. IEEE, 2008.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, 2021.

Chia-Hsiang Lin, Wing-Kin Ma, Wei-Chiang Li, Chong-Yung Chi, and ArulMurugan Ambikapathi. Identifiability of the simplex volume minimization criterion for blind hyperspectral unmixing: The no-pure-pixel case. IEEE Transactions on Geoscience and Remote Sensing , 53(10):5530-5546, 2015.

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems , 35:1950-1965, 2022.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations , 2016.

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameterefficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 565-576, 2021.

Hanna Mazzawi, Javier Gonzalvo, Michael Wunder, Sammy Jerome, and Benoit Dherin. Deep fusion: Efficient network training via pre-trained initializations. In Forty-first International Conference on Machine Learning , 2024.

Jekaterina Novikova, Ondˇ rej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-toend generation. In Kristiina Jokinen, Manfred Stede, David DeVault, and Annie Louis, editors, Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue , pages 201-206, Saarbrücken, Germany, August 2017.

Jose Javier Gonzalez Ortiz, John Guttag, and Adrian V Dalca. Magnitude invariant parametrizations improve hypernetwork learning. In The Twelfth International Conference on Learning Representations , 2024.

Ivan V Oseledets. Approximation of 2 d × 2 d matrices using tensor decomposition. SIAM Journal on Matrix Analysis and Applications , 31(4):2130-2145, 2010.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35: 27730-27744, 2022.

Edoardo M Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining modular skills in multitask learning. arXiv preprint arXiv:2202.13914 , 2022.

Jonathan K Pritchard, Matthew Stephens, and Peter Donnelly. Inference of population structure using multilocus genotype data. Genetics , 155(2):945-959, 2000.

Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, et al. Exploring universal intrinsic task subspace via prompt tuning. arXiv preprint arXiv:2110.07867 , 2021.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 784789, Melbourne, Australia, July 2018.

Joseph Reisinger, Austin Waters, Bryan Silverthorn, and Raymond J Mooney. Spherical topic models. In International Conference on Machine Learning , pages 903-910. Citeseer, 2010.

Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-LoRA: Enhancing parameter efficiency of LoRA with weight tying. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages 8686-8697, 2024.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations , 2016.

Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. S-LoRA: Serving thousands of concurrent LoRA adapters. arXiv preprint arXiv:2311.03285 , 2023.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages 1631-1642, Seattle, Washington, USA, October 2013.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research , 9(86):2579-2605, 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupała, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, Brussels, Belgium, November 2018.

Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics , 7:625-641, 2019.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284 , 2023.

Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Li Yun, Hejie Cui, Zhang Xuchao, Tianjiao Zhao, et al. Domain specialization as the key to make large language models disruptive: A comprehensive survey. arXiv preprint arXiv:2305.18703 , 2023.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024.

A Appendix

A.1 Hyperparameters and Computing Resources

The hyperparameters used for the natural language understanding, natural language generation and instruction tuning are provided in Table 7, 8 and 9. All experiments were conducted on a server equipped with 8 NVIDIA A100 80GB GPUs.

Computation overhead The proposed factorization in VB-LoRA is simple to implement in modern deep learning frameworks such as PyTorch, allowing us to fully leverage GPU acceleration. However, the use of subvector decomposition does introduce some computational overhead. This additional overhead is limited to the training phase and does not affect inference, as both LoRA and VB-LoRA merge their parameters back into the original model parameters during this stage.

Memory efficiency Despite the training time overhead, the reduced number of trainable parameters in VB-LoRA results in lower memory consumption. During LoRA fine-tuning, the forward pass is z = Ax , H = Bz , without the need to materialize ∆ W . This memory-saving technique can be seamlessly incorporated in VB-LoRA and has been implemented in our source code. Table 9 shows that VB-LoRA requires approximately 15%-20% more training time than LoRA, while it consumes less memory than LoRA in both the LLaMA2 7B model and LLaMA2 13B models.

A.2 Visualization of the Vector Selection

For visualization, we conducted experiments on the CoLA dataset using a 24-layer RoBERTa-large model with a vector bank of 30 vectors. We fine-tuned the query and value modules, setting the rank to 2 and the vector length to 1024, resulting in 192 sub-vectors.

Figure 4 displays the vectors selected by sub-vectors at the initialization (red) and at the end of training (blue), respectively. As we can see, most of the final selections differ from the initial selections, demonstrating the training dynamics of the vector selection process.

In Figure 5, we plot the footprint at different training periods. This visualization demonstrates that vector exploration predominantly occurs in the early stages of training, and the updates become progressively sparser in the later stages of training.

Figure 6 illustrates the sum of the top-k weights for each vector, grouped by the first, middle, and last 8 layers. It shows that certain vectors are favored by deeper layers, such as vectors #1 and #29, while some are favored by shallower layers, such as vectors #20 and #26.

We then group the same data with respect to query and value modules, as well as matrices A and B, shown in Figure 7. As we can see, some vectors are predominantly utilized by specific module or matrix types. For instance, vector #23 is heavily utilized in the formation of matrix A, while vector #29 is predominantly used in the formation of Query modules.

Load balancing To demonstrate that the vector selection is free from load balancing issue, we present the vector usage in a Gemma-7B model trained on the MetaMathQA dataset, as shown in

0

25

50

75

100

125

150

175

0

5

10

15

20

25

Vector # in the Vector Bank

Initially selected

Finally selected

Overlapped

Sub-vector #

0

25

50

75

Sub-vector #

0

20

40

60

80

Vector # in the vector bank

(a) Epoch 0-4

0

25

50

75

Sub-vector #

0

20

40

60

80

Vector # in the vector bank

0

25

50

75

Sub-vector #

0

20

40

60

80

Vector # in the vector bank

0

25

50

75

Sub-vector #

0

20

40

60

80

Vector # in the vector bank

0

25

50

75

Sub-vector #

0

20

40

60

80

Vector # in the vector bank

0

25

50

75

Sub-vector #

0

20

40

60

80

Vector # in the vector bank

(f) Epoch 25-29

A.3 Visualization of the Vector Bank and the Sub-vectors

Figure 9 illustrates the positioning of the sub-vectors along the edge of the simplex spanned by the vector bank. The vector bank is projected into a 2-D space using T-SNE [van der Maaten and Hinton, 2008] for visualization.

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

Vector #

0

1

2

3

4

5

6

7

Sum of the top-k weights

First 8 layers

Middle 8 layers

Last 8 layers

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

Vector #

0

2

4

6

8

Sum of the top-k weights

VA

VB

QA

QB

30

40

50

60

70

Frequency of vector usage

0

20

40

60

80

100

120

Number of vectors

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

Vector Bank

Sub-vectors

A.4 Instruction-tuning Examples

Tables 10 and 11 provide examples of the responses generated by the finetuned Llama2 13B model with LoRA and VB-LoRA.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction include the claims made in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed "Limitations" in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully disclosed all the information needed to reproduce the main experimental results, and we also provide the source code for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the source code in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setting is clearly stated in the main paper and the hyperparameters are listed in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our main experiments were repeated 3 to 5 times, and we report the standard deviation of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide related information in the main paper and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential societal impacts in Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No data or models are released as part of this work, so safeguards for responsible release are not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original paper and provide the name of the license for the datasets we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach to decrease the number of parameters in LoRA by introducing a collection (aka \"vector bank\") of sub-vectors and composing Lora matrices across modules/layers using this collection. Each sub-vector within a given LoRA module/layer is then formed as a linear interpolation of sub-vectors from the vector bank, where the coefficients are obtained by selecting top-k \"basis vectors\" and then applying a softmax. When applied to Roberta-large, the approach yields improvements over LoRA on the GLUE benchmark with only a tenth of the free parameters. When applied GPT-2 large , it obtains similar or slightly better performance than LoRA with around a sixth of the parameters, on the E2E task. The paper reports ablation studies showing the effect of the vector selection strategy (i.e. alternatives to top-k) and the sub-vector length.\n\nSTRENGTHS:\n* Proposes a novel approach to reduce the parameter count in LoRA by sharing parameters across modules and layers using a sub-vector approach. (Originality)\n* The use of sub-vector style decomposition is novel in the context of LoRA and could be applied in other PEFT settings (Significance).\n* Demonstrates that the approach can yield improvements over LoRA using Roberta or GPT-2 with far fewer parameters (Quality)\n\nWEAKNESSES:\n* The paper does not discuss if the proposed factorization approach (i.e. using subvectors) can be performed efficiently on accelerators such as GPUs. Also it would be good to know what is the overhead resulting from this subvector decomposition.\n* It would be useful to report the performance of an alternative PEFT scheme such as prefix tuning (or prefix tuning) or adapters which have a similar number of parameters as the proposed approach.\n\nQUESTIONS:\n* It would be useful to discuss this paper in related work: https://openreview.net/pdf?id=qKQu1ZcJjD : The idea is to initialize matrices in an LLM using pre-trained submatrix components, which bears resemblance to the subvector decomposition.\n\nTypos:\nL101:\n* parameters -> parameter\nL192\n* numebr -> number\nL314\n* frozen -> frozen\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an extremely parameter-efficient methods, VB-LoRA, for finetuning an LLM. Specifically, VB-LoRA has a shared vector bank (similar to a codebook), the adapter parameters (A and B) of the linear layers in an LLM are constructed from this bank by selecting the most effective bank vectors. Due to the shared mechanism, the number of trainable parameters are very little, much fewer than the strong baselines, like LoRA, VeRA and Tied-LoRA.\n\nTha authors also validate VB-LoRA on three benchmarks (GLUE, E2E benchmark and MT-Bench) on 6 LLMs from three model families (RoBERTa, GPT-2 and Llama2). The results show that VB-LoRA's results are better or comparable to above-mentioned baselines while requiring much fewer number of trainable parameters.\n\nSTRENGTHS:\n1. The proposed method, VB-LoRA, is effective, intuitive and easily applied.\n2. The number of trainable parameters are very limited, mostly less than 0.1M.\n\nWEAKNESSES:\n1. The paper is not easy to follow, especially Section 3 for the proposed method. VB-LoRA is basically an algorithm inspired by Mixture-of Expert or codebook, and is used for PEFT here. However, the authors seem to intend to make it complexer by relating VB-LoRA to other theoretical works (L137-143, L157-162), which would be better by putting in the related works. Such writing style easily distracts the reader's attention. \n\n2. Some experimental setting is weird, not in line with previous work.\n- In Table 1, two high-resource tasks of GLUE, i.e. MNLI and QQP, are deleted. The authors claim that the experiments are conducted on  a server equipped with 8 A100 GPUs (L215), which means the computation resource should not be a problem. Why are these two tasks are discarded?\n- All ablation studies are conducted on a very small task, CoLA, whose results have large variance. Normally, 24GB GPU memory and less than 1 hour are enough for finetuning RoBERTa on CoLA. Due to the small number of samples in CoLA, its results have relatively larger variance. Since the authors have much more resource, why choosing this task for ablation study?\n\n3. VB-LoRA has a potential drawback that is inherent in its design, i.e. unscalibility. One big problem for MoE is that not all expert embeddings can be well trained. We normally include a balance loss and use a small number of experts to avoid this problem. For VB-LoRA, could you offer this ablation study? I.e. increasing the number and length of the bank vectors.\n\n4. VB-LoRA doesn't consistently outperform baselines, or the gap is very narrow. I would be interested in more benckmarks on larger LLMs, like commonsen/arithmetic reseasoning tasks.\n\nQUESTIONS:\n1. In algorithm 1, if you implement delta_W in this way, it will comsume more GPU memory during finetuning than LoRA. For implementing LoRA, the forward pass is z = Ax, H = Bz. It means we don't calculate delta_W explicitely. If you calculate delta_W, it will be stored in the memory for gradient computation, which enlarges the memoy for activation. \n\n2. Check other questions in Weakness.\n\nLIMITATIONS:\nThe limitation discussion is not thorough.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores parameter-efficient fine-tuning (PEFT) methods in the context of further reducing the number of fine-tunable parameters, even to the extreme. The main idea is to reduce the number of parameter within LoRA modules as much as possible while maintaining or even improving fine-tuning performance. This line of work is a sub-area of PEFT research, where similar ideas have been explored in the past (e.g., the work of Karimi Mahabadi et al. on Compacter architectures, or a very recent work concurrent to this one: LoRA-XS). \n\nTo this end, the authors propose VB-LoRA which aims to use globally shared vector banks comprising 'constituent' sub-vectors that can be recombined into local 'delta' parameters (of the \\Delta W matrix) that modify/adapt the weights of the original model. The VB-LoRA is then evaluated on GLUE with RoBERTa (for NLU), E2E with GPT-2 (for generation), and Llama 2 on MT-Bench (for instruction tuning) to show its benefits in terms of paramater efficiency and performance.\n\nSTRENGTHS:\n- The work is well situated and shows excellent awareness of other work in this subarea which also partially inspired the proposed VB-LoRA method. The chosen baseline models are largely adequate and the only baseline missing (to the best of my knowledge) is LoRA-XS (which is concurrent work, so it couldn't have been included anyhow).\n\n- The method is well motivated from a conceptual level and it is also well described formally, in a succinct and convincing manner. The idea does resemble matrix factorisation and tensor product-inspired methods a lot, so I would like to see additional links to previous literature here.\n\n- I appreciate the fact that the authors aimed to provide a comprehensive evaluation over NLU, NLG as well as instruction-tuned models. However, this approach has traded some depth of evaluation for its breadth (and the evaluation is not entirely adequate and comprehensive - see under \"Weaknesses\" later).\n\n- A careful derivation related to the number of fine-tuned parameters across different methods is a very useful addition to the paper, given the fact that the parameter efficiency-performance balance is the main topic of the paper.\n\n- The paper is very clear and well written in general.\n\nWEAKNESSES:\n- Evaluation is not comprehensive: \n1) When it comes to instruction-tuned models, the main findings are based on a single model from a single family (Llama 2), fine-tuned on a single instruction tuning dataset and evaluated on a single evaluation dataset. This is simply enough to move from anecdotal to more generalisable findings, and experiments with (i) additional models, (ii) more instruction tuning datasets (e.g., Flan-v2, Tulu v2, there are more), and (iii) additional evaluation sets (e.g., MMLU, GSM, BBH, HumanEval) are required.\n2) The same holds for NLU and NLG experiments. NLU is evaluated only on GLUE (where it's widely established that performance on GLUE is already saturated and considered 'superhuman'). For NLG, only experiments with a single model on a single benchmark are run. As I mentioned above, the experiments are broad, but stay quite shallow, and this has to change in the revised version.\n\n- The paper is very quantitatively driven, but it doesn't delve into providing clarifications and explanations on why certain quantitative phenomena/findings are observed:\n1) What is the intuition behind VB-LoRA outperforming LoRA? How can we explain this? What makes VB-LoRA even more suitable performance-wise? Isn't it counter-intuitive that an extremely efficient model can outperform here? Are the tasks then too simple (with very low intrinsic dimensionality) as is the case with some of the standard GLUE tasks?\n2) How can one fine-tune the optimal VB size (b) and dimensionality of subvectors within the VB (Table 5) - is this task specific or model specific? What would happen if one increases parameter budget a bit - can we expect even better performance or not? In what cases?\n3) Are there any sub-vectors that get selected more often than some others? Why?\n\nOverall, while I'm quite happy regarding technical aspects of the work, the paper can be much improved in terms of evaluation/experiments and discussion of the key findings (some error analysis would be useful as well).\n\nQUESTIONS:\n- Lines 180-182. \"It’s worth mentioning that adding the multi-index information to the vector selection mechanism can make the TKAM model structure-aware, potentially yielding additional benefits.\" This is unclear to me and warrants further clarification/discussion. Could you write a short paragraph on how the multi-index information would be added and what type of structure would then TKAM learn (become aware of)?\n\n- Additional discussion on the similarities and differences between TKAM and standard (sparse and soft) MoE-s is required. Are there any MoE-style paradigms (which incur a higher parameter cost) that could have been used instead of TKAM? In general, the work doesn't really examine the choice of TKAM as a go-to routing method for the selection of sub-vectors.\n\n- Can the authors elaborate more on the similarities and differences between their work and the Compacter work (beyond the fact that Compacter focuses on bottleneck adapters while VB-LoRA focuses on LoRA)? Conceptually, the papers operate in the same space with similar ideas.\n\nLIMITATIONS:\nThere are limitations to this work that should be further elaborated (see also under 'Weaknesses'). The paper didn't explore the whole space of options when it comes to vector selection, VB size, VB content interpretation, and task dependency of findings, among other things. The work also stays within the realm of monolingual (i.e., English-only), mono-modal (i.e., text-only) and mono-PEFT (i.e., LoRA-only) contexts, and this should also be adequately signalled in the Limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a modified version of LoRA called VB-LoRA which is a highly parameter efficient fine-tuning method. It uses a vector bank to represent the model parameters as a composition of vectors. This vector bank is then used to select top-k vectors using the top-k softmax function which are thereby pooled and arranged to form the A and B low-rank matrices of LoRA. The top-k admixture module function selected is a differentiable function so the whole model is trained end-to-end making it more efficient for a particular task. Ultimately the authors show that such as divide-and-share approach results in an orders of magnitude more parameter efficient technique.\n\nSTRENGTHS:\n1. The most significant contribution of this paper is to develop an LoRA-like adaptation technique that is orders of magnitude more parameter efficient than LoRA but at the same time maintaining comparable or better performance in all the tasks. VB-LoRA has the best average performance in all the tasks for every model.\n2. The second most important aspect of VB-LoRA is that the number of parameters do not grow linearly with the model dimension (number of layers and model dimension).  Making the 'k' of top-k much smaller than This is highly valuable as the model sizes are getting increasingly bigger by the day. \n3. The usage of differentiable top-k softmax (TKAM - mentioned in eqn. 1) is a great idea as it enables the paper to be trained fully for a specific task. Also dicing the vector to sub vectors of same size was great for sharing among all layers.\n4. The authors have tried to go to the extreme in making the LoRA parameter count efficient by dividing even the low-rank vectors of LoRA and pooling them from a common vector bank.\n5. The authors also provide adequate supplementary information such as the detailed hyper-parameters and the hardware used which is highly appreciated as it is highly valuable for anyone who wants to replicate these results.\n5. The authors also provide the exact code as a supplementary material which is a huge plus, especially as the paper consists of several moving parts.\n6. The paper is presented in a very high quality way using multiple different concepts and giving proper references wherever needed. This fact would be appreciated by the readers as the paper uses several different concepts to come up with their design such as sparse admixture models, top-k gating module, canonical polyadic decomposition, etc.\n\nWEAKNESSES:\n1. The parameter count of VB-LoRA is defined by: hb + 1.5LMr(d/b). The second term of this is still linearly dependent on the model dimensions (L and M). In the paper's experiments r<<b in which case the second is quite small and doesn't impact the growth in parameter by much. But there exist several adaptation tasks in which much larger ranks are needed. For example this paper (https://ieeexplore.ieee.org/abstract/document/10445894) shows that for domain adaptation large ranks (eg. 64) are much better than smaller ranks. In some of my other works for domain and language adaptation I have needed to use even larger ranks (200+) which are very much comparable to the value of b. In such cases, VB-LoRA's parameter requirement would be very much comparable to other methods.\n\n2. Some parts of the paper are not entirely clear. For eg. in lines 166-169, the paper mentions that decomposed cumulative gradient parameters are more sensitive than the original model parameters during the training process. An intuition to why this is happening would be helpful.\n\n3. In line 153, the paper mentions that by keeping k << h makes the learnt vectors highly specialized. However figure 3 shows that even with k's value as small as 2 and 3, the model updates almost all the vectors during the training process. Inherently this means that almost all the vectors have some activation and they are not very specialized.\n\n4. Not a lot of details are provided for the virtual dimension 'b' and the results of table 5 are not explained.\n\n5. nit: line 167: 'parameters'\n\nQUESTIONS:\n1. Can you provide an intuition to your finding why the cumulative gradient parameters are more sensitive than than the original model parameters and how this affects adaptation?\n2. Have you experimented with larger ranks and how do they look in terms of performance and parameter count?\n3. Have you tried this technique on other tasks than what's listed in the paper? Also have you tried it on other architectures?\n\nLIMITATIONS:\n1. In my view the authors have correctly mentioned that this paper has no larger societal impact beyond what LLMs may have.\n2. Table-3 shows that the performance of LoRA may have improved with the changes in the GPT-4 model over time. But for the other tables the authors have used the old results for several competing models.\n3. I agree with the rest of the answers to the NeurIPS paper checklist that the authors have provided.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}