VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks

Authors: yang  li
shaobo  han
shihao  ji
top-k  softmax
  lora
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:13:56

Abstract: As the adoption of large language models increases and the need for per-user or per-task model customization grows, the parameter-efficient fine-tuning (PEFT) methods, such as low-rank adaptation (LoRA) and its variants, incur substantial storage and transmission costs. To further reduce stored parameters, we introduce a "divide-and-share" paradigm that breaks the barriers of low-rank decomposition across matrix dimensions, modules, and layers by sharing  parameters globally via a vector bank. As an instantiation of the paradigm to LoRA, our proposed VB-LoRA composites all the low-rank matrices of LoRA from a shared vector bank with a differentiable top-$k$ admixture module. VB-LoRA achieves extreme parameter efficiency while maintaining comparable or better performance compared to state-of-the-art PEFT methods. Extensive experiments demonstrate the effectiveness of VB-LoRA on natural language understanding, natural language generation, instruction tuning, and mathematical reasoning tasks. When fine-tuning the Llama2-13B model, VB-LoRA only uses 0.4% of LoRA's stored parameters, yet achieves superior results. Our source code is available at https://github.com/leo-yangli/VB-LoRA. This method has been merged into the Hugging Face PEFT package.

Main paper: 
 related work: 
 Exploit Global Redundancy for Enhanced Parameter Efficiency The parameters of deep neural networks (DNNs) can be naturally divided by layers, heads, or types (MHA or FFN). While LoRA [Hu et al., 2021] only exploits the intra-matrix dependency, Tied-LoRA [Renduchintala et al., 2024] employs a simple weight tying scheme on the low-rank matrices A and B across layers to reduce the inter-matrix redundancy. When A and B are randomly initialized, frozen, and shared across all layers, Tied-LoRA degenerates to VeRA [Kopiczko et al., 2024] , which only requires two scaling vectors to be updated, leading to impressive parameter efficiency. A concurrent work, LoRA-XS [Bałazy et al., 2024] , further improves the parameter efficiency of LoRA by introducing small trainable matrices between frozen LoRA projection matrices, which are initialized using Singular Value Decomposition (SVD) of the pretrained module weights. Our VB-LoRA pushes the limits of LoRA parameter efficiency by sharing parameters globally across modules and layers at the sub-vector level.On the low-dimensional reparameterization, Aghajanyan et al. [2020] empirically show that there exists a low-dimensional reparameterization that is as effective for fine-tuning as the full parameter space. The actualization of the random projection is achieved through the Fastfood transform [Le et al., 2013] for large-scale pre-trained language models. To make it structure-aware, a set of layerwise scaling parameters are included as part of the training parameters. Following this intuition, we study the lightweight fine-tuning within LoRA based on the customized reparameterization that arises from the rank-one matrix decomposition.Moreover, tensor decomposition has been leveraged for PEFT in ViT models [Jie and Deng, 2023] based on classical formats, such as tensor-train or Tucker [Kolda and Bader, 2009] . We find that forcing multilinear decomposition across multiple modes results in a higher rank number, which is detrimental to the objective of parameter compression. An indirect comparison of VB-LoRA to Jie and Deng [2023] can be conducted by referring the compression rate to LoRA. From this perspective, our VB-LoRA can be viewed as a customized tensor format endowed with a convex geometry structure, which is enabled by the sparse top-k admixture model we proposed.Compared to the deep fusion approach [Mazzawi et al., 2024] where LLM parameters are split and initialized using pre-trained smaller networks under a designed network growth mechanism, our parameter division operates on the rank-one component vectors. Sub-vector division allows for similar extensions to leverage pre-trained vector bank initializations from smaller models and distributed training using model parallelism.Parameter Modeling based on Sparse Admixture Models Admixture models have been widely used in population genetics [Pritchard et al., 2000] , topic modeling [Reisinger et al., 2010 , Inouye et al., 2014] , and hyperspectral unmixing [Li and Bioucas-Dias, 2008, Fu et al., 2015] to extract archetypal (or endmember) components from observed data. The archetypal components can be relaxed to have mixed sign [Ding et al., 2008] with identifiability guarantees [Lin et al., 2015] .Conventionally, parameters estimation are conducted based on linear programming [Chan et al., 2009] or combinatorial algorithms [Arora et al., 2013] . However, an involved integer programming problem arises when incorporating an extra top-k constraint into the mixing weights that is especially challenging for the large-scale language models. In this work, we propose learning archetypal vector banks not from observed data but from model parameters of LLMs. By modifying the sparse top-k module [Shazeer et al., 2016] commonly used in Mixture-of-Expert models [Jiang et al., 2024] , the mixing weights and vector banks are optimized by back-propagation under the objective of downstream fine-tuning tasks. The proposed top-k admixture model is model-agnostic in the sense that it can be readily integrated into any neural network parameters or accumulated gradient updates.3 Proposed MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discussed "Limitations" in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: transformer architecture and lora adapters: 
 The transformer architecture [Vaswani et al., 2017] consists of L layers, each containing two types of blocks: Multi-Head Attention (MHA) and Feed-Forward Network (FFN). We denote the query, key, value, and output matrices of MHA at layer ℓ asW ℓ t = {W i t } N h i=1 , t ∈ {q, k, v, o}, where W i t ∈ R d×d , and N h is the number of heads. Given FFN(x) = W down ReLU(W up x) with x ∈ R d , viewing FFN as a multi-head operation, we further divide LoRA [Hu et al., 2021] constrains the weight increments ∆W as a low-rank decomposition ∆W = BA, where B ∈ R m×r , A ∈ R r×n are trainable parameters, with r ≪ min(m, n). VeRA [Kopiczko et al., 2024] further limits the trainable parameters to two scaling vectors b and d, which form the diagonal elements of two diagonal matrices Λ b and Λ d . Hence, VeRA can be expressed as ∆W = Λ b BΛ d A, where B and A are randomly initialized, frozen and shared across layers.W up ∈ R cd×d and W down ∈ R d×cd into c matrices of size d × d, denoted by W ℓ up = {W ℓ,i up } c i=1 and W ℓ down = {W ℓ,i down } c i=1 . c = 4. Given a pre-trained matrix W 0 ∈ R m×n ,Collectively, we denote the model parameters of transformer asΩ = {{W ℓ q , W ℓ k , W ℓ v , W ℓ o } ∪ {W ℓ up , W ℓ down }} L ℓ=1 ∈ R 12L×d×d .In the sequel, we propose a global reparameterization on the weight increments of W ∈ Ω based on the LoRA decomposition ∆W = BA. we will show how extreme parameter efficiency can be achieved by (1) parameter sharing across matrix dimensions of A and B based on a rank-one decomposition and sub-vector partitions (Sec. 3.2), and (2) across modules and layers regardless of the index or matrix type (Sec. 3.3).The low rank decomposition of LoRA can be equivalently expressed in a rank-one form as follows:EQUATIONEQUATIONEQUATIONwhere ⊗ denotes the outer product operator and v(i)k is a vector of size d i .Divide Based on the rank-one decomposition above, we further represent each component vector v(i)k as a concatenation of a set of sub-vectors,EQUATIONwhere {d i } i=1,2 represents the size of the matrix dimension of ∆W . In general, {d i } i=1,2 are not equal across A and B, and we choose b as a common factor of d i such thatd ′ i = d i /b and d ′ i ∈ Z.Share To facilitate parameter sharing across model dimensions, we assume each sub-vector u(i) k,jas a top-k admixture of basic elements from vector bank B = {α 1 , . . . , α h }, where α i ∈ R b for i ∈ {1, . . . , h}, and is defined as follows (with the subscripts omitted for clarity):EQUATIONwhere TopK (σ, k) i = σ i if σ i is among the top-k of σ and TopK (σ, k) i = -∞ otherwise. For each sub-vector u, we introduce logits σ ∈ R h as its learnable parameters. We call the model expressed in Eq. 3.3 as the top-k admixture module (TKAM), which is differentiable. This design enables the joint learning of vector bank B and logits σ in an end-to-end manner, which is amenable for model fine-tuning to the downstream tasks.The TKAM module promotes sparsity by selecting k vectors of the largest logits from the vector bank. By setting k ≪ h, we restrict the sub-vector u to be sparse. That is, in each iteration, the updates to the vector bank remain locally dominated -with at most k basis vectors α ∈ B affected by the backpropagation through u -in the hope that the learnt vectors can be more specialized and the knowledge encapsulated in the vector bank can be activated and updated sparsely.Noise-free Top-k module The Noisy Top-k Gating module [Shazeer et al., 2016] has been widely used to replace the fully connected layers with the Mixture of Experts (MoE) layers in large language models [Jiang et al., 2024] . In contrast, we use Eq. 3.3 to learn the selective sharing scheme across the rank-one component vectors without changing the original model. Due to the decomposition, we find that the cumulative gradient parameter updates are more sensitive than the original model parameters during the training process. This may be related to the training instability issues observed in hypernetworks [Ortiz et al., 2024] , where parameters are generated by another parameterized model as well. Therefore, keeping zero noise in the gating function can help make the learning more efficient and stable. An ablation study of different vector selection methods, including Gumbel-softmax, is provided in Sec. 4.5.While LoRA only applies the low rank decomposition to each individual weight increment, the boundary can be broken by the divide-and-share scheme we proposed in Sec. 3.2. Our divideand-share approach can be interpreted as hierarchical and constrained tensor decomposition, which facilitates efficient global parameter sharing that goes beyond LoRA's low-rank representation of matrices.The divide operator was first introduced in Quantized Tensor Train (QTT) for super compression of large-scale matrices [Oseledets, 2010 , Cichocki, 2014] . For example, dyadic division reshapes a vector of length L = 2 p into a p-dimensional array which facilitates the efficient Tensor Train decomposition to be used. Our divide operator instead applies to the rank-one component vectors v(i)k , and the resulting hierarchical tensorial representation of ∆W can be viewed as a Canonical Polyadic Decomposition (CPD) [Kolda and Bader, 2009] with component vectors v (i) k folded into 2-dimensional arrays with sub-vectors u (i)k,j as columns. Each sub-vector u i is composed from a globally shared vector bank B via TKAM, where i = [j, v] is a multi-index including physical indices j, such as module, layer, head, and left/right decomposed matrix, and virtual indices v (created from vector partition).The share operator (TKAM module) can be viewed as a factor model with simplex constraints on the mixing weight (e.g., k = 2, the sub-vector u lies on the edges of the simplex) and common factors stored in B. Let u ∈ R b and u = h s=1 α s w s , where α s is the s-th factor, and w is the factor score for the sub-vector u. We consider the following options for w: (1) Admixture (convex combination): w ∈ [0, 1] h and h s=1 w s = 1, which is commonly used in various communities. (2) Sparse Admixture (TKAM): w ∈ [0, 1] h and h s=1 w s = 1 with only k ≪ h non-zero elements allowed. It's worth mentioning that adding the multi-index information to the vector selection mechanism can make the TKAM model structure-aware, potentially yielding additional benefits. One possibility is to make the logits of vector selection conditional on the embeddings of the layer, module, and matrix type, which can be implemented through a hypernetwork [Mahabadi et al., 2021] . However, we leave this for future work.In summary, LoRA provides a local low-rank factorization for each d 1 ×d 2 matrix ∆W independently. In contrast, our VB-LoRA introduces a global low-rank factorization on a b × |{i}| matrix composed of partitioned rank-one vectors, where |{i}| denotes the cardinality of the index set including both physical and virtual indices. As we will see below, this differentiation can better leverage the redundancy in the cumulative gradients, leading to extreme parameter efficiency.Figure 2 overviews our method. The left section demonstrates the high-level idea of VB-LoRA: the vector bank is shared across sub-vectors, modules, and layers. The right section details its architecture. To form each sub-vector, we use a top-k softmax function to select k vectors from the vector bank, which are then pooled into a sub-vector. These sub-vectors are arranged in the desired positions, forming the parameters for LoRA with negligible computational overhead. Algorithm 1 provides the PyTorch-like pseudocode for VB-LoRA, which can be seamlessly integrated into the PyTorch framework.In full fine-tuning, the number of trainable parameters is equal to the model size, i.e., LM d 2 , where L is the number of layers, M is the number of fine-tuned modules, and d is hidden dimension.Algorithm 1 Pseudocode of VB-LoRA in a PyTorch-like style # d: hidden dimension; b: length of sub-vectors; r: rank; h: size of vector bank # k: number of selected vectors used in the top-k admixture module # logits: Each linear layer has two trainable parameters: logits_A and logits_B. # Both parameters have a shape of (d/b)*r*h. # vector_bank: The shared vector bank with a shape of h*b. # x and W: input and the original weight. def get_low_rank_matrix(logits, vector_bank, k): topk_logits, topk_indices = logits.topk(k, dim=-1) topk_weights = torch.softmax(topk_logits, dim=-1) matrix = (topk_weights * vector_bank[topk_indices]).sum(-2) return matrix def VBLoRA_forward(x, vector_bank, logits_A, logits_B, k): r = logits_A.shape[1] A = get_low_rank_matrix(logits_A, vector_bank, k).transpose(0, 1).reshape(r, -1) B = get_low_rank_matrix(logits_B, vector_bank, k).transpose(1, 2).reshape(-1, r) # For memory efficiency, we avoid explictly computing \delta W = B @ A. return x @ W + (x @ B) @ A LoRA reduces this number to 2LM dr, while VeRA further reduces it to LM (d + r). The trainable parameters of LoRA and VeRA are the same as the parameters they need to store.In VB-LoRA, the trainable parameters consist of two parts: the parameters of the vector bank B and the parameters of logits σ. However, at the end of training, the logit parameters can be discarded and only the k selected indices and the top-k admixture weights need to be stored. Therefore, the stored parameters can be represented by a triplet Θ = {B, I, V}, whereB ∈ R h×b is a vector bank containing h vectors of b-dimensional, I ∈ R 2×L×M ×r×(d/b)×k is the top-k indices of the vectors in B for all sub-vectors, andV ∈ R 2×L×M ×r×(d/b)×(k-1)is the top-k admixture weights used to composite the sub-vectors from the bank. It is worth noting that the top-k admixture weights have only k -1 degrees of freedom since they must be summed to 1. Additionally, depending on the size of the vector bank h, the indices I can be efficiently stored as unsigned integers (e.g., uint8 when h ≤ 256), and hence, we count the number of parameters as the float32-equivalent size for a fair comparison. When we use k = 2 and uint8 for indices, the number of stored parameters of VB-LoRA is hb + 3LM r(d/b). Unlike LoRA and VeRA, the number of parameters in VB-LoRA does not increase linearly with the model size (determined by L and d) or the number of fine-tuned modules, i.e., M . While the second term of VB-LoRA's parameters is a linear function of LM d, the coefficient is 3r/b, which is typically very small. For example, in our experiments, the typical values are r = 4 and b = 256, leading to a coefficient of 0.04, whereas the coefficient is 2r for LoRA and 1 for VeRA. Most of the parameters in VB-LoRA reside within the shared vector bank, whose size does not increase linearly with the model size or number of fine-tuned modules.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: We provide the source code in the supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We provide related information in the main paper and appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experiments: 
 In this section, we conduct a comprehensive evaluation of our method through a series of experiments. We begin by comparing VB-LoRA to the state-of-the-art PEFT methods: LoRA, VeRA, and Tied-LoRA on the GLUE benchmark. Next, we extend our analysis to natural language generation tasks using GPT-2, instruction tuning tasks on the Llama2, as well as mathematical reasoning tasks on Mistral and Gemma models. All our experiments were conducted on a server equipped with 8 NVIDIA A100 GPUs. For reproducibility, we provide detailed hyperparameters and specifications of computing resources for each experiment in the appendix. The source code is available at https://github.com/leo-yangli/VB-LoRA.We adopt the General Language Understanding Evaluation (GLUE) benchmark3 [Wang et al., 2018] to assess the performance of VB-LoRA across various natural language understanding tasks, including [Dolan and Brockett, 2005] (paraphrase detection), STS-B [Cer et al., 2017] (semantic textual similarity), QNLI [Rajpurkar et al., 2018] (inference), and RTE (inference).Our experiments are performed with RoBERTa base and RoBERTa large [Liu et al., 2019] . While LoRA and VeRA only finetune the query and value modules, we explore two fine-tuning strategies: query and value only (VB-LoRA qv ), and all linear modules (VB-LoRA all ), including W q , W k , W v , W o , W up , and W down . We create a vector bank of 90 vectors of a length of 256, initialized with a uniform distribution U(-0.02, 0.02). The logits are initialized with a normal distribution N (0, 0.01). The learning rates for the vector bank and logit parameters are set to 0.001 and 0.01, respectively. We set the rank to 4 and k = 2 for all our experiments.Table 1 reveals that VB-LoRA achieves competitive or superior performance compared to VeRA and Tied-LoRA, while being more parameter efficient. For example, when fine-tuning the query and value modules on the RoBERTa large model, our method reduces the stored parameters to less than 40% of those required by VeRA or Tied-LoRA, while outperforming them across all tasks. These results suggest that model performance depends not only on the quantity of trainable parameters but also on how they are composed.Moreover, the results consistently indicate that fine-tuning all modules, beyond just the query and value modules, enhances performance for all the methods. However, LoRA, VeRA and Tied-LoRA requires 2-4 times of the parameters in this case because their parameter counts increase linearly with the number of fine-tuned modules. In contrast, our method uses only 37.5% additional parameters as we maintain the same vector bank size but add additional parameters for indices and top-k weights. Thus, with only 12.8% of the parameters compared to VeRA all (4% compared to LoRA qv ), our method achieves the best average performance.For natural language generation experiments, we fine-tune the GPT-2 Medium and Large models [Radford et al., 2019] Instruction tuning is a process of fine-tuning model with a set of instructions or prompts to enhance its performance on specific instructions [Ouyang et al., 2022] . We first experiment on a general instruction tuning dateset. We use the Cleaned Alpaca Dataset5 , which improves the data quality of the original Alpaca dataset [Taori et al., 2023] . We evaluate the fine-tuned models on the MT-Bench6 [Zheng et al., 2024] , which contains 80 multi-turn questions.Following Kopiczko et al. [2024] , we fine-tune the Llama2 model [Touvron et al., 2023] within the QLoRA [Dettmers et al., 2023] framework7 , which aims to reduce memory usage when finetuning large language models on a single GPU. We utilize the quantization strategy provided by QLoRA, including 4-bit NormalFloat for storage data, BFloat16 for computation parameters, double quantization and paged optimizers to train it on a single GPU. Our fine-tuned models generate responses to these questions, and subsequently, GPT-4 is employed to review and evaluate the generated answers, assigning a quantitative score on a scale of 10. Note that aligning with VeRA, we report the score of the first turn of the conversation. Following Kopiczko et al.[2024], we apply VB-LoRA to all linear layers except the top one. For Llama2 7B, we use a vector bank of 2,048 vectors, each with a length of 256, and the rank is set to 4, resulting in a total of 0.8M stored parameters. For Llama2 13B, we use the same-sized vector bank but increase the rank to 6, leading to 1.1M stored parameters. For all the experiments, we train for one epoch.The results are reported in Table 3 . Notably, we report two sets of LoRA results for each experiment: one from our implementation and the other from Kopiczko et al. [2024] , due to a noticeable discrepancy between the scores. Since we closely follow the experimental settings of Kopiczko et al.[2024], we speculate that the difference is due to changes in the GPT-4 model over time. However, comparing the relative improvements of VeRA and VB-LoRA with their respective implementations of LoRA remains fair. VB-LoRA achieves higher scores than LoRA while using only 0.5% (Llama2 7B) and 0.4% (Llama2 13B) of the stored parameters. While VeRA can reach similar scores with their implementation of LoRA, it requires more than twice of parameters compared to VB-LoRA.To evaluate mathematical reasoning capabilities, we fine-tune the Mistral-7B-v0.1 and Gemma-7B models on the MetaMathQA8 [Yu et al., 2023] dataset and test them on GSM8K9 [Cobbe et al., 2021] and MATH 10 [Hendrycks et al., 2021] datasets. We compare our results with the concurrent work LoRA-XS [Bałazy et al., 2024] , following its experimental configuration. The result is shown in Table 4 . Our method outperforms all baselines on GSM8K, with Mistral-7B utilizing only 0.4% of the parameters compared to LoRA, and Gemma-7B using just 0.3%. Compared with LoRA-XS, our method outperforms on both evaluation datasets while using 70% (Mistral-7B) and 83% (Gemma-7B) of LoRA-XS parameters.We conduct an ablation study to examine the impact of each individual component of VB-LoRA. The experiments are performed on RoBERTa-large, fine-tuning only the query and value modules.Vector Selection Methods Besides the top-k admixture module (abbreviated as Top-k below), there exist several commonly used discrete optimization methods for vector selection, including Noisy Top-k [Shazeer et al., 2016] , Gumbel-Softmax (GS), and Straight-Through Gumbel-Softmax [Jang et al., 2017 , Maddison et al., 2016] . For Top-k and Noisy Top-k, we evaluate the impact of different k to the performances on the CoLA dataset. For GS and Straight-Through GS, we set the temperature τ = 1/3 during training and use Top-1 and Top-2 Softmax for inference. Additionally, we explore "Select All", a special case of Top-k with k equals to the vector bank size h. As shown in Table 5 , Noisy Top-k, GS, and Straight-Through GS significantly underperform Top-k and "Select All". We hypothesize that random noise injected by these methods likely disrupts the parameters of vector bank, leading to instability in the learning process.We further investigate the impact of k to the training dynamics and performance of VB-LoRA. As discussed in Sec. 3.4, the choice of k affects not only the model's performance but also the number of parameters to be stored. Hence, a smaller k is generally preferred for improved parameter efficiency.Table 5 shows that k = 2 yields the best result on CoLA, whereas k = 1 performs significantly worse.To explain this, we delve into the training dynamics of VB-LoRA. As shown in Figure 3 (a), when k = 1, the selected vectors remain largely unchanged during training. In contrast, when k > 1, the model actively explore the vector bank as illustrated in Figure 3 (b) and (c), i.e., different vectors are selected and updated actively during the training process. Additionally, we observed that this vector exploration primarily occurs in the early stages of training, with updates becoming progressively sparser in later stages, as shown in Figure 5 in the appendix. This suggests that the vectors become increasingly specialized for specific sub-vectors as training progresses.Sub-vector Length b VB-LoRA introduces a new virtual dimension that divides the original dimensions of LoRA matrices into sub-vectors of length b. Note that b must be a common factor of all hidden dimensions to ensure compatibility across the entire model. However, the optimal value of b is task-specific and requires tuning as a hyperparameter. Theoretically, with a fixed vector bank budget, a larger b reduces the number of vectors in the vector bank, potentially making each vector less specialized. On the other hand, a smaller b increases the number of trainable parameters and complicates the vector selection process. As shown in Table 6 , a moderate b = 256 yields the best performance on the CoLA task.  
conclusion: 
 This paper introduces a "divide-and-share" paradigm and a differentiable top-k admixture module for extreme parameter-efficient fine-tuning with vector banks. Our proposed VB-LoRA achieves the competitive or higher accuracy while using significantly smaller number of stored parameters compared to the state-of-the-art PEFT methods, including LoRA, VeRA, and Tied-LoRA. In addition, VB-LoRA is model-agnostic and applicable to other PEFT methods [Ding et al., 2023] , including inserted adapters [Karimi Mahabadi et al., 2021] , prompt tuning [Qin et al., 2021] , and BitFit [Ben Zaken et al., 2022] . Although VB-LoRA focuses on reducing the storage and transmission costs for LLM fine-tuning, we believe the proposed scheme can be extended to memory-efficient fine-tuning and parameter-efficient pre-training. We leave these for future exploration.Fine-tuning a pre-trained model requires making design choices about which layers of the model should be frozen or updated. Multitask fine-tuning adds extra complexity about which parameters should be shared or task-specific. Along this line of work, Polytropon [Ponti et al., 2022] jointly learns a small inventory of LoRA adapters and a routing function that selects a variable-sized subset of adapters for few-shot adaptation. Caccia et al. [2023] emphasize the importance of routing granularity and further propose a finer-grained mixing across multiple heads. Following these works, it would be interesting to explore a finer-grained parameter transfer across tasks, heads, types, and layers at the sub-vector level for multitask fine-tuning.Limitations and broader impacts Our experiments are limited to monomodal (text-based), monolingual (English), and LoRA-only settings. Additionally, our exploration of the vector bank is somewhat limited, as we only examine a small range of configurations for bank size and vector length. In terms of broader impacts, VB-LoRA reduces the storage and transmission costs of LLM adapters and demonstrates improved memory-efficiency, making customized LLMs more accessible. We do not foresee any negative societal impact beyond those generally associated with LLMs. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the potential societal impacts in Section 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No data or models are released as part of this work, so safeguards for responsible release are not applicable.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper and provide the name of the license for the datasets we used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
: 
 Parameter-efficient fine-tuning (PEFT) casts a new paradigm that leverages strong prior knowledge built in foundation models and adapts them to a wide range of downstream tasks by updating a small amount of trainable parameters [He et al., 2021] . Compared to prefix/prompt tuning [Li and Liang, 2021, Lester et al., 2021] or in-context learning [Brown et al., 2020] , fine-tuning a large-scale pre-trained model yields better domain specialization dictated by high-quality datasets [Brown et al., 2020 , Liu et al., 2022 , Zhao et al., 2023] . This process can be repeated to suit the needs of ever-changing deployment scenarios and personalizations. However, the sheer volume of parameter space across a multitude of instantiations [Sheng et al., 2023] poses challenges for storage, transmission, and computation, especially for low-resource hardware and consumer-grade networks [Borzunov et al., 2024] .To mitigate these challenges, various PEFT methods have been proposed by adding or adapting a small amount of trainable parameters per task without sacrificing performance [Houlsby et al., 2019, 3 . We propose a sparse top-k module based on the admixture model to learn a global sharing mechanism, making our framework end-to-end differentiable and compression-aware.4. Our method achieves extreme parameter efficiency while maintaining comparable or better empirical performance compared to the state-of-the-art PEFT methods on natural language understanding, natural language generation, instruction tuning, and mathematical reasoning tasks. 
 
 Appendix: The hyperparameters used for the natural language understanding, natural language generation and instruction tuning are provided in Table 7, 8 and 9 . All experiments were conducted on a server equipped with 8 NVIDIA A100 80GB GPUs.The proposed factorization in VB-LoRA is simple to implement in modern deep learning frameworks such as PyTorch, allowing us to fully leverage GPU acceleration. However, the use of subvector decomposition does introduce some computational overhead. This additional overhead is limited to the training phase and does not affect inference, as both LoRA and VB-LoRA merge their parameters back into the original model parameters during this stage.Memory efficiency Despite the training time overhead, the reduced number of trainable parameters in VB-LoRA results in lower memory consumption. During LoRA fine-tuning, the forward pass is z = Ax, H = Bz, without the need to materialize ∆W . This memory-saving technique can be seamlessly incorporated in VB-LoRA and has been implemented in our source code. Table 9 shows that VB-LoRA requires approximately 15%-20% more training time than LoRA, while it consumes less memory than LoRA in both the LLaMA2 7B model and LLaMA2 13B models. For visualization, we conducted experiments on the CoLA dataset using a 24-layer RoBERTa-large model with a vector bank of 30 vectors. We fine-tuned the query and value modules, setting the rank to 2 and the vector length to 1024, resulting in 192 sub-vectors. The assistant also shows respect for the supervisor's time and expresses a willingness to learn from their feedback. The response is relevant, accurate, and detailed enough for the context. However, it lacks a bit of creativity, but given the professional context, this is not a major issue.The assistant's response is highly relevant, accurate, and helpful. It provides a professional and concise email draft that meets the user's requirements. The assistant has included all the necessary elements such as the specific areas where feedback is needed (data analysis, presentation style, and clarity of conclusions), a brief description of the work done, and a polite request for the supervisor's time and feedback. The response is also creative and detailed, providing a clear context for the supervisor. Therefore, I would rate it highly. Hawaii is a paradise on earth, with its beautiful beaches, lush green landscapes, and warm tropical weather. My recent trip to the islands was filled with unforgettable experiences and breathtaking sights. The first stop on my itinerary was the city of Honolulu, the capital of Hawaii. I explored the historic landmarks, stunning beaches, and world-renowned cuisine. The Pearl Harbor Memorial was a must-visit, as it commemorates the tragic attack that occurred on December 7, 1941. The USS Arizona Memorial is located above the sunken battleship and is a solemn reminder of the sacrifices made during the war.From Honolulu, I ventured to the island of Oahu, where I spent a few days relaxing on the beautiful Waikiki Beach. The beach is lined with palm trees and crystal-clear waters, and is the perfect spot for swimming, sunbathing, and snorkeling.After exploring the island, I ventured to Maui, known as the "Valley Isle." Maui is home to some of the most beautiful beaches in Hawaii, including the world-famous Ka'anapali Beach. I spent the day soaking up the sun and enjoying the warm, turquoise waters.In conclusion, my trip to Hawaii was filled with unforgettable experiences and breathtaking sights. The islands are a paradise on earth, with something for everyone. Whether you're looking for history, culture, or natural beauty, Hawaii has it all. I can't wait to return and explore more of these beautiful islands.Dear reader, I recently had the opportunity to travel to the beautiful island of Hawaii, and I am thrilled to share my experience with you. This trip was full of cultural immersion, adventure, and breathtaking scenery. Here is a glimpse of my journey: On the first day, I landed in Honolulu, the capital city of Hawaii. I was immediately struck by the lush greenery and tropical weather. I spent the day exploring the city, visiting famous landmarks such as Pearl Harbor and the USS Arizona Memorial. The USS Arizona Memorial is a somber yet powerful tribute to the brave soldiers who lost their lives in the attack on Pearl Harbor. I spent the next few days relaxing on the beautiful beaches of Maui. I went snorkeling in the crystal-clear waters, and saw tropical fish, turtles, and even a few sharks. The snorkeling was truly an unforgettable experience.One of the highlights of my trip was a visit to the Polynesian Cultural Center. This is an interactive museum that showcases the culture and traditions of Polynesian islands. I got to try traditional dances, make fire, and even learn how to play the ukulele.In conclusion, my trip to Hawaii was truly unforgettable. I experienced cultural immersion, adventure, and breathtaking scenery. If you're looking for a trip of a lifetime, I highly recommend visiting this beautiful island paradise.The assistant's response is a well-written and engaging travel blog post about a recent trip to Hawaii. It provides a detailed itinerary of the trip, highlighting the cultural experiences and must-see attractions in each location. The assistant also does a good job of describing the natural beauty and unique features of each island, which adds depth and interest to the post. The assistant's response is relevant, accurate, and creative, making it a high-quality response to the user's request. However, it could have included more about the cultural experiences, such as local food, music, or traditions.The assistant's response is highly detailed, engaging, and relevant to the user's request. It provides a comprehensive overview of a trip to Hawaii, highlighting cultural experiences and must-see attractions. The assistant's use of descriptive language helps to paint a vivid picture of the experiences, making the blog post more engaging for readers. The assistant also provides a personal touch by sharing their own experiences and impressions, which adds depth to the response. The assistant's response is accurate, as it mentions real places and experiences in Hawaii. Overall, the assistant's response is highly creative and provides a high level of detail, making it an excellent travel blog post. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://gluebenchmark.com/Licensed under CC BY-SA 4.0. URL: https://github.com/tuetschek/e2e-datasetThe original and cleaned Alpaca datasets are licensed under CC BY-NC 4.0. URLs: https://huggingface.co/datasets/tatsu-lab/alpaca, https://huggingface.co/datasets/ yahma/alpaca-cleanedLicensed under CC BY 4.0. URL: https://huggingface.co/datasets/lmsys/mt_bench_human_ judgmentshttps://github.com/artidoro/qloraLicensed under MIT. URL: https://huggingface.co/datasets/meta-math/MetaMathQALicensed under MIT. URL: https://huggingface.co/datasets/openai/gsm8kLicensed under MIT. URL: https://github.com/hendrycks/math/ 


Decision: Reviewers consistently praise VB-LoRA's ability to achieve comparable or even better performance than strong baselines like LoRA and VeRA while using significantly fewer trainable parameters (often less than 0.1M). This is a major contribution in the growing field of PEFT. The paper provides a reasonably comprehensive evaluation across different tasks (NLU, NLG, Instruction Tuning) and model families (RoBERTa, GPT-2, Llama2). Furhter, the the core concept of a shared vector bank and composing adapter parameters from it is intuitive and easy to implement. The most critical reviewer, eNJC raised concerns about the scalability of VB-LoRA, particularly the potential for load balancing issues similar to MoE. The authors clarified the differences in the routing mechanism between VB-LoRA and MoE, arguing that VB-LoRA is less prone to load balancing problems. They also provided new experimental results showcasing a more balanced distribution of vector usage in a larger model (Gemma-7B).

I recommend accepting this paper. As the authors agreed to do, please expand the limitations section based on reviewer feedback. This should include acknowledging the limitations in the explored space of options (vector selection, VB size, etc.) and the focus on monolingual, mono-modal, and LoRA-only contexts.

Reviews: review: paper_summary: paper_summary: The authors present a modified version of LoRA called VB-LoRA which is a highly parameter efficient fine-tuning method. It uses a vector bank to represent the model parameters as a composition of vectors. This vector bank is then used to select top-k vectors using the top-k softmax function which are thereby pooled and arranged to form the A and B low-rank matrices of LoRA. The top-k admixture module function selected is a differentiable function so the whole model is trained end-to-end making it more efficient for a particular task. Ultimately the authors show that such as divide-and-share approach results in an orders of magnitude more parameter efficient technique.
strength_weakness: strength_weakness: 1. The parameter count of VB-LoRA is defined by: hb + 1.5LMr(d/b). The second term of this is still linearly dependent on the model dimensions (L and M). In the paper's experiments r<<b in which case the second is quite small and doesn't impact the growth in parameter by much. But there exist several adaptation tasks in which much larger ranks are needed. For example this paper (https://ieeexplore.ieee.org/abstract/document/10445894) shows that for domain adaptation large ranks (eg. 64) are much better than smaller ranks. In some of my other works for domain and language adaptation I have needed to use even larger ranks (200+) which are very much comparable to the value of b. In such cases, VB-LoRA's parameter requirement would be very much comparable to other methods.

2. Some parts of the paper are not entirely clear. For eg. in lines 166-169, the paper mentions that decomposed cumulative gradient parameters are more sensitive than the original model parameters during the training process. An intuition to why this is happening would be helpful.

3. In line 153, the paper mentions that by keeping k << h makes the learnt vectors highly specialized. However figure 3 shows that even with k's value as small as 2 and 3, the model updates almost all the vectors during the training process. Inherently this means that almost all the vectors have some activation and they are not very specialized.

4. Not a lot of details are provided for the virtual dimension 'b' and the results of table 5 are not explained.

5. nit: line 167: 'parameters'
questions: questions: 1. Can you provide an intuition to your finding why the cumulative gradient parameters are more sensitive than than the original model parameters and how this affects adaptation?
2. Have you experimented with larger ranks and how do they look in terms of performance and parameter count?
3. Have you tried this technique on other tasks than what's listed in the paper? Also have you tried it on other architectures?
limitations: limitations: 1. In my view the authors have correctly mentioned that this paper has no larger societal impact beyond what LLMs may have.
2. Table-3 shows that the performance of LoRA may have improved with the changes in the GPT-4 model over time. But for the other tables the authors have used the old results for several competing models.
3. I agree with the rest of the answers to the NeurIPS paper checklist that the authors have provided.
paper_summary: paper_summary: This paper explores parameter-efficient fine-tuning (PEFT) methods in the context of further reducing the number of fine-tunable parameters, even to the extreme. The main idea is to reduce the number of parameter within LoRA modules as much as possible while maintaining or even improving fine-tuning performance. This line of work is a sub-area of PEFT research, where similar ideas have been explored in the past (e.g., the work of Karimi Mahabadi et al. on Compacter architectures, or a very recent work concurrent to this one: LoRA-XS). 

To this end, the authors propose VB-LoRA which aims to use globally shared vector banks comprising 'constituent' sub-vectors that can be recombined into local 'delta' parameters (of the \Delta W matrix) that modify/adapt the weights of the original model. The VB-LoRA is then evaluated on GLUE with RoBERTa (for NLU), E2E with GPT-2 (for generation), and Llama 2 on MT-Bench (for instruction tuning) to show its benefits in terms of paramater efficiency and performance.
strength_weakness: strength_weakness: - Evaluation is not comprehensive: 
1) When it comes to instruction-tuned models, the main findings are based on a single model from a single family (Llama 2), fine-tuned on a single instruction tuning dataset and evaluated on a single evaluation dataset. This is simply enough to move from anecdotal to more generalisable findings, and experiments with (i) additional models, (ii) more instruction tuning datasets (e.g., Flan-v2, Tulu v2, there are more), and (iii) additional evaluation sets (e.g., MMLU, GSM, BBH, HumanEval) are required.
2) The same holds for NLU and NLG experiments. NLU is evaluated only on GLUE (where it's widely established that performance on GLUE is already saturated and considered 'superhuman'). For NLG, only experiments with a single model on a single benchmark are run. As I mentioned above, the experiments are broad, but stay quite shallow, and this has to change in the revised version.

- The paper is very quantitatively driven, but it doesn't delve into providing clarifications and explanations on why certain quantitative phenomena/findings are observed:
1) What is the intuition behind VB-LoRA outperforming LoRA? How can we explain this? What makes VB-LoRA even more suitable performance-wise? Isn't it counter-intuitive that an extremely efficient model can outperform here? Are the tasks then too simple (with very low intrinsic dimensionality) as is the case with some of the standard GLUE tasks?
2) How can one fine-tune the optimal VB size (b) and dimensionality of subvectors within the VB (Table 5) - is this task specific or model specific? What would happen if one increases parameter budget a bit - can we expect even better performance or not? In what cases?
3) Are there any sub-vectors that get selected more often than some others? Why?

Overall, while I'm quite happy regarding technical aspects of the work, the paper can be much improved in terms of evaluation/experiments and discussion of the key findings (some error analysis would be useful as well).
questions: questions: - Lines 180-182. "It’s worth mentioning that adding the multi-index information to the vector selection mechanism can make the TKAM model structure-aware, potentially yielding additional benefits." This is unclear to me and warrants further clarification/discussion. Could you write a short paragraph on how the multi-index information would be added and what type of structure would then TKAM learn (become aware of)?

- Additional discussion on the similarities and differences between TKAM and standard (sparse and soft) MoE-s is required. Are there any MoE-style paradigms (which incur a higher parameter cost) that could have been used instead of TKAM? In general, the work doesn't really examine the choice of TKAM as a go-to routing method for the selection of sub-vectors.

- Can the authors elaborate more on the similarities and differences between their work and the Compacter work (beyond the fact that Compacter focuses on bottleneck adapters while VB-LoRA focuses on LoRA)? Conceptually, the papers operate in the same space with similar ideas.
limitations: limitations: There are limitations to this work that should be further elaborated (see also under 'Weaknesses'). The paper didn't explore the whole space of options when it comes to vector selection, VB size, VB content interpretation, and task dependency of findings, among other things. The work also stays within the realm of monolingual (i.e., English-only), mono-modal (i.e., text-only) and mono-PEFT (i.e., LoRA-only) contexts, and this should also be adequately signalled in the Limitations section.
paper_summary: paper_summary: The authors propose an extremely parameter-efficient methods, VB-LoRA, for finetuning an LLM. Specifically, VB-LoRA has a shared vector bank (similar to a codebook), the adapter parameters (A and B) of the linear layers in an LLM are constructed from this bank by selecting the most effective bank vectors. Due to the shared mechanism, the number of trainable parameters are very little, much fewer than the strong baselines, like LoRA, VeRA and Tied-LoRA.

Tha authors also validate VB-LoRA on three benchmarks (GLUE, E2E benchmark and MT-Bench) on 6 LLMs from three model families (RoBERTa, GPT-2 and Llama2). The results show that VB-LoRA's results are better or comparable to above-mentioned baselines while requiring much fewer number of trainable parameters.
strength_weakness: strength_weakness: 1. The paper is not easy to follow, especially Section 3 for the proposed method. VB-LoRA is basically an algorithm inspired by Mixture-of Expert or codebook, and is used for PEFT here. However, the authors seem to intend to make it complexer by relating VB-LoRA to other theoretical works (L137-143, L157-162), which would be better by putting in the related works. Such writing style easily distracts the reader's attention. 

2. Some experimental setting is weird, not in line with previous work.
- In Table 1, two high-resource tasks of GLUE, i.e. MNLI and QQP, are deleted. The authors claim that the experiments are conducted on  a server equipped with 8 A100 GPUs (L215), which means the computation resource should not be a problem. Why are these two tasks are discarded?
- All ablation studies are conducted on a very small task, CoLA, whose results have large variance. Normally, 24GB GPU memory and less than 1 hour are enough for finetuning RoBERTa on CoLA. Due to the small number of samples in CoLA, its results have relatively larger variance. Since the authors have much more resource, why choosing this task for ablation study?

3. VB-LoRA has a potential drawback that is inherent in its design, i.e. unscalibility. One big problem for MoE is that not all expert embeddings can be well trained. We normally include a balance loss and use a small number of experts to avoid this problem. For VB-LoRA, could you offer this ablation study? I.e. increasing the number and length of the bank vectors.

4. VB-LoRA doesn't consistently outperform baselines, or the gap is very narrow. I would be interested in more benckmarks on larger LLMs, like commonsen/arithmetic reseasoning tasks.
questions: questions: 1. In algorithm 1, if you implement delta_W in this way, it will comsume more GPU memory during finetuning than LoRA. For implementing LoRA, the forward pass is z = Ax, H = Bz. It means we don't calculate delta_W explicitely. If you calculate delta_W, it will be stored in the memory for gradient computation, which enlarges the memoy for activation. 

2. Check other questions in Weakness.
limitations: limitations: The limitation discussion is not thorough.
paper_summary: paper_summary: This paper proposes an approach to decrease the number of parameters in LoRA by introducing a collection (aka "vector bank") of sub-vectors and composing Lora matrices across modules/layers using this collection. Each sub-vector within a given LoRA module/layer is then formed as a linear interpolation of sub-vectors from the vector bank, where the coefficients are obtained by selecting top-k "basis vectors" and then applying a softmax. When applied to Roberta-large, the approach yields improvements over LoRA on the GLUE benchmark with only a tenth of the free parameters. When applied GPT-2 large , it obtains similar or slightly better performance than LoRA with around a sixth of the parameters, on the E2E task. The paper reports ablation studies showing the effect of the vector selection strategy (i.e. alternatives to top-k) and the sub-vector length.
strength_weakness: strength_weakness: * The paper does not discuss if the proposed factorization approach (i.e. using subvectors) can be performed efficiently on accelerators such as GPUs. Also it would be good to know what is the overhead resulting from this subvector decomposition.
* It would be useful to report the performance of an alternative PEFT scheme such as prefix tuning (or prefix tuning) or adapters which have a similar number of parameters as the proposed approach.
questions: questions: * It would be useful to discuss this paper in related work: https://openreview.net/pdf?id=qKQu1ZcJjD : The idea is to initialize matrices in an LLM using pre-trained submatrix components, which bears resemblance to the subvector decomposition.

Typos:
L101:
* parameters -> parameter
L192
* numebr -> number
L314
* frozen -> frozen
limitations: limitations: Yes

Comments: title: The response clarifies some of my concerns
comment: Thank you for the new results. I'm willing to raise my score to 5, because some of my following concerns are not fully addressed:

1. **CoLA for ablation study**. I really suggest to use a more recent dataset (like math related or commonsense related datasets) and recent LLMs (Llama2-7B) to reproduce the ablation study. CoLA only contains 8.5K samples with short sentences. As your new results shown above, the variance is about 1 (much larger than the improvement), which makes the observation less convincing. 

2. **Scalability**: As your new results shown above, VB-LoRA's scalability is not good. When increasing the bank size or the vector length, the performance doesn't really change. This may hint that VB-LoRA is not easy to be applied to knowledge-intensive tasks. Please include a related discussion in the limitation section, which is very important for your followers.
Thanks for the follow-up question. We did the ablation study as you suggested, i.e., increasing the number and length of the bank vectors, to demonstrate the scalability. The table below shows the results on COLA (median over 5 runs, with standard deviations in parentheses).
 
As we can see, as the bank size (90->8192) and vector length (256->1024) increase, the performance of VB-LoRA is relatively stable. Once the number of trainable parameters is very large, the performance degrades a little bit, showing the sign of overfitting. This result further illustrates the scalability of our method.
 
We will include this table in the revised paper.

| Bank size | Vector length | # Params (M) | COLA  |
|-----------|---------------|--------------|-------|
| 90        | 256           | 0.03         | 69.3 (1.5) |
| 1024      | 256           | 0.28         | 69.2 (1.2) |
| 8192      | 256           | 2.12         | 69.4 (1.3) |
| 8192      | 1024          | 8.39         | 68.8 (0.7) |
Thank you for the clarification and etra eperiments. I still have one more question for W3:

Why don't you show the scalability results here? In stead, you only simply mentioned "we have conducted experiments with a range of model sizes, from Robota 125M to Llama 13B, and varied the number and length of vector banks from 30 to 2048 vectors and 128 to 1024 dimensions, respectively. Our method demonstrated consistent performance across these variations."
I would like to thank the authors for the provided response and the additional results, and I'm happy to increase my score in light of the provided new evidence.
Thanks to the authors for sharing the newer results and adding clarifications.