Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates

Venue: ICML_cc_2024_Workshop_WANT
Published: 2024-06-02 22:13:45

Abstract: It is a common practice in natural language processing to pre-train a single model on a general domain and then fine-tune it for downstream tasks. However, when it comes to Large Language Models, fine-tuning the entire model can be computationally expensive, resulting in very intensive energy consumption. As a result, several Parameter Efficient Fine-Tuning (PEFT) approaches were recently proposed. One of the most popular approaches is low-rank adaptation (LoRA), where the key insight is decomposing the update weights of the pre-trained model into two low-rank matrices. However, the proposed approaches either use the same rank value across all different weight matrices, which has been shown to be a sub-optimal choice, or do not use any quantization technique, one of the most important factors when it comes to a model's energy consumption. In this work, we propose Bayesian-LoRA which approaches low-rank adaptation and quantization from a Bayesian perspective by employing a prior distribution on both quantization levels and rank values. As a result, B-LoRA is able to fine-tune a pre-trained model on a specific downstream task, finding the optimal rank values and quantization levels for every low-rank matrix. We validate the proposed model by fine-tuning a pre-trained DeBERTaV3 on the GLUE benchmark. Moreover, we compare it to relevant baselines and present both qualitative and quantitative results, showing how the proposed approach is able to learn optimal-rank quantized matrices. B-LoRA performs on par with or better than the baselines while reducing the total number of bit operations by roughly 70\% compared to the baseline methods.

Main paper: 
 introduction: 
 Pre-trained language models (PLMs) have become the de-facto models in various natural language processing tasks (Devlin et al., 2019; Liu et al., 2019; He et al., 2021b; Radford et al., 2019; Brown et al., 2020b) . Although full fine-tuning (FT) has been the most common way to adapt pre-trained models to downstream tasks Qiu et al. (2020) ; Raffel et al. (2020) , with the rise of large pre-trained models full FT is becoming unfeasible. For instance, while BERT (Devlin et al., 2019) consists of up to 300 M parameters, GPT-3 (Brown et al., 2020b) has up to 175 B parameters, making full FT computationally and energy demanding. To address this issue, existing works (Hu et al., 2022; Dettmers et al., 2023) focus on reducing the fine-tuning parameters while maintaining or even improving the downstream performance of PLMs. One approach is to mitigate such a problem by adapting only some parameters or learning external modules for new tasks, while keeping the base model frozen and shared across tasks. As a result, only a small number of task-specific parameters need to be stored and loaded, greatly boosting the operational efficiency when deployed. For example, Adapter Tuning approaches (Houlsby et al., 2019; Rebuffi et al., 2017; Pfeiffer et al., 2020; He et al., 2022) employ small neural modules called adapters within the layersFrozen Pretrained Weights W ∈ ℝ d×d A ∈ ℝ r×d B ∈ ℝ d×r E ∈ ℝ r E ∈ ℝ re 1 e 2 e 3 e 4 e 5 e 6 e 7 e 8 g 1 g 2 g 3 g 4 g 5 g 6 g 7 g 8 As mentioned in Sec. 1, every weight W can be decomposed as W = W 0 + BEA. Since E is a diagonal matrix, we represent it as a vector of size r that acts on matrix A with pointwise multiplication. (Right) Rank Adaptation and Quantization techniques are visually represented, following equation 13 for Rank Adaption and equations 7 and 8 for Quantization, respectively. Visual Representation of quantization technique is taken from (Van Baalen et al., 2020) .of the pre-trained model. Prefix tuning (Li & Liang, 2021) and Prompt tuning (Lester et al., 2021) attach additional trainable prefix tokens to the input or hidden layers of the base model. These methods have been shown to achieve comparable performance to full fine-tuning, while only updating less than 1% of the original model parameters, significantly releasing the memory consumption.Another line of research proposes to model the incremental update of the pre-trained weights in a parameterefficient way, without modifying the model architecture (Zaken et al., 2021; Guo et al., 2020; Hu et al., 2022; Zhang et al., 2023; Valipour et al., 2022) . Among this family of methods, the most widely used is LoRA (Hu et al., 2022) , which parameterizes weight updates ∆ as a low-rank matrix by the product of two much smaller matrices:EQUATIONwhere W 0 , ∆ ∈ R d×d , A ∈ R r×d and B ∈ R d×r with r ≪ d. During fine-tuning, only A and B are updated. The rank r is chosen to be much smaller than the dimension of W (e.g., r = 8 when d = 1024).With less than 0.5% additional trainable parameters, training overhead can be reduced up to 70%, achieving comparable or even better performance than full fine-tuning (Hu et al., 2022) . However, LoRA still has limitations since searching the optimal rank value requires re-running the entire fine-tuning for each new value (Valipour et al., 2022) and it sets the same rank r of each incremental matrix ∆ across different LoRA blocks (Zhang et al., 2023) . The latter, as pointed out by Zhang et al. (2023) , does not take into account that the impact of the weight matrices on downstream performances varies significantly across modules and layers when fine-tuning pre-trained models.While PEFT approaches are proved to be very successful in reducing the number of parameters needed for specific downstream tasks, the LoRA-based approaches, proposed in the literature, either use the same rank value across all different weight matrices or do not use any quantization technique. However, to reduce the computational cost of neural network inference and the related energy consumption, quantization and compression techniques are often applied before deploying a model in real life (Van Baalen et al., 2020; Xu et al., 2024) . Indeed, the former reduces the bit width of weight and activation tensors by quantizing floating-point values onto a regular grid, allowing the use of cheap integer arithmetic, while the latter Under review as a conference paper at ICLR 2025 aims to reduce the total number of multiply-accumulate (MAC) operations required (Kuzmin et al., 2019; Krishnamoorthi, 2018) .Recently, Van Baalen et al. (2020) proposed the BayesianBits approach, which introduces a novel and hardware-friendly decomposition of the quantization operation and allows for adaptable and optimal quantization levels, resulting in optimal quantization levels and, therefore, lower model energy consumption.Inspired by BayesianBits (Van Baalen et al., 2020) , we propose Bayesian-LoRA (B-LoRA)1 which approaches LoRA matrix decomposition and quantization from a Bayesian perspective. Indeed, by positioning a prior distribution on both quantization levels and rank values of the low-rank matrices weights, the optimal rank values and quantization levels for each individual LoRA block are learned. We validate the proposed approach, using the GLUE (Wang et al., 2019) benchmark, and compare it with state-of-the-art baselines, such as LoRA (Hu et al., 2022) , DyLoRA (Valipour et al., 2022) , and AdaLoRA (Zhang et al., 2023) . Moreover, we perform a qualitative analysis of quantization levels and rank values across the fine-tuned quantized LoRA blocks, which shows how B-LoRA is able to reduce the total amount of bit operations of roughly 70%, while performing on par or better than the related SOTA baselines. 
transformer-based language model: 
 Pre-trained language models have gained significant attention in the field of natural language processing (NLP), due to their impressive capabilities in language generation, in-context learning, world knowledge, and reasoning.The GPT family, including GPT-3 (Brown et al., 2020a) , ChatGPT (OpenAI, 2022) , GPT-4 (OpenAI, 2023) , and InstructGPT (Ouyang et al., 2022) are some of the representative works on autoregressive LLMs. A second family of language models are bi-directional models, like DeBERTa (He et al., 2021b) , DeBERTa-v3 (He et al., 2021a) , RoBERTa (Liu et al., 2019) , T5 (Raffel et al., 2020) . It is a common practice to train transformer models on Language Modelling or Masked Language Modelling task in an unsupervised manner, which does not require annotated data, and adapt it for multiple downstream applications. Such adaptation can be done via fine-tuning, which updates all parameters of a model (Hu et al., 2022) . Since transformer models often have billions of parameters, computing gradient updates for the entire model can be infeasible without appropriate hardware. This computational challenge motivated research into parameter-efficient fine-tuning techniques, aiming to reduce hardware requirements while maintaining model performance (Hu et al., 2022; Zaken et al., 2021) .Low-Rank Adaptation. LoRA (Hu et al., 2022 ) is an efficient fine-tuning method that updates only a small subset of model weights. It approximates weight changes using low-rank matrix decomposition, significantly reducing the number of trainable parameters for downstream tasks. This results in the following forward pass:W x = W 0 x + ∆x = W 0 x + BAx (2)where W 0 , ∆ ∈ R d×d , A ∈ R r×d and B ∈ R d×r with r ≪ d. DyLoRA, which aims to optimize matrices for as many ranks as possible, AdaLoRA (Zhang et al., 2023) searches for optimal rank values. Given parameter budget, it is allocated among weights according to their importance score. Authors reparameterize LoRA modules using SVD decomposition and during training diagonal values can be truncated. Recently, it was proven that a nearly linear time approximation exists for LoRA (Hu et al., 2024) .Quantization of LLMs. Quantization is a compression technique that reduces the bit width of the parameters and/or activations of LLMs to improve their efficiency and scalability (Xiao et al., 2023; Dettmers et al., 2022; 2023) . Existing methods mostly focused on preserving or restoring the accuracy of quantized LLMs during the inference stage (Zhu et al., 2023) , where the key is to reduce the memory footprint and computational costs without re-training the LLMs. In the context of low-rank adaptation, QLoRA (Dettmers et al., 2023) uses a novel high-precision technique to quantize a pre-trained model to 4-bit, and adds a small set of learnable low-rank Adapter weights that are tuned by backpropagating gradients through the quantized weights. Moreover, QA-LoRA (Xu et al., 2024) quantizes the weights of the pre-trained language model during fine-tuning to reduce time and memory usage. However, both QLoRA and QA-LoRA use vanilla LoRA blocks, inheriting their limitations related to rank values. In this work, we jointly optimize quantization levels and rank values to reduce the complexity of the model, while fine-tuning LoRA blocks to achieve better downstream performances. 
method: 
 Our method searches for optimal precision and rank allocation in transformer models. In this section, we discuss these components separately.Following BayesianBits (Van Baalen et al., 2020) , for a given weight x with values in the range [α, β] we apply uniform quantization with different bitwidth b n = n, n ∈ N , where N = {2, 4, 8, 16, 32}. For bitwidth b n , quantized weights are computed as:EQUATION)where s is the step size of the quantized value and ⌊•⌉ represents the round-to-nearest-integer function.Van Baalen et al. ( 2020) derive an expression for a residual error between consecutive quantization levels, using bitwidth b n and b n+1 = 2 * b n :ϵ bn+1 = s bn+1 x -x bn s bn+1 , s bn+1 = s bn 2 b + 1 (4)Given this expression, weight x can be reconstructed from its quantized version by adding error terms:EQUATIONTo make weight precision controllable, gating variables z i , i ∈ {4, 8, 16, 32} are introduced:EQUATIONReinterpreting the model from a Bayesian perspective, we can introduce a prior distribution on gates z i . The prior can be described with the following equations:EQUATIONthat represent consecutive active gates, andEQUATIONwhich are used for inactive gates. Notably, using this notation, whenever gate n is inactive, all the consecutive ones will be inactive as well. Then, we can define the posterior distribution of gates q ϕ as:q ϕ (z m |z n = 1) = Bern(σ(ϕ m )) q ϕ (z m |z n = 0) = Bern(0) (9)where ϕ i are used to parameterize the defined Bernoulli distributions and σ(•) is a sigmoid function.Algorithm 1 B-LoRA block. Individual quantizer module parameters ϕ are not indicated for the sake of clarity.Require: Input x, rank r, pre-trained matrix W ∈ R d1×d2 , LoRA matrices A ∈ R r×d2 and B ∈ R d1×r , vector with diagonal entries E ∈ R r , rank distribution parameters ξ 2 . . . ξ r , quantiz- ers Q w , Q a , Q e , Q b , used for weight matrices, and Q A , Q E , Q out , used for output variables.1: W , Ā, Ē, B = Q w (W ), Q a (A), Q e (E), Q b (B) # compute rank gates 2: g 1 = 1, g 2 = σ(ξ 2 ) , g i = i j=1 σ(ξ j ) # apply gates on diagonal entries 3: Ēi = Ēi * g i # compute output 4: return Q out ( W x + B • Q E ( Ē • Q A ( Āx)))Algorithm 2 Quantizer Module (Q); Hyperparameters ζ 1 , ζ 2 and t are fixed and defined in Appendix C Require:Input x; Quantizer parameters ϕ 1: clip(x, min = α, max = β) 2: s 2 ← β-α 2 2 -1 , x 2 ← s 2 ⌊ x s2 ⌉ 3: x q ← x 2 4: 5: for b in {4, 8, 16, 32} do 6: if training then 7: u ∼ U [0, 1], g ← log u 1-u , s ← σ((g + ϕ)/b) 8: z b ← min(1, max(0, s(ζ 1 -ζ 2 ) + ζ 2 )) 9: else 10: z b ← I σ β log -ζ2 ζ1 -ϕ < t 11: end if 12: s b ← s b/2 2 b/2 +1 13: ϵ b ← s b x-(x2+ j<b ϵj ) s b14:x q ← x q + z b j<b z j ϵ b 15: end for 16: return x q Van Baalen et al. ( 2020) provide results for convolutional models like LeNet (Simonyan & Zisserman, 2014) and VGG (Lecun et al., 1998) . In our work, we apply learnable quantization to transformers. We limit our experiments by applying the method discussed above only to attention modules.Consider an attention module, parameterized by matrices W k , W q , W v corresponding to keys, queries, and values, respectively. Following Van Baalen et al. (2020) , we apply the learnable quantization approach to both weights and variables defined within the attention module. During fine-tuning, we define W k , W q , W v as LoRA blocks and optimize quantization levels of each weight and variable within the attention module. Specifically, we use a different quantizer for every matrix of each LoRA block W 0 , A, B, and the related output variables.In this section, we formalize the LoRA parametrization as in Zhang et al. (2023) and apply the gating mechanism defined in equation 6 to optimize the rank value of each LoRA block. We follow Zhang et al. (2023) and extend LoRA parameterization to have an SVD structure. As a result, LoRA blocks are modified to include the diagonal matrix E. Following Zhang et al. (2023) , we store diagonal entries in a vector, therefore E ∈ R r . Hence, the forward pass in equation 2 can be expressed as:W x = W 0 x + BEAx (10)Under review as a conference paper at ICLR 2025In order to control and optimize rank values during training, the entries of the vector E are multiplied by gating variables as follows:EQUATIONAs for z i priors defined in equations 7 and 8, we define the g i priors as follows:EQUATIONwhere p(g 1 ) is always 1 because all LoRA matrices should have at least rank 1. Such parametrization ensures that every diagonal entry e j is inactive if e i , j > i is not active. Consistently to equation 9, we can model the posterior distribution of gates r ξ as:EQUATIONThe pseudocode for our method is provided in Algorithm 1. An algorithm for a forward pass of weight and activation quantizers can be found in Algorithm 2.As LoRA (Hu et al., 2022) , our proposed approach is agnostic to any training objective. Consistently to prior works (Hu et al., 2022; Valipour et al., 2022; Zhang et al., 2023) , we focus on language modeling as our motivating use case.Suppose we are given a pre-trained autoregressive language model P Φ (y|x) parametrized by Φ. Consider adapting this pre-trained model to a given downstream task, represented by a training dataset of contexttarget pairs: Z = {(x i , y i )} i=1,..,N , where both x i and y i are sequences of tokens.Following Hu et al. ( 2022), we can define the LoRA objective function as:EQUATIONwhere Φ 0 represents the initial set of parameters of the pre-trained model and ∆Φ(Θ) represents the set of LoRA parameters that are optimized during the fine-tuning.In order to optimize the proposed B-LoRA blocks, we follow the optimization scheme defined by Van Baalen et al. (2020) . Since the gating variables are sampled from Bernoulli distributions, we use an approximation of the KL divergence term, which results in the following objective:EQUATION)where B is a set of available bitwidth, k denotes the index of the quantizer, λ q and λ r are hyperparameters that weight quantization and rank adaptation regularizers, respectively. In all our experiments, we set λ r = λ q = 1. We follow Van Baalen et al. ( 2020) and employ straight-through estimator (STE) (Bengio et al., 2013) for rounding operation, performing rounding in the forward pass, while using identity in the backward pass. 
experiments: 
 Under review as a conference paper at ICLR 2025 Here, the parameter r in LoRA and the parameter b in AdaLoRA correspond to the rank value and the parameter budget, respectively. We evaluate B-LoRA on two configuration: using quantization + rank adaptation (q + ra) and using quantization only (q). The best results for each data set are shown in bold, while second best ones are underlined. # of parameters refers to the number of trainable parameters of encoder (excluding classification head). 1 1 1 1 1 1 1 8 8 7 8 8  1 1 1 1 1 1 1 8 8 8 8 8  1 6 7 7 7 8 8 7 7 7 8 8 1 2 3 4 5 6 7 8 9 10 11 12 Layers K Q V QQP 1 1 1 1 1 6 6 1 1 7 6 8 1 1 1 1 5 1 1 1 7 8 6 6 1 1 1 7 8 7 8 8 8 8 8 8 In this section, we design empirical experiments to understand the performance of B-LoRA and its potential limitations by exploring the following questions: (1) How does optimizing quantization levels and rank values affect the downstream usefulness of LoRA-based fine-tuning approaches? (2) Can we observe consistent patterns of quantization levels and rank values across different tasks? (3) How many bit operations (BOPs) can we save by using adaptive quantization levels and rank values? 4.1 EXPERIMENTAL SETUP Following AdaLoRA (Zhang et al., 2023) , B-LoRA is implemented for finetuning DeBERTaV3-base (He et al., 2020) on natural language understanding using the GLUE benchmark (Wang et al., 2018) . We set the number of training epochs and scaling parameter alpha (Hu et al., 2022) according to AdaLoRA. However, while AdaLoRA uses specific hyperparameters for each different GLUE dataset, we use the same set for the whole benchmark, showing the robustness of the proposed method. In contrast to AdaLoRA, our method is applied to W k , W q and W v while W o , W f1 and W f2 are kept frozen. More details on hyperparameters are stated in Appendix C. The only layers that are fine-tuned with W q , W k , W v are two linear layers in the task-specific head. We provide results for the full method B-LoRA(q + ra) and an ablation of it that uses only adaptive quantization B-LoRA(q). We can compute the number of training parameters for the proposed approach as follows:K Q V RTE 1 1 1 1 1 1 2 1 4 7 7 8 1 1 1 1 1 1 1 1 8 8 8 8 1 3 1 1 6 7 8 8 8 8 8 8 K Q V MRPC 1K Q V SST-2 1 1 1 1 1 1 1 1 1 8 8 7 1 1 1 1 1 1K Q V MNLI 1 1 1 1 1 1 1 1 3 7 8 3 1 1 1 1 1 5 1 4 7 7 8 8 1 4 1 4 8 6 8 6 7 8 8 7 K Q V QNLI#params = 6 × r × l × d (16)where l represents the base model layers and d the hidden model's sizes, respectively. The number of parameters in the classification head is not included in the parameter count, since it is fixed for all methods.A full description of B-LoRA and related baselines number of parameters computation can be found in Under review as a conference paper at ICLR 2025 Appendix E. B-LoRA is implemented using PyTorch (Paszke et al., 2019) , publicly available HuggingFace Transformers weights (Wolf et al., 2019) , BayesianBits2 and AdaLoRA3 repositories.To evaluate B-LoRA's performance against QLoRA (Dettmers et al., 2023) , we fine-tuned Phi-2 (Hughes) and Qwen2 (Yang et al., 2024 ) models using both methods and assessed them on the MMLU benchmark. MMLU is a comprehensive evaluation framework that challenges models across 57 diverse subjects, spanning from elementary science to advanced topics in economics and law. This benchmark effectively measures a model's reasoning capabilities and factual knowledge retention. Baselines. In order to assess the capabilities of the proposed method with respect to the current state of the art, we consider the following related baselines. Full Fine-tuning (FT): This approach initializes the model with pretrained weights and updates all parameters during the training process. Gradient computations are performed for the entire model.LoRA (Hu et al., 2022) . A popular parameter-efficient fine-tuning method that updates only a subset of model weights. LoRA approximates weight updates as the product of two low-rank matrices, significantly reducing the number of trainable parameters. The efficiency can be controlled by adjusting the rank of these matrices, known as the intrinsic dimension. We adopt the experimental setup from Zhang et al. ( 2023) for both LoRA and AdaLoRA implementations. This setup utilizes DeBERTaV3 (He et al., 2021a) as the pre-trained model and applies LoRA blocks to the following weight matrices: W q , W k , W v , W o , W f1 , W f2 . We compute the number of parameters trained by LoRA as:EQUATIONwhere d i is the dimension related to the weight matrix W f1 .AdaLoRA (Zhang et al., 2023) . It is an extension of LoRA that aims to limit the total sum of rank values used in different LoRA blocks. They define a computational budget and prune rank values according to an importance score (Zhang et al., 2023) . We compute number of training parameters in AdaLoRA using Eq. 17 with r which corresponds to the maximum rank value. According to Zhang et al. (2023) , r = b T nwhere n is the number of adapted weights and b T is the target budget. We report the number of parameters for b T ∈ {144, 576}, which results in r ∈ {3, 12}.DyLoRA (Valipour et al., 2022) : DyLoRA is another extension of LoRA, that enables adapting rank values dynamically. However, the goal of this method is to optimize the model fine-tuning for a range of ranks, in such a way that different versions of the fine-tuned model can be used if needed. Number of parameters for DyLoRA can be computed with Equation 17with r set to maximum rank.QLoRA (Dettmers et al., 2023) . QLoRA combines low-rank adaptation with 4-bit quantization to enable efficient fine-tuning of large models. Instead of fine-tuning the entire model, QLoRA applies 4-bit quantization to the pre-trained model weights, reducing memory usage while preserving model performance.It then fine-tunes the model by introducing low-rank updates, similarly to LoRA, but over the quantized Under review as a conference paper at ICLR 2025 model. This approach allows for fine-tuning on consumer-grade hardware with significantly reduced computational costs.We follow the setup in (Dettmers et al., 2023) , where 4-bit NormalFloat (NF4) quantization is applied to the weights of pre-trained models, followed by LoRA updates (see Table 5 for details on pre-trained models and hyperparameters).Metrics. To evaluate our proposed approach and compare it with related baselines, we employ two categories of metrics. The first category focuses on downstream performance, utilizing the GLUE (Wang et al., 2019) and MMLU (Hendrycks et al., 2020) benchmark datasets. The second category assesses efficiency, measuring the number of parameters (#params) and the number of Bit Operations (BOPs) for each method. To compute the BOP count we follow Van Baalen et al. ( 2020), which uses # Bit Operations as a hardware-agnostic proxy to model complexity and have an impact on energy level and device lifetime.According to Yang et al. (2017) and Van Baalen et al. (2020) , BOPs impact the energy consumption of the deployed model. Moreover, Yang et al. (2017) points out how the number of bits accessed scales linearly with the corresponding bitwidth and that most of the energy is consumed by the multiplication operations, which scales linearly with the used variables bitwidth. Therefore, we use BOPs as a proxy measure to show how the proposed approach affects the energy consumption with respect to the related baselines. A list of the downstream metrics used for the GLUE benchmark can be found in Appendix F.Quantitative Results. Table 1 presents the comparison between the proposed model and the related baselines described in Section 4.1. On all datasets, B-LoRA achieves on-par performance with all other baselines, while presenting a much lower BOPs. Specifically, our method shows slightly worse results for MNLI and QQP, but performs better than baselines on SST-2 and RTE (B-LoRA(q): 96.44 → AdaLoRA: 96.10 and B-LoRA(q+ra): 88.33 → AdaLoRA: 88.09, respectively). Interestingly, we can see that optimizing quantization levels and rank values results in better performances for RTE and STS-B datasets than using only quantization (B-LoRA(q+ra): 88.33 → B-LoRA(q): 86.52 and B-LoRA(q+ra): 91.76 → B-LoRA(q): 91.64, respectively). Moreover, Table 2 , presented in Appendix B, reports B-LoRA BOPs for every dataset within the GLUE benchmark, showing how quantization levels and amount of BOPs are correlated.Results on MMLU are summarized in Figure 3 . Results reported are the average accuracy on all 57 categories of questions. BLoRA with rank adaptation only performs on par with QLoRA, achieving 68.2.Compared to experiments on GLUE benchmark, rank adaptation without quantization performs better than with quantization on both models: accuracy is decreased by 6% for Qwen-2 and 5% for Phi-2. This decrease is not observed on GLUE benchmark.Qualitative Results: Task-Specific Head Quantization Levels. We examine precision levels of taskspecific head layers after fine-tuning. In all experiments layers of the task-specific head remained at the highest possible precision (32 bit). This result aligns with findings reported by Van Baalen et al. (2020) , where they observed that the first and last layers were kept in higher precision in most of their experiments, however, we only observed higher precision in the last layers. Since Task-Specific Heads plays a central role when fine-tuning a pre-trained model, quantizying their weights has a big impact on downstream performances.LoRA blocks quantization levels and rank value patterns. We analyzed the distribution of quantization levels and rank values after fine-tuning. We observed that B-LoRA matrices are often kept with low precision of 2 or 4 bits, while pre-trained weights are usually kept with higher precision. Plots of quantization levels distribution can be found in Appendix H. A correlation between the quantization level of pre-trained weights and final output and the dataset size is present: the newer data the model observes during training, the less precision of pre-trained weights is needed. Indeed, datasets with a training set size below 10k (RTE, MRPC, STS-B, CoLA) present a median number of bits used above 8, while the remain Under review as a conference paper at ICLR 2025 ones (SST-2, MNLI, QNLI, QQP) use a median number of bits below 8. We hypothesized that there might be a correlation between specific attention weights (i.e., W k , W q , and W k ), optimal precision level, and related rank value. In accordance to our hypothesis, Figure 2 shows that W v has on average larger rank values, compared to W k , W q , which indicates that most of the information is retained within attention values. On the other hand, queries and keys can discard most of the information, since they are only used to compute attention weights and highlight the information retained within attention values. A similar pattern can be observed in Figure 1 , where B-LoRA blocks used for values use more bits on average. In Appendix G, AdaLoRA rank values are provided for budget b = 576. The overall pattern observed in Zhang et al. (2023) aligns with our results, however, for B-LoRA rank reduction is more significant, since many LoRA modules are truncated to rank value 1. 
discussion: 
 In this work we present B-LoRA, a parameter-efficient fine-tuning approach based on LoRA that allows to optimize quantization levels and rank values using Bayesian gating mechanisms proposed by Van Baalen et al. (2020) . While works such as DyLoRA (Valipour et al., 2022) and AdaLoRA (Zhang et al., 2023) propose different approaches for optimizing rank values, they do not quantize variables and weights. Moreover, while our approach does not require any hyperparameter search, AdaLoRA requires specifying several hyperparameters for every dataset (i.e., computational budget, scheduler hyperparameters, learning rate). The main limitation of this work is that B-LoRA is only evaluated on the GLUE and MMLU benchmarks, while both LoRA and AdaLoRA provide results for natural language generation (Narayan et al., 2018; Hermann et al., 2015) . In future works we will validate the model on the two question answering (QA) benchmarks SQuADv1.1 (Rajpurkar et al., 2016a) and SQuADv2.0 (Rajpurkar et al., 2018a) , as well as the E2E benchmark (Novikova et al., 2017) , using GPT-3 (Brown et al., 2020a) as pre-trained model. 
conclusion: 
 In this study, we introduced Bayesian-LoRA (B-LoRA), a novel approach for optimizing quantization levels and rank values in model parameters, using Bayesian techniques. Our method extends the Bayesian-Bits framework by Van Baalen et al. (2020) , enabling a hardware-friendly and adaptive quantization that significantly reduces computational demands without sacrificing model performance. We empirically demonstrated that B-LoRA achieves competitive results on the GLUE and MMLU benchmarks, matching or even surpassing state-of-the-art methods such as LoRA, DyLoRA, and AdaLoRA, while also reducing bit operations by approximately 70%. This efficiency is achieved without the need for extensive hyperparameter tuning, contrasting sharply with approaches like AdaLoRA that require detailed configuration, tailored to each dataset. However, our evaluation was limited to the GLUE benchmark. Future work will aim to validate B-LoRA across a broader range of tasks, including question answering and natural language generation, using benchmarks like SQuAD v1.1 (Rajpurkar et al., 2016b) and 2.0 (Rajpurkar et al., 2018b) , and the E2E generation benchmark (Novikova et al., 2017) . Additionally, applying B-LoRA to other pre-trained models like GPT-3 (Brown et al., 2020a) will help establish its utility and robustness in diverse natural language processing contexts.Overall, B-LoRA presents a promising direction for energy efficient, scalable, and effective model finetuning, making a step to bridge the gap between computational efficiency and performance.Under review as a conference paper at ICLR 2025 A APPENDIX B ADDITIONAL RESULTS  
 
 Appendix: In contrast to AdaLoRA, where different set of hyperparameters is used for every dataset as shown in Table 4 , most of the hyperparameters in our experiments are the same for all datasets. The only value that is changed is number of training epochs, which can be found in Table 3 . (l) = C o * W * H * W i * W f * H f , where C o -number of output channels, W i -number of input channels, W, H -dimensions of output map, W f , H f -dimensions of filterUnder review as a conference paper at ICLR 2025 A BOP corresponds to Bit OPerations, as defined in (Van Baalen et al., 2020) . BOP count measures multiplication operations, multiplied by bit width of the corresponding components, which makes this metric a hardware-agnostic estimate of the complexity of a model. BOP count is computed the following way: BOPs(l) = MACs(l) * b w * b a where b w , b a are weight and input activation bit width, respectively. BayesainLoRA method is additionally compared to AdaLoRA in terms of BOP count. Below derivation of BOP and MAC for self-attention mechanism is provided.Self-attention is a basic block of transformer models (Vaswani et al., 2017) . For evaluating B-LoRA, BOP is computed for self-attention blocks of DeBERTa-v3 and compared to BOP of the same blocks with all weights and activation set to highest possible precision (32 bits).Self-attention module is parameterized with 3 matrices W k , W q , W v ∈ R × where d is a hidden size of a model. Define maximum length of an input sequence as l, then MACs(q) = MACs(k) = MACs(v) = d 2 * l Other operation that increases MAC count for self-attention is dot product between keys and queries (attention scores). Assuming that number of attention heads is h, MACs of attention scores can be computed as where last term corresponds to a scaling factor.Since in all experiments DeBERTa-v3 was used, MAC calculations need to be extended to attention variant proposed by (He et al., 2020) . Disentangled attention utilizes positional information by introducing two extra matrices for keys and queries that are applied on positional embeddings. Then scores between positional keys and queries (context to position) and positional queries and keys (position to context) are computed and added to the attention scores.Computations described above have components for which MAC need to be calculated. Assuming that positional embeddings size is e: LoRA is applied to 6 matrices in attention layer. W q , W k , W v , W o have d 1 = d 2 = d, therefore, number of parameters in each of them is(r × d) + (d × r) = 2 × r × d (19)Additionally, it is used in intermediate and output layers of attention, W f1 ∈ R d×di , W f2 ∈ R di×d . Number of trainable parameters in each of these layers is:EQUATIONSumming parameters for all weights in attention layer results in:EQUATIONFor a model with l layers, number of trainable parameters in the encoder is:EQUATIONE.2 B-LORA B-LoRA is applied for W q , W k , W v ∈ R d×d . In total, it gives #params = 2 × l × r × (3 × d) = 6 × l × r × d (23) parameters. Github link to Bayesian-LoRA implementation: https://github.com/KseniaSycheva/ Bayesian-Lorahttps://github.com/Qualcomm-AI-research/BayesianBitshttps://github.com/QingruZhang/AdaLoRA/ 


Decision: The paper revisits well-known Bayesian tools and tricks to build adaptive quantization and low-rank model adaptation with adaptive rank.

The reviewers acknowledge the technical novelty and insights of the current manuscript, but all (including the AC) criticize the soundness of the empirical evaluation. The AC would suggest including results on more tasks and neural architectures in the future manuscript.