Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation
Authors: Vaibhav Seth
Ayan Sengupta
Arinjay Pathak
Aastha A K Verma
Natraj Raman
Sriram Gopalakrishnan
Niladri Chatterjee
Tanmoy Chakraborty
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) are highly resource-intensive to fine-tune due to their enormous size. While low-rank adaptation is a prominent parameter-efficient fine-tuning approach, it suffers from sensitivity to hyperparameter choices, leading to instability in model performance on fine-tuning downstream tasks. This paper highlights the importance of effective parameterization in low-rank fine-tuning to reduce estimator variance and enhance the stability of final model outputs. We propose MonteCLoRA, an efficient fine-tuning technique that employs Monte Carlo estimation to learn an unbiased posterior estimation of low-rank parameters with low expected variance, stabilizing fine-tuned LLMs with only $\mathcal{O}(r)$ additional parameters, for a given rank $r$. MonteCLoRA shows significant improvements in accuracy and robustness, achieving up to $3.8$% higher accuracy and $8.6$% greater robustness than existing efficient fine-tuning methods on natural language understanding tasks with pre-trained RoBERTa-base. Furthermore, in generative tasks with pre-trained LLaMA-1-7B and LLaMA-3.2-3B-Instruct, MonteCLoRA demonstrates robust performance with $50\%$ and $62\%$ lower spreads, respectively, than the contemporary, efficient fine-tuning methods. The theoretical and empirical results presented in the paper underscore how parameterization and hyperpriors balance exploration-exploitation in the low-rank parametric space, therefore leading to more optimal and robust parameter estimation during efficient fine-tuning.

Published in Transactions on Machine Learning Research (08/2025)
Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation
Ayan Sengupta*
agon.senguptadeee.itddoo
Indian Institute of Technology Delhi, India
Vaibhav Seth"
Vaibhav.Seth.co mL111 Amaths..itte ac ac.in
Indian Institute of Technology Delhi, India
Arinjay Pathak
ais2486788ssri. itt.:ssoiiiiiid.ooooooo
Indian Instiuute of Technology Delhi, India
Aastha Verma
cs11211670@itd..c...
Indian Institute of Technology Delhi, India
Natraj Raman
natrijyramaneijimongn..com
JPMorgan Al Research
Sriram Gopalakrishnan
JPMorgan Al Research
Niladri Chatterjee
wsleddri@mats..ittd..o
Indian Institute of Technology Delhi, India
Tanmoy Chakraborty
tanchak-@ittd.co ac.
Indian Institute of Technology Delhi, India
Reviewed on OpenReview: https: openreview. net/ forum? id= 2HFmd cB8kh
Abstract
Large Language Models (LLMs) are highly resource-intensive to fine-tune due to their enor- mous size. While low-rank adaptation a prominent parameter-efficient fine-tuning ap- proach, suffers from scnsitivity to hyperparameter choices, lcading to instability model performance fine-tuning downstream tasks. This paper highlights the importance of ef- fective parameterization in low-rank fine-tuning to reduce estimator variance and enhance the stability of final model outputs. We propose MonteCLoRA, nique that employs Monte Carlo estimation to learn an unbiased posterior estimation of low-rank parameters with low expected variance, stabilizing fine-tuned LLMs with only O(r) additional parameters, for a given rank r. MonteCLoRA shows 0.5% and 1.6% improve- ments in accuracy and robustness over unregularized low-rank adaptation method on natural language understanding tasks with pre-trained RoBERTa- base. Furthermore, in generative tasks with pro-trained LLaMA-1-7B and
efficient fine-tuning tech-
MonteCLORA demonstrates robust performance with 50% and 62% lower spreads respectively than the contemporary efficient fine-tuning methods. The theoretical and empirical results presented in the pa- per underscore how parameterization and hyperpriors balance the low-rank parametric space, therefore leading to more optimal and robust parameter estimation during efficient. fine-tuning.
in
'Equal contribution
o *Corresponding author

---

Published in Transactions on Machine Learning Research (08/2025)
1 Introduction
The rise of large language models (LLMs) has initiated transformative shift in natural language processing. revolutionizing extensive array of tasks (Zhao et al. 2023 Chang et al. 2024). Vaswani et al. (2017) introduced the self-attention-baseee
Transformer architecture, which is capable of more efficient handling of long-range dependencies in texts than prior methods that relied on recurrent neural networks (RNNs) and convolutional neural networks (CNNs). Since then, a monumental shift has begun in developing Transformer- based pre-trained language models (PLMs) for solving a wide range of lasks involving natural languages. Over the past few years, the size of these PLMs has dramatically increased from multi-million parameter BERT (Devlin et al. 2018), RoBERTa (Liu et al. 2019), T5 (Raffel et al. 2020) models to recently developed billiibiliin parameter DeepSeek (Liu et al.) 2024a). LLaMA (Grattafiori et al.) 2024b Touvron et al, 2023), Falcon (Almazrouei et al., 2023) and Mistral (Jiang et al. 2023) models. Scaling laws of language models (Kaplan et al. 2020) suggest that the superior performance of these models scales with pre-training data size and the required computation. With deeper and larger models and more extensive pre-training, these models exhibit, emerging properties such as zero-shot and few-shot in-context learning (Brown et al. 2020), complex reasoning, and gencralization capabilities. Despite these emerging properties, LLMs require fine-tuning on downstream tasks for competitive performance (Liu et al. 2022b and domain and task adaptation.
Given their enormous size and computational requirements, fine-tuning LLMs on every downstream task is often unrealistic and computationally infeasible. For feasibly fine-tuning LLMs, parameter-efliient fine- tuning (PEFT) techniques such as Adapters (Houlsby et al. 2019), selective fine-tuning (Zaken et al. 2021), and low-rank adaptation (Hu et al., (022) have become immensely popular. Among these PEFT mothods, low-rank adaptation (LoRA) has garnered significant attention due to its flexibility, adaptiveness, and ability to mitigate catastrophic forgetting during fine-tuning. LoRA reparameterizes pre-trained model weights to lower dimension, such that only the low-rank matrices are tuned during fine-tuning, keeping the weights of the original pre-trained model frozen. The low-rank decomposition significantly reduces the number of trainable parameters during fine-tuning, offering great computational benefits. For instance, with a latent rank of 8, the number of trainable parameters of a RoBERTa-base {Liu et al.) 2019) model can be reduced by 99% (from 110M to 0.3M) with LoRA. Despite its effectiveness, recent studies (Liu et al. 2022c Valipour et al. 2022 Biderman et al. 2024) showed that LoRA is sensitive to hyperparameters like learning rate and training batch size and often requires longer training iterations for convergence.. Figure [ illustrates the performance of full fine-tuning (where we fine-tune the whole pre-trained LLM) and LoRA fine-tuning on different natural language understanding lasks with the pre-trained RoBERTa-base model under different learning rates and training batch sizes. The results highlight that the distribution spread for accuracy (difference between maximum and minimum accuracy) on the validation dataset can go up to 17 and 24 points for LoRA and full fine-tuning, respectively. Although the inter-quartile range (IQR) for LoRA remains modest for most tasks, the high distribution spreads suggest that both LoRA and full fine-tuning are sensitive to marginal cases. However, as we discuss in the subsequent sections, distributional spread is not always the best measure of robustness due to outlier sensitivity (full fine-tuning more sensitive to outliers than LoRA) and distribution median forms a better metric for measuring robustness. We observe that median performance of full fine-tuning is higher than LoRA in 4 out of 5 tasks. We further observe that the accuracy spread for LoRA, particularly for generative tasks remains considerably high relative to alternative hods and median accuracy is also fairly low compared to other methods, underscoring the need for more stable formulation of low-rank adaption methods for fine-tuning LLMs. These observations highlight that careful consideration of hyperparameter selection is necessary to balance adapting new knowledge and preserving the pre-trained knowledge. The most obvious way to figure out the most appropriate hyperparameters is to perform extensive hyperparameter tuning [Tribes et al. 2023 Xie et al., 2024). However, unlike small-scale machine learning models, performing grid or random scarch within the hyperparameter space of LLMs is very costly and impractical. Bayesian methods (Wilson & Izmailov] 2020 Wang & Yeung, 2020), on the other hand, offer an organized solution to hyperparameter sensitivity by marginalizing the predictive distribution. Through appropriate knowledge priors, Bayesian methods diminish the importance of hyperparameter tuning (Papamarkou et al. 2024a] and offer robust alternatives to post-hoc regularization techniques while training on small datasets.
2

---

Published in Transactions on Machine Learning Research (08/2025)
90-
90-
80
Accurso
Accuratty
70
60-
50-
40
19.2
7L.3
800 768
B81 BA8 507
103
327
BL7
78.
WTC
BoolQ
CoLA Task
MIPC
5S12
MNU
0GP
ONU
Pull
LoRA
MonbeCl oRA [ours]
Full
LoRA
MonteCt (ours)
(a) GLUE Small Tasks
(b) GLUE Large Tasks
Figure 1: The distribution of validation accuracy for the fine-tuned RoBERTa-base under different learning rates and batch sizes. We present the findings for two approaches: full fine-tuning (denoted Full FT) and low-rank adaptation (denoted as LoRA). Full FT tends to be more sensitive to outliers and exhibits wider spread (the difference between the maximum and minimum scores) compared to LoRA. However, LoRA's median performance lower than that of Full FT in 4 out of 5 small GLUE tasks, suggesting that Full FT offers greater robustness than LoRA when the fine-tuning data limited. In the larger GLUE tasks, LoRA gives better median value but the spread of accuracy much greater for Full FT. Our proposed method, MonteCLoRA demonstrates better average spread and median performance than both Full FT and LoRA.. This highlights the significance of appropriate parameterization for achieving stable low-rank adaptation in LLMs.
Motivated by the advantages of Bayesian methodologies, we propose a robust low-rank adaplation method for fine-tuning LLMs efficiently. Contrary to the existing sensitivity studies of LoRA, our work provides a structured overview of the challenges faced by LoRA and the full fine-tuning method and proposes a systematic approach to mitigate these challenges. To overcome the sensitivity challenges of LoRA fine- tuning, we propose Monte Carlo-enhanced low-rank adaptation method, MonteCLoRA, which learns posterior distributions of low-rank parameters with appropriate prior distributions. MonteCLoRA parameterizes the low-rank paramcters as mixture of multivariate Gaussian distributions, where each distribution's 's precision matrix is assumed to follow Wishart knowledge prior. Through Monte Carlo estimation from multiple parameters sampled from the parameter space, MonteCLoRA stabilizes the reparametrized parameters and generates robust and unbiased low-rank adaptation for LLMs.
Our theoretical and empirical results justify the importance of parameterization for low-rank adaptation of LLMs. We perform thorough empirical analysis with five natural language understanding (NLU) and six natural language generation (NLG) tasks with three pre-trained LLMs RoBERTa- basse (Liu et al.
2019), LLaMA-1-7B (Touvron et al. 2023). abd LLaMA-3.2-20 2-.-2----Innseeee Instruct (Grattafiori et al. 2024a). Empir- ical results on NLU tasks suggest that MonteCLORA is more stable, where the average spread of accuracy distribution is 10% lower than LoRA and 50% lower than full finc-tuning. In torms of the robustness metrics, MonteCLoRA is 5% more robust. and achieves 2% better accuracy than LoRA fine-tuning. Remarkably, on Commonsense NLG tasks with LLaMA-1-7I. MonteCLoRA has a 53% lower spread (2.19 points) than LoRA (4.69 points) with zero-shot validation accuracy distribution. With LLaMA-3.2 3.--BB--BBBBB- Instruct MonteCLoRA re- duces the spread by 62% average and exhibits 13.3% higher robustness over LoRA. Our further in-depth analysis highlights the superiority of MonteCLoRA in terms of stable and faster convergence while fine-tuning LLMs.
The key contributions of our work can be summarized as follows
The source code of MonteCLoRA made available http::////gttthoooo////////////0

---

Published in Transactions on Machine Learning Research (08/2025)
Our work provides an in-depth theoretical analysis of the impact of hyperparameters on fine-tuning LLMs and highlights the key challenges with low-rank adaptation techniques. To the best of our knowledge, no comprehensive study exists on the sensitivity analysis of LLM fine-tuning.
We propose Bayesian alternative to the low-rank adaptation of LLMs for estimating trainable parameters during fine-tuning. The paper theoretically justifies the robustness of the posterior estimation.
The proposed fine-tuning method adds only O(r) additional parameters for posterior estimation and shown to be effective both the performance and the stability of the fine-tuned LLMs.
We provide a thorough empirical study where we demonstrate the effectiveness of the proposed fine- tuning methods with two pre-trained language models RoBERTa-beee and LLaMA-1-7B on five NLU and six NLG (commonsense reasoning) tasks.
The paper is organized as follows. Section 2 describes the related work on efficient fine-tuning strategies for LLMs. Scction 8 claborates on the background concepts used in the paper. Scction | describes our proposed method, MonteCLoRA, along with the theoretical results obtained in this work. Section describes the experimental details used in the empirical study done in the paper. In Section i present the results of the empirical study. Section highlights few case studies to analyze robust fine-tuning of LLMs. Additional background materials and supplementary results are furnished in the appendix.
2 Related Work
In this section we briefly describe the related work around robust fine-tuning LLMs. We divide this section into three broad subjects (1) fine-tuning methods for LLMs, (2) efficient fine-tuning methods, and (3) Bayesian methods for robust fine-tuning.
Fine-tuning LLMs. Fine-tuning refers continually training PLMs smaller, task-specific dataset. Fine-tuning allows the model to adapt its generalized capabilities to perform better particular tasks, domains or applications that are not prevalent during the pre-training phase. For the given task-specific training data Z = {();,,))) in full fine-tuning of LLMs, the model starts with pre-trained weights Og and updates to = 0. + Al by maximizing the conditional language modeling objective
Mul
max Σ
Σ (z t=1
log Po x. Mex).
In one the foundation works, Devlin et al. 2018) demonstrated that models pre-trained on large text corpora could fine-tuned with just one additional output layer to perform a wide range of lasks, effectively transferring learned knowledge to specific tasks. Howwadd & Ruder (0018) introduced the concept of universal model fine-tuning, emphasizing the importance of discriminative fine-tuning and gradual unfreezing for each model layer to adapt effectively to different downstream tasks. This approach helps mitigate the catastrophic forgetting problem often observed in LLMs. However, as the models became larger and more capable, fully fine-tuning LLMs turned out to be more computationally infeasible.
Parameter-cfficien methods of fine-tuning. PEFT aims to adapt large PLMs to specific tasks by optimizing small subset of the model's parameters rather than undertaking the computationally expensive training the entire pro-trained model. These tochniques significantly reduce the computational overhead and memory usage typically associated with fine-tuning large models. There are three broad classes of PEFT techniques additive, selective, and reparameterization- based. Additive PEFT strategies (Houlsby et al, 2019 Pfeiffer et al. 2020) modify the underlying model's architecture by injecting new additive trainable parameters. Selective PEFT methods [aken et al.] 2021 Sung et al. 2021) select a subset of parameters for fine-tuning, and the rest remain frozen. On the other hand, reparanneterization--aass PEFT met introduce low-dimensional reparameterized parameters for fine-tuning.
One of the most prominent efficient fine-tuning methods, LoRA Hu et al. 2022), uses low-rank reparam- eterization to reduce the learnable parameter space to lower-dimensional space. Subsequently, several
4

---

Published in Transactions on Machine Learning Research (08/2025)
alternatives to LoRA have been proposed, mostly keeping efficiency and downstream performance in focus. One such method, AdaLoRA (Zhang al.] 2023), dynamically allocates parameter budgets among weight matrices based on their importance, employing singular value decomposition to optimize incremental up- dates. Sparse LoRA (SoRA) extends LoRA by incorporating a gate unit optimized with a proximal gradient method (Ding et al., 2023). allowing dynamic adjustments to the intrinsic rank during training and effec- tively controlling the sparsity of updates. DoRA (Liu et al.) (02-bb) decomposes the pre-trained weight into magnitude and direction components for fine-tuning, utilizing LoRA for efficient directional updates, which enhances both the learning capacity and training stability without increasing inference overhead. Although these PEFT methods demonstrate superior downstream performance in terms of extrinsic metrics like accu- racy, these methods are susceptible to task-specific configurations. For instance, SoRA depends on tuning the sparsity controls, introducing instability during training due to abrupt changes in sparsily levels. More- over, these methods often overfit when fine-tuned on small catasets and generate overconfident predictions during inference (Yang et al. 2024).
Bayesian Methods for Model Robustness. Bayesian frameworks are particularly effective in handling model uncertainty (Daxberger et al. 2021 Zhang et al. 2021 Deng et al. 2022), offering robust solution to this issue by enabling a probabilistic inter interrrctttion of model weights, which helps in assessing the uncertainty of the predictions. Papamarkou et al. (024b) strongly argued that Bayesian deep learning is more favorable over similar frequentist alternatives. This is because Bayesian methods provide advantages that can help overcome many of the challenges that deep learning faces. Bayesian deep learning is known to reduce the importance of hyperparameter tuning by incorporating relevant hyper-priors. Bayesian deep learning is also known to enable domain knowledge priors, as opposed to post-hoc regularization on small datasets. Uncertainty quantification is another aspect where Bayesian methods gain an advantage over the alternative frequentist methods by using it to improve the reliability of the decision-making process, which helps the model generalize better on dit-off-dstibuttion inputs. Also, by dynamically updating prior beliefs in response to new evidence, Bayesian frameworks allow selective retention of valuable information from previous tasks while adapting to new ones. This mitigates the issue of model decay, which occurs in static models, assuming that underlying data patterns remain constant over time. Tn a recent development, Taplace-LoRA. (Yang) let al.) 2024), Bayesian post-hoc treatment of LoRA was introduced that leverages Laplace approximation to estimate the posterior distributions of the parameters involved in the low-rank adaptations, significantly improving the calibration of fine-tuned models. Albeit robust solution to the overconfidence problem of PEFT methods, Laplace-LoRA is a post-hoc calibration method requiring longer training iterations to bring the low-rank parameters from an unstable basin (a subspace associated with the same local optimuum to a more stable parametric space. Therefore, Laplace-LoRA often leads to sub-optimal downstream performance.
Uniqueness of MonteCLoRA. Our mothod balances performance and stability through Monte Carlo esti- mation over the low-dimensional parametric space. With appropriate parametric assumptions, MonteCLoRA can provide unbiased and robust estimate for the posterior distribution, providing excellent stability and performance gain. Moreover, unlike existing Bayesian methods like Laplace-LoRA, MonteCLoRA can be used on both ad-hoc and post-hoc basis and, therefore, can follow the same optimization path as the other PEFT methods to learn the reparameterized model parameters, ensuring better flexibility and adaptiveness.
3 Background
In this section we elaborate the concepts of LoRA for fine-tuning LLMs. We also lay down the theoretical motivation behind robust low-rank paramcterization. All the mathematical notations used in this section are sunmarized in Table
3.1 LoRA for Fine-tuning LLMs
LoRA leverages low-rank matrix decomposition to incorporate trainable parameters that adapt the pre- trained model's weights in a parameter-efficient manner. Consider pre-trained weight matrix W, e Brout Xna, LoRA constrains its update by expressing it through low-rank decomposition as follows:
W Wn + AW W, + BA.
5

---

Published in Transactions on Machine Learning Research (08/2025)
Notation Type
Notation
Description
Wo
Pre trained weight matrix
Rm
Input dimeusions of W
Low-rank adlaptation
Rout
Output dimensions of W/
LoRA rank
LaRtA matrix
B
LoRA matrix
Objective loss function
Model parameters
VJ
Gradient of the loss function
L
Lipschitz constant of the loss function
Standard deviation bound of the stochastic gradieuts
Optimization
Stochastic noise in the gradient
H
Hresian matrix of loss function
Eigenvalue matrix
Point for value theorem
Gaussian random variable added the model parameter
Amax
Maximum eigenvalue of Hessian
Table 1: Glossary of all the mathematical notations used in Scction
where B E Routtt A E grx",, and the rank < min(Nin Nont). During training, Wo remains frozen and is not subject to gradient updates, while the matrices, A and B, contain the trainable parameters. Both Wo and AW = BA are applied to the same input, and their resulting output vectors are summed element- wise. This approach allows LoRA to efficiently adapt large pre-trained models to new tasks with a relatively small number of additional parameters. Figure provides pictorial illustration of LoRA.
3.2 Sensitivity of Gradient Descent Hyperparameters
Stochastic gradient descent (SGD) a widely popular optimization tech- nique for training neural networks. In SGD, we calculate the gradient of the training objective function with respect to the learnable neural param- eters sample training batch and iteratively update the parameters towards the edge of basin of optima, shown by Izmailov et al. (019)), where the training objective is minimized. In this section, we describe some of the properties of SGD to understand how different hyperparam- eters such as optimizer learning rate impacts the optimization process.
Output
W
TaRA A
Inpus
3.2.1 Factors Influencing Convergence of SGD
Figure 2: Illustaation of the
The convergence of SGD is known to depend on the Lipschitz constant through its influence on the choice step sizes. Several studies have established an inverse relationship between the maximum step sizes that ensure convergence and the learning rate. Ghadimi Lan (2013), in their seminal work, demonstrated that selecting the learning rate within specific range guarantees the convergence of the SGD algorithm, based foundational assumptions in stochastic optimization. The following lemmas formalize the relationships between the convergence of SGD with the step size of the algorithm?
LoRA fine-tuning framework.
Lemma 3.1 Let J: R -
+R R. be a twice continuously differentiable function, where n E N is the dimension the space of model parameters. Then the gradient JJ Lipschitz continuous with Lipschitz constant L equal the supremam of the marimum eigenvalue of ils Hessian matris (()) V2J(
L sup Xmaz (H()). BER"
(1)
2Proofs Lemma 3. and 3.2 are furnished in Section of Appendix.

---

Published in Transactions on Machine Learning Research (08/2025)
Lemma 3.2 Let [0) be the loss function over trainable model parameters 0 E R", and y represents Gaus- sian noise with zero meen. We define the expected loss function as [() [)0++ We also define H(0) as the Hessian matria for J(0) and H() as the Hessian matrix for J(0). Then the marimum eigenvalue of H))) at any given is less than the marimum eigenvalue of H over all R
Lemma 3.3 Training a model with Gaussian noise added to parumelers is less sensilive to learning rales.
Proof. Let the model parameters be denoted by 0, and the Gaussian noise added these parameters by y- The learning rates of the model with and without Gaussian noise are represented as fj and 77, respectively, and the corresponding Lipschitz constants are L and L. The remaining notations follow from the previous lemmas. From Lemma 3.2 we know that for all 0 E R.,, the maxinum eigenvalue of the Hessian of the smoothed loss function satisfies,
max (0) < Amax-
Taking the supremum over all o and using the properties of the supremum, we obtain
Amax S Amax
where Amax represents the supremum of the maximum eigenvalues of the Hessian of the original loss function J(0), and max is the supremum of the maximum eigenvalues of the expected Hessian of the smoothed loss function J()). From Lemma 3.1 we have the relationship between the Lipschitz constants and the supremum of the Hessian cigenvalues,
= Amax and ī= Amax.
Therefore, we conclude
LSL.
Since L < L, and we know that the range of learning rates on which we can guarantee convergence is inversely proportional to the Lipschitz constant, we can say that training with Gaussian noise permits a broader range of allowable learning rates, thus reducing sensitivity the choice of learning rate during optimization.
3.3 Measuring Robustness of a Fine- Muni-Tunng Strategy
The previous section discusses the theoretical underpinning of sensitivity of a fine-tuning strategy. Therefore, it is of immense importance to quantify the sensitivity (or robustness) of a fine-tuning method. Robustness in statistical modeling refers to the ability of model to perform well and provide reliable results across a perform reliably under
wide range of conditions and assumptions. In model fine-tuning, robust strategy different hyperparameters affecting the fine-tuning process, such as the learning rate of the optimizer, training batch size, etc. Although robustness measures can be defined differently, we resort to statistical measures to quantify the robustness of different fine-tuning strategies. Suppose a model M is fine-tuned on a given task T with two different tinettunigg strategies, S(1) and 8(2), with hyperparameter settings, A Ag...., Ag and obtains scores ,/")""))) s...s (1) and 5(2) (2) .... Sk respectively on the validation dataset. Depending on the nature of these scores, we can calculate the metrics to compare the robustness of these fine-tuning strategies. For intrinsic metrics like negative loglikelihood, we calculate intrinsic robustness as,
1
Rga med((>) (1) 82 (1)
(2)
Here, med is the median of the score distribution. Similarly, for extrinsic metrics like accuracy, we calculate eatrinsic robustness as,
Rgn = med(fs .62 . (
(3)
During model validation, typically aim for minimizing the intrinsic metrics (for instance, lower validation
loss is expected) and maximizing the extrinsic metrics. Therefore, having Rgn > Rga ensures that the probability of achieving lower intrinsic metric is higher for S(1) than S(2), indicating more robustness for S(1), A similar argument also holds for the extrinsic robustness metric, where a higher score indicates a more robust fino-tuning strategy. As we use the same hyperparameter configurations for comparing different strategios, these robustness metrics allow us to compare different methods without incorporating the variances within the hyperparameters.
7

---

Published in Transactions on Machine Learning Research (08/2025)
Notation
Description
i-th column vector in the low rank matrix
A
i-th column meen vector
W
Wishart Distirbution
Scale matrix Wishart distribution
Sompled emaariance matrix
Dirichlet Distirbution
Dirichlet distribution concentration parameter
II
Mixture weights vector
A-th mixture weight
N
Number of inner Monte Carlo samples
M
Number of outer Moute Carlo sample
W aoochaviic
Stochastic componeut sampled from the Monte Carlo process Sample scales
Table 2: Glossary of all the mathematical notations used with MonteCLoRA.
3.4 Bayesian Inference and Monte Carlo Estimation Methods
In the Bayesian treatment of machine learning and deep learning models, we are often concerned with the predictive distribution and aim to incorporate prior information into our estimates. Bayesian methods typically regularize the weight space of the model, resulting in a robust reparameterization of the models. With Baycsian formulation, we can paramctorize the probability distribution of model parameter as,
P(O) = P(O P)P(B)
(4)
where 33 denoles the parameters that we learn to represent the distribution of the weight space of the model. The integral is often highly intractable due to the complex and high-dimensional nature the distributions involved. Monte Carlo estimation provides a simple computational algorithm for calculating computationally intractable statistics to calculate analytically.
4 Proposed Methodology
In this section, we introduce proposed method, MonteCLoRA, a Monte Carlo enhanced Low-Rank Adaptation for fine-tuning I.MMs. Tt. learns posterior distribution for each low-rank A parameter and esti- mates robust and unbiased posterior estimator. With Monte Carlo estimation, MonteCLoRA prevents mode collapse of the low-rank parameters, instead learns to sample from a basin of optima for improved general- ization and robustness. Figure 3 illustrates our overall methodology. Table 2 describes all the mathematical notations descríbe MonteCLoRA.
4.1 Gaussian Factorization of Weight Space
Learning a posterior distribution over the entire weight space of a model a complex and computationally demanding task. To address this, we adopt factorization assumption, where we presume that the modcl's weight space discribution can be decomposed into the product of the distributions across all layers. This approach decouples the distribution learning of one weight matrix from others, enabling efficient parameter estimation.
Our primary focus is on low-rank matrices, though this methodology extends to full-rank matrices. Suppose we have a low-rank weight matrix E [R We denote the probability distribution of its weight space as P(O). key assumption that each the nin column vectors (0, E IR") independently follows Gaussian distribution with shared covariance matrix but with distinct means,
P(0) = P(0,, 02. On..) = II P(O.).
(5)
1=1
We parameterize the distribution of each 0, by assumiig that, given covariance matrix Σ Rx each 0, is generated from N(,, E), with μr being the corresponding column mean vector. Using common covariance
8

---

Published in Transactions on Machine Learning Research (08/2025)
Wishart Prior
LoRA A Variance
Dirichlet Prior
A
Σ
II
Output
Sample
Sample Weights
Sample
LoRA B
W
LoRA A Prior
π
Convex Averaging
Sample
Input
Learnable vector parameters
Latent vectors
Learnable matrix parameters
Latent matrices
Frozen parameters
Figure 3: Overview of MonteCLoRA for Bayesian Estimation of LoRA Parameters. MonteCLoRA models the LoRA-A matrix as a mixture of Gaussians centered at shared learnable mean with covariance matrices E sampled from a Wishart prior. Mixture weights æ are drawn from a Dirichlet prior. (1) A diagonal seale matrix V defines the Wishart prior from which covariances Σ are sampled. (2) & samples are drawn from N(J, Σ) produce candidate LoRA-A matrices. (3) A convex combination of these samples is computed using Dirichlet weights π Dir(a). This approach introduces only O(r N) additional parameters per LoRA layer.
matrix allows for efficient posterior learning without adding a significant number of additional parameters. The covariance matrix is not a fixed parameter; instead, it is sampled from a Wishart distribution (for details on the distribution, check Appendix C) with learnable priors. The learnable prior parameters enable explore the low-rank parametric weight space, leading more robust parameter estimates for the model. We define P(O,) as weighted sum of N samples drawn from common multivariate Gaussian distribution:
N
P(O.) =
Σ Te 0 s.t. b=1
Σ T = 1=1
(6)
where each sample 0i.k (N, Σ) is independently drawn from single Gaussian distribution with shared mean and covariance Σ. The mixture weights II = (2.., TN) are drawn from Dirichlet distribution, Dir(a), with concentration parameter a = (a1, a2, aN). Unlike a traditional Gaussian mixture model, this formulation does not involve separate components with different means or covariances; instead, it constructs a convex combination multiple samples from the same Gaussian prior.
The mean vector μ4 initialized randomly the start of training (essentially the LoRA A matrices) and is subsequently learned. The covariance matrix Σ is sampled from Wishart prior distribution W, nm)): where V is learnable diagonal scale matrix, Rin is the degrees of freedom, and r is the LoRA rank. Given the independence assumption across dimensions, we restrict V to be diagonal. Therefore, using Equations and i we can write the joint density as,
Plam
PI
Σ Thk :. Dir(II: W, V, V.n)).
(7)
Note that different LoRA parameters may use distinct covariance matrices learned with different Wishart scale matrices.

---

Published in Transactions on Machine Learning Research (08/2025)
4.2 Monte Carlo Estimation of LoRA Parameters
Given that II and Σ are independent, we can simplify the Monte Carlo estimation for the expectation of 0,. The expectation of 0, is,
0 / P40 II, II P(TT) P(E) d0, dII dΣ.
In the Monte Carlo formulation, we approximate this integral by sampling from the distributions and E and then computing the expectation over 0, given each sampled pair (II, Σ) and a fixed p4
1. Sample II and? Σ
(a) Draw M samples IT6) Dir(a), where j = 1...., M.
(b) Draw M samples [ (,(V, na), where j = 1, ...... M.
Sample 0 given II, Σ and μi
For each pair (IT6), ) sample N values of 0!h-5) ~ P(O II6), E where k = I,...,N N.
Compute the Monte Carlo Estimate
Using the samples, the expectation E[O.] is estimated as:
E
Σ 0 (kj)
(9)
M 1=1
1T6), The inner sum N Ik=1 (( approximates the conditional expectation, E(0, IT6), Σ(] The outer sum averages over the prior samples of II and E and fixed pi, estimating the marginal expectation E[O,]. This two-layer Monte Carlo approach approximates the integral by leveraging the independence of II and Σ and sampling 0 conditioned on each sampled pair (II, Σ). The independence assumption permits separate sampling for each 0,, combined into a final sampled weight matrix.
Algorithm MonteCLoRA Estimation of LoRA Parameters
Require: Niai Input feature size
Require: Nout: Output feature size
Require: N: Number of mixture components
Require: V E Ruouut Xouut Wishart Distribution Prior (Trainable Diagonal Matrix)
Require: a E NN; Dirichlet Distribution Prior (Trainable Vector)
Require: f E R"ixXPout Weight matrix of the lincar layer (Trainable Dense Matrix)
Require: 6: Sample Scaler
Σ Wmonr (V,ntn)
Sample from Wishart Distribution
11 T1,72, ......... AN Dir(a)
Sample from Dirichlet Distribution
5() ((2) ,((N) (/ E)
> Sampling from multivariate Gaussian distribution Compute the MonteCLoRA estimato Update the weight matrix with the MonteCLoRA estimate
W WonteeCLoA + c ΣN Tk ((k)
1 W HontooLLOR
Algorithm formalizes the forward method of MonteCLoRA for each LoRA parameter initialized with μ and M I. We scale the sample variance before adding to μe by the sample scaler (c) to have finer control over changes in the model weights. The learnable parameters in the above approach are the LoRA parameters, Wishart distribution scale matrix V and the Dirichlet concentration parameter a. The Wishart distribution scale is a diagonal matrix of order Routs and the Dirichlot concentration is another vector of size N. Hence, for each MonteCLoRA layer, we add nout + N parameters only. When this applied to LoRA
3please that use and refer the LoRA weights but in different contexts.. contexts distributions we use i and in contexts of learnable parameter use
10

---

Published in Transactions on Machine Learning Research (08/2025)
down projection matrix, the number of extra parameters is just + N, and introduce only fraction of c+N parameters O ( x obtained from LoRA, which is approximately ((1) as & min( Bout), and N are usually of same order which in turn is a only small fraction of total parameters for layer. Hence, the total number of parameters over the LoRA parameters very small and ((1) compared to LoRA parameters.
4.3 Why do we use mixture of Gaussian
Using a single Gaussian for all of the low-rank parameters forces the entire posterior to be unimodal and centered around a single mean-covariance estimate. In contrast, drawing multiple samples from the same Gaussian and then forming a convex combination, allows MonteCLoRA to explore different modes in the weight space and avoid collapsing onto a single point estimate.
Moreover, by drawing multiple samples, MonteCLoRA obtains Monte Carlo approximation that reduces variance, providing a robust estimate of the original LoRA parameters, under Bayesian framework. As proved in Lemma 3.3, adding noise to the weights during training makes a neural model less sensitive to different learning rates during training. Therefore, using the Gaussian parameterization, MonteCLoRA provides smoother and more reliable convergence during training the low-rank parameters.
4.4 Theoretical Results
This section presents the theoretical results involving MonteCLoRA, for demonstrating the effectiveness of Bayesian parameterization for robust fine-tuning of LLMs. We show that MonteCLoRA derives an unbiased and robust estimate of the true model parameters and how influences and stabilizes the final output obtained by the fine-tuned model.
Lemma 4.1 Sample scale factor shifts the final output proportionally.
Proof. Consider simplified version of the model where we ignore activations. The neural network can be represented product of multiple weight matrices, 01, O2........................ On. The output for an input X is given by,
Y 0.02...00 O.
Now, we consider the parameterization used by MonteCLoRA. Fach weight matrix 0 includes a scaled random variable added to it. We represent random variables as di, and the scaling factor is e. The output for an input X in this case becomes,
Y Y= e02) cδ²2) .O.O + xQ.)X.
Expanding the terms and neglecting O(e²) terms (this assumption is valid as typically use E ((10-))), we obtain
Y = 0,02.... 1..0...0,.... + (0,00 ... ... + -0.00000-0. O0 .0.
1.....+.0.. +0,02 0 0n-10m) x + +
...02..000
Y +E Σ
00.0000.100000 0m) X.
Taking the norm, we obtain,
| YI = Σ
n
0 0 0-100 0..) |x||
Assuming X > 0
-YI =
(0.00000010600 ... e.)| -
X
This expression shows that for an input X, the output shift is directly proportional the sample scaling factor
11

---

Published in Transactions on Machine Learning Research (08/2025)
Lemma 4.2 The final model output estimated by MonteCLoRA is unbiased estimator of the original model oulpul.
Proof. An important conclusion from the parametcrization shown in the previous section the expectod value of Y. Since each weight matrix is assumed to be independent, we have,
| (0 + = (02 + < ...E [0.. + = x Y (0) + (E( + cE(g))) ... (0,, + cB(
Since each 6, a weighted sum samples from a Gaussian distribution with zero mean, we have B[g.] = Thercfore,
Y 0,02
0., ......00x---- 0,X
Hence, in expectation, the model's prodiction is not dependent on the sample scaler. This property is beneficial during inference. We the model at inference time like the standard LoRA model, utilizing the fine-tuned weights guided by MonteCLoRA enhancements. As result, inference times remain the same because we do not need to perform sampling during inference.
Lemma 4.3 The estimator described in Equation an unbiased estimator for the expected posterior defined in Equation
Proof. Let 0; denole the estimator in Equation g To show that the Monte Carlo estimator is an unbiased estimator of the true expectation, we compute and Σ() the expectation of N
0.] and demonstrate that it equals E[O,]. For fixed II6)
Σ 1= o (ka)) over the samples 0! is,
N
E
(k))
II x = E I1 E
N
By the law of large numbers, N xo, the sample mean N/ D keh converges to the conditional expectation E[O, II6), E6IJ. Next, we consider the expectation of the outer sum over the samples IIG)
and E as,
M
E
3 E
116),
= En. II
M
J=1
Using the law of total expectation, we have
E En E(O,
II, Σ| = -E
Therefore, as M + oc, the outer sum E0 IT ز converges to E,O,), which is the original integral. Hence, in expectation, the Monte Carlo estimator O converges to the true integral [( E[O.). This demonstrates that, given sufficiently large N and M, the Monte Carlo estimate will approximate the desired expectation E[0,].
Lemma 4.4 The estimator described Equation robust estimator for the integral in Equation
Proof. To find the covariance of the Monte Carlo estimator described in Equation we need to analyze both the inner and outer layers of sampling over 0, II, and Σ. The variance of this estimator will depend on the variability from both layers. We first decompose the covariance using the law of total covariance; we can decompose the variance θ; as follows,
Cov(O.) = =Euy Covo. (
N " ((()))) k=l
N II, 1)
oft N
(
II,
+ COVILE
Eo,
N
12

---

Published in Transactions on Machine Learning Research (08/2025)
The first term, EIE Cove, ( Ei= ( II, ] given II and Σ.
represents the expected conditional covariance
The second term, COVILE Eo, 1 E 0(m3) ability in II and Σ.
II, Σ
represents the covariance due to the vari-
For fixed II and Σ, the covariance of the average
(.4-3) is; to Ek=1
N
Cove;
(
II, Σ
Cova, (0; II, E),
N
N
where Cove, (0; II, Σ) the conditional covariance of 0, given II and E. So, the first term becomes,
ET.E
Cov(O,
II,
En En sCove, (0, II, E)}.
N
The conditional covariance can be computed as,
Cov(O = Σ n. E
On taking expectation over II and Σ, we obtain,
N EI
Σ
E
Σ
E N
Σ
En
N
The second term in the covariance decomposition accounts for the variability of the conditional expectation E II, Σ| due to the sampling of II and Σ. We can write this term as,
1 COVIIY (E(0 11, E))
COVIL (E 0 II, EJ
M
The expectation of 0 given II and Σ is just the mean of the distribution, which is independent of II and Σ. Hence the second term goes to zero. Putting it all together, the covariance of the estimator 0 is,
Σ
Cov Cov(O,) = ET
N
Therefore, as N + co, the covariance reduces, making the estimator robust.
4.5
Training Objectives
4.5.1
Reparameterization Losses
We incorporate a series of Kullback-Leibler divergence (KLD) losses into the objective function to regularize the learned prior distributions. These losses are crucial for shaping the model's latent weight space and preventing overfitting. For each sampling process within the model, corresponding KLD loss is added. The sum of all these KLD losses across layers is scaled by KL divergence weight parameter v and incorporated into the final loss
Multivariate Gaussian KL Divergence Loss. In MonteCLoRA, the distribution to be optimized
multivariate Gaussian distribution with parameters (1, E) and it is optimized against a multi-
variate Gaussian with parameters (H, We refer the distribution be optimized as P and the standard normal as Q The KI. divergence loss can be computed as,
KL Derivation of the KT. divergence losses furnished in the Appendix
tr (r((t)
In Σ
nout
(10)
13

---

Published in Transactions on Machine Learning Research (08/2025)
Wishart KL Divergence Loss. We optimise the Wishart Distribution of MonteCLoRA, Wnon (V, Mmn), against the standard Wishart distribution, Wa (I, (m). We keep the degree of freedom and dimensionality the same to focus on optimizing the scale matrix. We refer to the distribution to be optimized as P and the standard distribution as Q. The KL divergence can be calculated in closed form as,
1 (nin (- in [V]) + Biny tr( RmmRout) KLW 2
(11)
Dirichlet KL Divergence Loss. We have N-Dimensional Dirichlet random vectors following the distributions (P and Q) with parameters an and 02, respectively, where 0n, 02 E IRN_ The KL divergence of P
MonteCLoRA, we take ag as a constant vector with all the same values. from Q is given by,
T EE N
an
N r(a)) T(o):
N
N Σ
KLD In
In
Σ
(an azi) ((ani) -
(12)
r 2i=1 022
Here, I is the gamma, and & the digamma (logarithmic derivative of gamma) function.
4.5.2 Cooperative Loss
To ensure maximal participation from each of the N mixture component, we compute cooperative loss. For mixture weights T1, #2....... # obtained from the Dirichlet samples, the cooperative loss is calculated as ΣN कद. To promote higher cooperation, we minimize the cooperative loss,
Le
m
(13)
Lemma 4.5 Cooperative loss defined in Equation
13 is minimized when
N Vi
E [].
Proof. Since the samples from dirichlet distribution sum to one, we have as,
N ni 1, we can rewrite Le
N-1 = Σ
N-1
N-1
--1
म + n2 + + (1
Σ n.) )
π.
Σ
+ 2
Σ
Σ
dLe = 0, we get, Therefore, for 0m.
4m; 2 +2 Σ πīj
jti
N-1
2T, = = 2 Σ
m
2T 2
Hence, T1 = 42 AN N- As
@L 8pm; 4 0, N is the minima.
Lemma 4.6 Mönimizing cooperative loss defined in Equation 13 is equivalent to minimizing entropy of
misture of Gaussian defined in Equation 6
Proof. Using the fact that 0r N Nep, Σ) and entropy of N(p. Σ) c 1 log det(E), for some suitable constant c, we get
14

---

Published in Transactions on Machine Learning Research (08/2025)
N entropy(P(0:)) entropy Th (M, Σ))
N log det det(2)) + log m (14)
1
Σ
log - det(22)
2
Monotonicity of log(2)) suggests that minimizing Σx= म minimizes the entropy of P(O:).
Therefore, minimizing the cooperative loss Le reduces the uncertainty of the mixture of Gaussian learned in Equation lcading to robust estimate of the LoRA paramcters.
4.5.3 Final Loss Function
Combining all the training objectives, the final loss function can be defined as,
== Σ Σ log, P (Bt x. x,y<s) + (KLD + KLw KLN) + Lc-
NNontact.oRA
t=
Herc, NANotaCLLORA denotes the total number of MonteCLoRA layers. The division by NonotacLORA and the KLD loss weight 7, serve to normalize and scale down the contribution of the KL divergence losses, ensuring balanced training dynamics.
5
Experiments
5.1 Datasets and Tasks
To evaluate the effoctiveness of MonteCLoRA, we conduct thorough empirical study across two different ranges of tasks natural language understanding (NLU) and natural language generation (NLG).
For NLU, use five tasks from the GLUE (Wang et al. 2018 and SuperGLUE WWag et al. 2019 benchmarks, namely MRPC (Dolan & Brockett 2005), CoLA (Warstadt et al. 2019). RTE (Dagan et al.
2005 Haim et al. 2006 Giampiccolo el al. 2007 Bentivogli et al. 2009), WiC (Pilehvar & Camacho- Collados| 2018) and BoolQ (Clark et al. 2019 Details of these datasets can be found in Appendix These
gold standard datasets for these tasks contain scparate train and dev (also called validation) split, where we use the train dataset to fine-tune LLMs and dev dataset for evaluation. For these tasks, we consider the intrinsic evaluation metric negative loglikelihood (NLL) and extrinsic evaluation metric accuracy.
On commonsense NLG, consider six commonsenss reasoning tasks PiQA (Bisk al. 2020). Social Sap et al. 2019), WinoGrande (Sakaguchi et al., 2021). ARC-easy, ARC-challenge (Clark et al. 2018 and OpenBookee BookQA (Mihaylov et 2018). Details of these tasks are mentioned in Appendix B For these tasks, we use the Commonsensel 15K dataset (Hu et al. 2023 This dataset particularly curated for instruction fine-tuning of LLMs and comprises of subsets of training samples from different commonsense reasoning tasks, The final output for these NLG tasks contains the answer multiple-choice question and evaluated using accuracy. We conduct further evaluation instruction following tasks mathematical reasoning and code generation. For evaluation dataset. For code generation, we construct a dataset from the publicly available Magicoder- Serecoddr--SS--nnaae Instrust 75K corpus (Wei et al. 2024h. Specifically, we filter for Python examples whose tokenized sequence length is below 512 tokens, and randomly sample 10,000 such data points to form a consistent and lightweight training set. To assess model performance, we evaluate on the official test split GSM8k for math reasoning and Humanbvval benchmark (Chen al. 2021), a widely-used dataset for functional code synthesis performance. Details of these tasks are mentioned in Appendix
mathemalical reasoning lasks, consider the GSM8k (Cobbe el al.) 2021
5.2 Models
For NLU, we use a pre-trained RoBERTA- base (Liu et 2019) (110M parameters) model. For commonsense generative tasks, we the pre-trained 1.A-A-1-7B Touvron al. 2023) model, and for math and coding
15

---

Published in Transactions on Machine Learning Research (08/2025)
Type
Hyperparameter
RoBERTa-
LLaMA-1-7B
LoRA r
8
32
LoRA a
16
Statie
Max Sequence Length
256
256
Learning Rate Scheduler
Linear
Linear
Epochs
20
Batch Size
{8, 32,
64
[8. 16,32)
Tunable
Learning Rate
{3 x 10-4.5x00 10
11 10
(3 x 10 3x 10
Table 3: Static and tunable hyperparameters used in fine-tuning RoBERTa-base and LLaMA-1-7B models.
Hyperparameter
Value
Sample Scaler (c)
5 10-3
2.5
10
10
(I.
KL Loss Weight (1)
1 x 10
Dirichlet Prior (a)
Mixture Components (N)
4
Table 4: Hyperparameters used for MonteCLoRA.
tasks we use Llama--32-3B--nstttee (Grattafiori et al. 2024a All the pre-trained model weights are obtained from Huggingface (Wolf et al. 2020).
5.3 Baselines
Apart from full fine-tuning and LoRA (Hu et al. 2022) fine-tuning strategies, we compare the performance of MonteCLoRA against the competitive parameter-efficiett fine-tuning methods.
AdaLoRA. Zhang al. (2023) improved upon vanilla LoRA by dynamically adjusting the ranks of the rank-decompositinn matrices, facilitating the allocation of more capacity to important weights and reducing it for less significant ones.
DoRA. Liu et al. 2024b improved on LoRA by decomposing the pre-trained weight into two components, magnitude and direction, for fine-tuning. By employing LoRA for directional updates to efficiently minimize the number of trainable parameters, DoRA makes sure enhance the lcarning capacity and training stability of LoRA while avoiding any additional inference overhead.
Laplace LoRA. Yang et al. 2024g brought Bayesian approach to LoRA by applying Laplace approximation to the posterior over LoRA parameters to overcome uncertainty and overconfidence and ensure better model calibration.
LoRA+ Hayou et al. (2024) extended the original LoRA by introducing additional trainable compo- nents such bias and layer norm parameters, along with a gating mechanism to selectively control LoRA updates. This enables improved flexibilityy and performance, especially in tasks requiring maanced adaptation, while maintaining a low parameter footprint.
Prompt Tuning, Lester et al. (2021) proposed a lightweight fine-tuning strategy where a small set of continuous trainable vectors (soft prompts) are prepended the input embeddings of frozen pre- trained model. This method drastically reduces the number of trainable parameters while achieving strong task performance, especially in language understanding tasks.
IA3. Liu et al. 2022a) introduced IA3
Adaptation) as a parameler--fffcient fine-tuning method by inserting learnable scaling vectors into the key, value, and intermediate pro- jection layers transformer blocks. By modulating the flow of information through attention and MLP layers, LA enables effective adaptation with minimal parameter updates.
16

---

Published in Transactions on Machine Learning Research (08/2025)
90
2.5
Negative Log Loglikellhood
80
2.0
Accuracy
70
1.5
1.0
50
L.
0.5
BoolQ CoLA MRPC RTE SST2 MNLI
QQP
QNLI
BoolQ CoLA
SST2 MNLI QQP Task
RTE
QNLI
Task
Full FT
LoRA
AdaLoRA
MonteCLoRA
Figure 4: Distribution of accuracies and negative loglikelihood of RoBERTa-base on different GLUE tasks with different finc-tuning strategics.
5.4 Hyperparameters
As described in Section [ LLM fine-tuning strategies are very sensitive to hyperparameters. For the repro- ducibility of our study, we describe the hyperparamcters used in MonteCLoRA and the bascline on different tasks. Table contains the static and tunable hyperparameters used for all the fine-tuning methods with RoBERTa and LLaMA models. Static hyperparameters used for all the model-specifie training tasks and are the same for all the fine-tuning strategies. We tune the optimizer learning rate and the training batch size for the robustness studies for different strategies. Table g reports the hyperparameters specific to MonteCLoRA. The hyperparameters are the default ones used in our method and are used in all the NLC and NLG tasks. In the subsequent sections, we discuss the importance of these hyperparameters in greater detail. All our experiments were conducted on NVIDIA A100-80GB GPUs that had access to the CUDA 12.5 environment.
6 Experimental Results
In this section, we discuss the results obtained from the empirical study with the NLU and NLG tasks.
6.1 Evaluation on NLU Tasks
Figure highlights the distribution of accuracy (extrinsic metric) and negative loglikelihood (intrinsic metric) caleulated on the validation datasct of the GLUE and SuperGLUE tasks with the RoBERTa-ban model. We compare MonteCLoRA with full fine-tuning (denoted as Full FT), LoRA and AdaLoRA. The distribution spread (calculated as best worse score) is observed to be highest for full fine-tuning, with the average spread being 0.30 and 28.7, respectively, on the intrinsic and extrinsic metrics, significantly higher than all the PEFT methods. Additionally, this spread in accuracy is highly pronounced in the larger GLUE tasks, highlighting the sensitivity of full fine-tuning on hyperparameters particularly for larger and semantically more challenging tasks. Among the low-rank adaptation methods, AdaLoRA has the highest average spread of 0.61 with the intrinsic metric, whereas it is 0.10 for MonteCLORA, which highlights the ability of MonteCLoRA to calibrate output confidence under different hyperparameter configurations. LoRA has NLL spread 0.18, which is less than that of AdaLoRA, but still considerably higher than that of MonteCLoRA. MonteCLoRA also lares much better than the other methods in terms of average accuracy spreads. It is the least for MonteCLoRA, with a value of 5.40 and worst for LoRA with a value of 19.44. On smaller tasks like WiC, LoRA demonstrates points higher spread than MonteCLORA. MonteCLoRA shows higher stability terms of both metrics, justifying its effectiveness in maintaining stability in performance and confidence in its predictions.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Method
Robustness
MRPC
CoLA
RTE
WiC
BoolQ
SST2
QNLI
QQP
MNLI
Average
FT
2.21
1.98
1.45
1.45
1.71
1.91
2.06
2.05
1.32
1.79
LoRA
3.11
2.38
1.86
1.49
1.76
4.81
2.87
3.06
2.11
2.61
Intrinsic 7
AdaLaRA
1.48
1.61
0.97
0.35
0.93
3.79
2.91
3.12
2.14
1.94
MonteCLoRA
3.16
2.43
2.18
1.63
1.75
3.20
2.81
2.27
2.61
Full
89.1
$3.8
52.7
63.0
80.1
72.7
84.0
81.4
60.3
74.1
LoRA
88.6
83.8
80.1
60.6
80.0
92.6
85.3
85.5
81.7
82.0
Extrinsic
AdaLoRA
87.7
82.6
78.5
65.7
80.2
90.0
86.0
85.8
82.1
82.1
MonteCLoRA
89.3
84.3
81.2
69.2
79.9
91.0
89.6
85.0
83.3
83.6
Table 5: Robustness of different fine-tuning strategies with RoBERIa-base on GLUE and SuperGLUE tasks. We underline the tasks where MonteCLoRA achieves higher robustness score than LoRA fine-tuning.
Method
Metric
MRPC
CoLA
RTE
WiC
BoolQ
SST2
QNLI
QQP
MNLI
Average
Full FT
89.2
84.0
76.5
70.4
S0.4
94.6
92.2
88.4
85.6
84.6
LoRA
89.9
85.1
80.1
67.5
80.0
98.1
89.1
87.0
83.0
83.9
Accuracy f
AdaLoRA
88.7
84.5
81.2
67.7
80.6
92.4
89.7
86.6
84.2
84.0
MonteCLoRA
91.2
84.9
81.2
71.3
79.9
92.4
89.8
85.7
83.5
84.4
Full FT
0.34
0.40
0.60
0.68
0.51
0.26
0.28
0.39
0.42
LoRA
NLL +
0.32
0.40
0.54
0.62
0.49
0.20
0.27
0.31
0.44
0.40
AdaLoRA
0.42
0.54
0.62
0.83
0.55
0.21
0.28
0.30
0.42
0.46
MonteCLoRA
0.31
0.46
0.60
0.50
0.21
0.26
0.32
0.44
0.39
Table 6: Comparison of different fine-tuning strategies with RoBERTa- on GLUE and SuperGLUE lasks. We report the highest accuracy and lowest negative loglikelihood (NLL) obtained across different hyperparameter configurations. The best strategy is highlighted in bold and the second best strategy is underlined for each task.
Table 5 shows the intrinsic and extrinsic robustness defined in Section 3.3 for different fine-tuning methods. The above discussion confirms that MonteCLoRA lends to have lesser chances of having abysmally bad intrinsic or extrinsic scores (outliers), unlike FFT or LoRA. The robustness metrics highlight the method's stability in terms of maintaining the probability of having good performance. We observe the highest average intrinsic robustness value of 2.61 both with MonteCLoRA and LoRA. AdaLoRA demonstrates the lowest intrinsic robustness among the low-rank methods, 26% lower than MonteCLORA. MonteCLoRA also exhibits the highest extrinsic robustness of 83.6, which is 1.6 points higher than LoRA, and 1.5 points higher than AdaLoRA. We perform Wilcoxon signed-rank lest to statistically validate whether MonteCLoRA is more robust than the other baselines. The rank tests conclude that MonteCLoRA is more robust than LoRA and its variants, in terms of extrinsic (p-value 0.08) robustness motrics. Full fine-tuning displays lowest robustness valuos, which is more pronounced for the bigger tasks. has the lowest extrinsic robustness of 74.1, and intrinsic robustness of 1.79 indicating inability to generalize well across different. NLU tasks. The underperformance of full fine-tuning also suggests that training the full language model on all downstream tasks is inefficient and could be ineffective for certain tasks. Particularly poor performance on validation. Remarkably, MonteCLoRA achieves higher intrinsic and extrinsic robustness than LoRA and its variant in six out of nine tasks, emphasizing its ability to showcase robustness across different tasks and training configurations.
tasks like RTE, full fine-tuning
diverge and exhibit very
Table reports the highest accuracy and the lowest negative loglikelihood (correspondingly, highest log- likelihood) achieved by different fino-tuning methods on NLU tasks. Interestingly, the variance among the baselines is significantly lower when the best results are concerned. On average, the best accuracy obtained with full fine-tuning is 84.6%, which is higher than LoRA and AdaLoRA. MonteCLoRA comes close second with 84.4% accuracy, which the best among the low-rank methods, demonstrating its superiority in achieving better performance across different hyperparameter configurations. It also achieves the lowest NLL among all the baselines, indicating its superiority in generalization across multiple tasks.
We compare MonteCLoRA against the contemporary Bayesian post-hoc methods on the NLU tasks Table 7 The Bayesian methods such as maximum a-posteriori (MAP), Monte Carlo dropout (MC Dropout), and Laplace-LoRA (L.A offer flexible solutions to applying Bayesian treatment. to LoRA fine-tuning in a post-
18

---

Published in Transactions on Machine Learning Research (08/2025)
Metric
Method
MRPC
CoLA
RTE
WIC
BoolQ
Average
MonteCLoRA (best)
91.2
84.9
81.2
71.3
79.9
81.7
MonteCLoRA (median)
89.3
84.3
81.2
69.2
79.9
80.8
MAP
86.1
81.8
70.9
63.9
77.2
76.0
Accuracy 1
MC Dropout
(Gal Ghahramami 2016
87.1
82.6
72.4
68.8
76.6
77.5
Checkpoint Eusemble CChen el al. 2017
86.3
$1.4
71.8
77.2
76.3
Temp (Guo
81.8
72.6
65.4
77.3
L.J.A Nang nl.
T2.6
77.4
LA Yang al. 2021
86.4
72.6
77.4
76.7
MonteCLoRk [best.]
0.31
0.41
0.46
0.60
0.50
0.46
MonteCLoRA (median)
0.32
0.11
0.46
0.61
0.57
0.47
MAP
0.66
0.50
0.76
1.00
0.54
0.69
MC Dropout Gal & Ghahuamani 2016
0.39
0.39
0.58
0.72
0.50
0.52
NLL
Checkpoint Ensemble Chen et al. 2017
0.44
0.49
0.57
0.63
0.53
0.53
Temp (Gno al. 2017
0.32
0.40
0.54
0.62
0.49
0.47
LLLA Yang et Rl 2021
0.44
0.56
0.51
Yang et al. 2024
0.54
0.48
Table 7: Comparison of different Bayesian post-hoc methods applied to pre-trained RoBERTa-base model and LoRA fine-tuning on various GLUE and SuperGLUE tasks. Results of baselines apart from MonteCLoRA are obtained from Yang et al. 2024). The bold results indicate the cases where MonteCLoRA performs better than the other Bayesian baselines.
80
75-
Accuracy
70-
0.79
Validation NLL
82
0.77
60-
0.76
55
0.75
PIQA
Social
WinoGrande
ARCve
ARC-c
OpenbookQA
Task
Adapter
LoRA
DoRA
MonteCLoRA
Adapter
LaRA
DoRA
MonteCLoRA
Model
(a) Accuracy
(b) Negative loglikelihood
Figure 5: Distribution of test accuracies and validation negative loglikelihood of LLaMA-1-7B on different commonsense reasoning lasks with different fine-tuning strategies.
Method
PIQA (f)
(tc
WinoGrande(1)
ARC-e(f)
ARC-c(f)
OpenbookQA(f)
Average
Adapter
73.0
68.3
61.0
72.4
57.1
67.4
67.0
LoRA
77.7
71.5
67.7
79.2
62.8
76.2
72.3
DoRA
72.6
69.7
64.9
78.0
58.8
73.4
69.6
MonteCLoRA
78.0
71.0
65.8
79.9
62.9
76.0
72.3
Table 8: Performance of different fine-tuning strategics with pre-trained LLaMA-1-7B on generative tasks.
hoc manner (i.e., can be used after the LoRA method is trained). Our comparative analysis suggests that MonteCLoRA can perform better than the existing Bayesian post-hoc methods with fewer training steps. As argued by Yang et al. (2024), these methods require longer training steps for calibrating output probabilities generated in the fine-tuning phase. Even when consider the median performance of MonteCLoRA, it still achieves 3.2% higher accuracy than the best baseline, Monte Carlo Dropout. In terms of the negative loglikelihood, MonteCLoRA achieves the best performance among all these Bayesian post-hoc methodologies.
6.2 Evaluation on Commonsense NLG Tasks
We follow a similar empirical study for the generative tasks as in the NLU tasks. Figure highlights the distribution of test. accuracy and the validation NLL scores. Contrarily to the NLU tasks, the generative
19

---

Published in Transactions on Machine Learning Research (08/2025)
Method (T)
(Tcial(T)
WinoGrande(()
ARC-e(1)
ARC-c(TT OpenbookQA(1) Average(1)
Adapter
72.6
66.8
63.4
71.3
56.3
66.4
66.1
LoRA
75.9
70.5
62.5
74.2
DoRA
71.9
66.7
57.7
71.4
67.3
MonteCLoRA
77.0
69.6
65.6
78.2
61.7
74.3
71.1
Table 9: Extrinsic robustness of different fine-tuning strategies with LLaMA-1-7B on zero-shot generative tasks.
0.7
0.6
Accuracy
Accuracy
0.6
0.5
0.5
Strict match
Flexible extract
Pass@2
Pass@4
Metric for GSMBk Evaluation
Metric for HumanEval
LoRA
LoRA+
Prompt Tuning
LoRA
LoRA+
Prompt Tuning MonteCLoRA
DoRA
IA3
MonteCLoRA
DoRA
IA3
GSM8k Accuracy Distribution
(b) HumanEval Accuracy Distribution
Figure 6: Distribution of accuracies obtained by LLaMA-3.2-3B in GSM8k and HumanEval tasks for different
PEFT strategies.
experiments are more intriguing as the extrinsic metrics are calculated on a test dataset in a zero-shot manner, ie., the model might not be aware of the test data distribution during the fine-tuning phase. terms of the intrinsic metric (NLL), DoRA has the tiniest spread of 0.004 points, whereas both LoRA and MonteCLoRA have modest spread of 0.03. However, with extrinsie metric, the spread with MonteCLoRA is only 2.2, significantly lower than LoRA (4.6) and DoRA (4.0). Remarkably, tasks like WinoGrande, the spread with DoRA and LoRAA can be as large 9.5; however, with MonteCLoRA, the spread remains meagre for all the tasks, ensuring higher stability.
In Table we report best accuracies achieved by the different fine-tuning methods on generative tasks. We observe that MonteCLoRA and LoRA exhibit a similar performance on the zero-shot generative tasks. The average accuracics obtained with MonteCLoRA and LoRA remains 72.3, higher than Adapter and DoRA. Out of six tasks, on three lasks, MonteCLoRA achieves better accuracy than LoRA. Out of the remaining three tasks, only in WinoGrande, the margin between LoRA and MonteCLoRA is significant. LoRA and MonteCLoRA are marginally similar for the remaining two tasks. In terms of the robustness metrics (c.f. Table 9 LoRA performs slightly better than MonteCLoRA. Particularly on ARC-challenge and WinoGrande, LoRA achieves ~ point better robustness than MonteCLoRA. However, is worth noting that the long- tail nature the accuracy distribution of LoRA makes these results less reliable and less robust. On the other hand, MonteCLoRA balances good performance with excellent stability, overmining its effectiveness robust fine-tuning.
6.3 Evaluation of NLG Tasks with LLaMA-332-BBIInttrul on GSM8k and HumanEval
We further comprehensively evaluate various PEFT methods on GSM8k and HumanEval datasets using LLaMA-3.2-3B-Instt as the basc model. These tasks are strong benchmarks for mathematical reasoning and code synthesis, respectively. Both the evaluations were done using LM-evahhation-havee harness (Gao et al. 2024).
20

---

Published in Transactions on Machine Learning Research (08/2025)
PEFT
SM
(max)t
FE (max)1
SM (ext. robust.)1
FE (ext. robust.)
LoRA
0.70
0.71
0.65
0.65
DORA
0.75
0.75
0.67
0.67
LoRA+
0.76
0.76
0.71
0.71
IA
0.76
0.77
0.74
Prompt Tuning
0.71
0.71
0.68
0.68
MonteCLoRA
0.78
0.78
0.75
0.76
Table 10: Comparison of PEFT methods on GSM8k using SM (strict match) and FE (flexible match) metries with robustness evaluation.
PEFT p (max)1
pass@4 (max)1
pass@2 (ext. robust.)
pass@4 (ext. robust.)T
LoRA
0.52
0.58
0.50
0.55
DoRA
0.55
0.60
0.51
0.54
LoRA+
0.56
0,60
0.53
0.58
IA3
0.56
0.63
0.56
0.62
Prompt Tuning
0.45
0.54
0.45
0.54
MonteCLoRA
0.58
0.65
0.55
0..1
Table 11: Comparison PEFT methods on HumanEval using pass@2 and pass@4 metrics with robustness evaluation.
On GSM8k, we adopt the exact match (EM) metric to evaluate the strict correctness of the model's generated answers. Due to the presence of varied generation formats, we apply two filtering strategies: strict-match, which enforces precise output formatting, and flezible-eztract, which tolerates verbose reasoning and isolates the final answer via regex-based parsing. This dual evaluation scheme enables muanced understanding of the model's reasoning accuracy and its robustness to formatting constraints. Table TO shows that MonteCLoRA consistently outperforms the baselines across all metrics. It achieves the highest strict match score (0.78) and flexible extract score (0.78) while attaining top robustness scores, demonstrating its superior generalizatiog clean and noisy test environments. IA and LoRA+ also show strong performance, though with slightly lower robustness margins. Prompt Tuning trails behind the adaptation-based methods in both maximum and robust scores. The spread analysis (Figure Ga) indicates that MonteCLoRA has significantly tighter distribution than LoRA and DoRA, whereas it is behind IA3 and prompl. On strict match metric, LoRA shows a performance spread of up to 2.6 points, 2.4 for DoRA, and MonteCLoRA's spread remains within 1.26 points, which 51% less compared to LoRA and 47.5% less compared to DoRA, underscoring its robustness and reliability.
On the HumanEval benchmark (Table which measures functional correctness in code generation using pass@k metrics, MonteCLoRA again emerges as the most performant method. With maximum pass@2 and pass @4 scores of 0.58 and 0.65, respectively, it outperforms all other PEFT strategies. Its robustness (median) pass@2 Q2 and pass@4 scores remain high (0.55 and 0.61) and beat all LoRA family baselines, indicating strong consistency in generating functionally correct code across different learning rates. LA3 and LoRA+ perform competitively, and IA beats MonteCLoRA by a small margin in robustness. Prompt Tuning, in contrast, underperforms significantly, suggesting limited capacity for structured generative tasks like code synthesis. As seen in Figure Gb, the robustness spread in HumanEval similarly minimized with MonteCLoRA. While LoRA and DoRA suffer from spread of 3.8 and 1.1 points, MonteCLoRA maintains a tight band of 0.4 around its peak, reducing spread by 89% for LoRA and 63% for DoRA, respectively, further reinforcing its stability.
In conclusion, while traditional methods such as LoRA, LoRA+, and IA3 provide competitive baselines, MonteCLoRA balances top-tier accuracy and robustness, solidifying its effectiveness for fine-tuning complex NLG tasks involving symbolic reasoning and code generation. Its consistently narrow spread in both max and robust metrics makes it promising candidate for real-world applications requiring stability under distributional shifts.
21

---

Published in Transactions on Machine Learning Research (08/2025)
(a)
(b)
(c)
(d)
(e)
Accuracy
60
2
1.6
Fixed
Random
0.001 0.005 0.01
-6.0
-5.0
-4.0
With
Without
0.6
0.4
0.0
Fixed
Random
0.001 0.005 0.01
-6.0 -5.0
Without Cooperative Loss
With
Mixture Components
'iiiiclett Concentration
Sample Scaler
KLD Weight (log10 scale)
MRPC
CoLA
RTE
WIC
BoolQ
Figure 7: Ablation of MonteCLoRA with RoBERTa-base different GLUE tasks with respect to (a) the number of mixture components N, (b) Dirichlet concentration parameter a, (c) Sample scaler e, (d) KLD weight (7), and (e) cooperative loss.
6.4 Ablation Study
We study the importance of different components MonteCLoRA with thorough ablation analysis. We use the NLU tasks with the RoBERTa-bass coupled with MonteCLoRA fine-tuning for this study. Figure 7 highlights the performance of MonteCLoRA by changing the hyperparameters, including the number of mixture components (N), Dirichlet concentration parameter (a), sample scaler (e), KLD loss weight (π), and the importance of cooperative loss defined in Equation
Effect of the Number of Mixture Components. Lemma 4.4 suggests that MonteCLoRA estimates a robust estimator with a large number mixture components. However, Figure 7a),, we observe that MonteCLoRA can achieve robust performance even with only four mixture components. Increasing the mixtures to 16 has very minimal impact on the accuracy and the loglikelihood of the fine-tuned model. Having too less number of mixture components could lead to underfitting, hurting the performance of the fine-tuned model.
Importance of Dirichlet Concentration Initialization. To understand the impact of initialization of the Dirichlet concentration, we perform an ablation where instead initializing the value with a = 1, we use a random voctor 4(001)* where is a uniform distribution. We call this ablation 'random' Dirichlet concentration. The results are particularly surprising (c.f. Figure b()), where we observe an average 2% improvement accuracy with random initialization, Random prior can improve the accuracy on tasks like RTE by a margin of 4%. Random initialization of Dirichlet concentration parameters allows the model for more exploration, learning the mixture dynamics beller.
Sample Scaler and KLD Loss Weight for Balancing Exploration-Exploitttie In Algorithm 1 we described the importance of the sample scaler e to dynamically control the exploration of different optimization basins the low-rank parameters. The ablation results Figure highlights an interesting 0.005, MonteCLoRA achieves the best validation accuracy. The results emphasize
pattern: with a moderate the importance of balancing exploration and exploitation for stable convergence of the fine-tuned model. In RTE, WIC, and BoolQ, high can diverge the model, whereas low lead to overfitting the training
22

---

Published in Transactions on Machine Learning Research (08/2025)
Logllkel Peolooood
Laglikel hood
80
Accuracy
Accuracy
Negative
Negalive
All
QKV
All
QKV
Dense
Sparse
Dense
Sparse
Mod Applied
Modules Applied
Type of Mixture
Type Mixture
MRPC
RTE
WIC
BoolQ
CoLA
RTE
WIC
BoolQ
Figure 8: Performance of MonteCLoRA at different modules of RoBERTa-base.
Figure 9: importance of mixture of samples in MonteCLoRA.
data, both impacting the validation performance. A similar observation is found with the KLD loss weight (n) in Figure d(d.. A high KLD loss weight indicates more regularization lowards the prior distributions, whereas low KLD loss weight indicates assigning more importance to the likelihood term, relaxing the prior conditions. Having a moderate 7 = 10-5 leads to better and more stable performance. fact, 1J = 10 or 1) == 10 leads to almost same performance drop of 1% 4%, for different tasks.
Cooperative Loss for Better Allocation Mixture Importance. Another critical component of MonteCLoRA training the cooperative loss. Without cooperative loss, the mixture importance values remain unconstrained and independent. A sense of cooperation between different mixture components ensures that the dynamics of the complex low-rank paramcterization is captured. Morcover, with cooperative loss, the imporlance of different mixture components can be allocated proportionately, maintaining the system's 's entropy. In Figure ([), we see drastic > 3% performance drop without the cooperative loss component, indicating its importance achicving good gencralization.
Flexibility of MonteCLoRA on Different LLM Components. Figure shows the difference in per- formance on GLUE tasks when MonteCLoRA is applied on all the parameters (including attention query, attention key, attention value, attention output, intermediate output and layer output modules) versus when applied only on attention parameters (query, key and value parameters). The analysis strikes an interesting pattern where we improvement in performance for larger tasks and marginal performance drop for smaller tasks when MonteCLoRA is applied to all parameters. In larger tasks like BoolQ, applying MonteCLoRA on all parameters improves the negative loglikelihood and accuracy by 0.5 and 0.3 points, respectively. However, on smaller tasks, enabling MonteCLoRA to only attention parameters improves accuracy by a margin of 2% on average. This phenomenon can be justified using the fact that Transformer MLP (which includes the intermediate and layer output modules) blocks encourage sparse activation Li et al. 2022 ; therefore, having dense mixtures can adversely affect the fine-tuned model. Morcover, sparse connections are more impor- (ant smaller tasks, where overfilting could be key issue. Therefore, the dense mixture introduced by MonteCLoRA is not particularly effective with ML. blocks, specifically when applied to smaller downstream tasks.
Introducing Sparsity in MonteCLoRA. The previous analysis highlights that sparser mixture of Gaus- sian components could be deemed important while fine-tuning LLMs on smaller tasks. To encourage sparsity in the mixture components, we perform an ablation experiment where we choose only one component. For- mally, given a mixture weights {TT, M2, ) we only activate the component k, where = arg max, max, Therefore, the updated mixture weight values become,
f1 प. = otherwise
ifi=k
Figure 9 highlights the results with dense (the default mixture of components) and sparse components on NLU tasks. We observe a significant accuracy improvement on two smaller tasks, MRPC and WiC (1.2% and 5%, respectively) with the sparse mixture. The results illustrate the flexibility of adapting MonteCLoRA
23

---

Published in Transactions on Machine Learning Research (08/2025)
different complexities of tasks, which most existing low-rank parameterization-basse fine-tuning methods fail to exhibit.
Post-hoc Abilities of MonteCLoRA. In Section 2 we described the post-hoc Bayesian methods for robust and calibrated reparameterization of fine- tuned LLMs. Therefore, to assess the effectiveness of MonteCLoRA in post-hoc execution, we perform experiments where we first fine-tune the LLM with only the μ parameter (defined in Algorithm ! for epochs and fine-tune the Wstochastic parameter in the remaining 10 epochs, keeping l frozen. We refer to this experiment 'post-hoe' MonteCLoRA. Decoupling the training of μ and W attchastic has sev- eral computational advantages. Post-hoc execution is particularly useful when the LLM is already fine- tuned and requires incremental calibration. On the downside, post-hoc execution might require more careful consideration, as if the fine-tuned model is struck in local optima, may not be able to come out even after the post-hoc execution. Our observation with post-hoc MonteCLoRA in Figure 10 is rather more intriguing, where we observe performance improvement smaller tasks WiC (5%) and MRPC (0.7%) and almost similar performance CoLA and RTE. On BoolQ, RoBERTa trained with MonteCLoRA post-hoc diverges due to instability during previous LoRA fine-tuning. However, it is important to note that existing Bayesian post-hoc operations require significant effort to mitigate the calibration and robustness challenges of low-rank fine-tuning. On the other hand, MonteCLoRA offers faster convergence (more discussion on this topic in the next section) with more robust performance, irrespective of whether it is applied ad-hoc post-hoc.
0.8
Negative Log
0.6
Accuracy
60-
0.4
40-
20-
0.24
0.0
Ad-hoc
Post-hac
Ad-hoc
Post-hoc
MRPC
CoLA
RTE
WiC
BoolQ
Figure 10: Effectiveness of MonteCLoRA in post-hoc operation.
7 Discussions
7.1
Convergence Analysis of MonteCLoRA
The previous section describes the sensitivity with existing low-rank adaptation based fine-tuning meth- ods. We also highlighted that given inappropriate hyperparameter selection, the existing techniques can- not demonstrate generalization capabilities post-fine-tuning process, defying the purpose of acquiring new knowledge during fine-tuning. In this section, we shed light on the training instability with these existing fine-tuning methods with two selected use cases.
Figure 11 a) illustrates a scenario where LoRA fine-tuned RoBERTa-base model on the BoolQ task di- verges. The training loss remains the same over the entire training period. Under the same hyperparameter configuration, MonteCLoRA achieves convergence with a steady drop in training loss. The training loss curve highlights that due the postcrior estimation, MonteCLoRA does not converge any saddle point but rather robustly learns the global optima. A similar observation made in Figure 11),,, where observe abrasive optimization with LoRA fine-tuning strategy. RoBERTa-base with LoRA converges only after 6000 training steps. On the other hand, MonteCLoRA converges significantly faster in just 3000 steps with smoother optimization trajectory. With appropriate prior regularization, MonteCLoRA diminishes the sensitivity of the optimization process on hyperparameters like learning rates. Typically, with a high learning rate, a model can get struck at a local minima or even diverge, which prominently seen with LoRA. However, this behavior is less frequent with MonteCLoRA.
7.2 Computational Complexity of MonteCLoRA
In Section we discussed the asymptotic complexity of MonteCLORA and highlightod that it introduces only (rr) additional parameters. While the vanilla implementation incurs runtime overhead due to online sampling of stochastic low-rank matrices during the forward pass, this overhead can be significantly reduced
24

---

Published in Transactions on Machine Learning Research (08/2025)
(a)
(b)
LoRA
0.6
MonteCLoRA
Training Loss
LoRA
0.4
MonteCLoRA
0.2
0.0
0
4000
8000
12000
0
2000
4000
6000
Training Step
Training Step
Figure 11: (a) Training loss curves on BoolQ with LoRA and MonteCLoRA finc-tuning. The loss curve with LoRA highlights its the instability during fine-tuning, where the model eventually diverges. (b) Training loss curves on WiC, where MonteCLoRA achieves twice faster convergence than LoRA. For both the cases, we use RoBERTa-base with learning rate of 3 10-4 and a batch size of 8 for both the fine-tuning strategies.
Model
Modules
Runtime (x LoRA) Memory (x LoRA)
QKV
1.63x
1.10x
RoHaaTa-base
All
1.75x
1.15x
QKV
1.45x
1.07x
LLMMA-322-3- Instruct
1.78x
1.25x
QKV
1.23x
1,0xx
LLaMA-3.1-8D-Istttee
All
1.53x
1.23x
QKV
1.27x
1.09x
HaMA-2-13B
All
1.47x
1.13x
Table 12: Empirical runtime and memory overhead of MonteCLoRA compared to LoRA with buffered pre- sampling.
with appropriate buffered sampling. Specifically, we decouple the sampling from the forward pass by pre- generating samples and caching them in a fixed-size buffer. During training, samples are retrieved from this buffer via constant-time (O(1)) lookups, effectively removing sampling latency from the critical path. We replenish the buffer once we have used all the samples. Since the sampling can be parallelized on GPU, the wall-clock time for replenishing the buffer is minimal. To evaluate this improved implementation, benchmarkod MonteCLoRA on the HumanEval dataset using LLaMA-3.2-3B-Inste. Instruct, and LLaMA-2-132 and the CoLA dataset using RoBERTa-base For LLaMA models, we trained for 1000 training steps with an effective batch size of 8, maximum sequence length of 512 tokens, and a LoRA rank of 32. For RoBERTa-base we used a sequence length of 128 with a LoRA rank of 8. We experimented on combination query, key, and value matrices for both models. For AaBERTa-base we also experimented with the dense matrix, and for I.LaMA models, we experimented with output projection. For both models, we used buffer size 15. All our experiments were carried out on single NVIDIA 80GB A100 GPU. The benchmarking results are presented in Table 12
LLaMA-3.1-90
We observe that the runtime overhead of MonteCLoRA decreases significantly for larger models. For LLaMA- 3.1-SB-Instruct, the relative training time is within 1.23-1.30x of LoRA, while memory usage remains under 1.10x. This improvement stems from amortizing the cost of sampling over multiple forward passes. Moreover, in larger models, the forward pass dominates the overall training cost, making the sampling cost proportionally smaller. Overall, these results demonstrate that MonteCLoRA achieves favorable trade-off between expressivity and efficiency, making it practical even for largo-scale fine-tuning settings with long sequences and large models.
25

---

Published in Transactions on Machine Learning Research (08/2025)
7.3 Limitations
While MonteCLoRA offers signilicant improvements in stability and robust over prior low-rank fine-tuning methods, several limitations remain:
Sampling complexity: The Monte Carlo estimation introduces additional training-time overhead, Although have shown that this effect is smaller in larger models, it still increases training times.
GPU-dependent sampling latency: Sampling from Wishart and multivariate Gaussian distribu- tions can introduce variability in training time depending on the GPU hardware, memory bandwidth, and compute capabilities.
Additional hyperparameters: MonteCLoRA introduces new hyperparameters such as the sample scaler and KT, loss weight 7 which may require tuning for each downstream task and model architecture.
Scale limitations: MonteCLoRA has currently been evaluated on models of sizes up to 7B. Its scalability and effectiveness for larger models such as LLaMA-70B or Qwen-14B remain untested.
Synchronous sampling: The sampling operations occur at regular intervals during the forward pass, introducing latency that could be mitigated with parallel or asynchronous methods.
7.4 Future Work
We identify several promising directions to extend this work:
Asynchronous sampling: Designing an asynchronous sampler that decouples the sampling of Monte Carlo weights from the forward pass could significantly reduce latency and improve through- put.
Scaling to larger models: Future work could explore fine-tuning of larger LLMs, such as LLaMA- 70B to test MonteCLoRA's robustness at scale and in longer-context settings.
Adaptive sample scaler scheduling: Rather than fixing the sample scaler, an adaptive scheduling strategy based on layer-wiss wiser-wiee uncertainty, entropy or the variation in cross entropy loss could improve learning and generalization.
Variational inference extensions: Replacing the Dirichlet-Wishart sampling with variational approximations may yield tighter bounds and faster convergence.
Inference-time posterior averaging: Exploring lightweight, ensemble techniques using posterior samples could improve prediction calibration and performance in duitr--disttibuttinn settings during test time.
Cross-domain extensions: Adapting MonteCLoRA vision transformers, speech models, or multi- modal architectures could extend its applicability beyond language.
8 Conclusion
This paper introduced Bayesian reparameterization of low-rank adaptation for efficiently fine-tuning LLMs. We highlighted the sensitivity of existing fine-tuning strategies and proposed MonteCLoRA that provides robust and unbiased estimate of parameter-officient fine-tuning. MonteCLoRA parameterizes low-rank fine- tuned parameters as a mixture of Gaussian with appropriate prior paramelerization. With robust Monte Carlo estimates, MonteCLoRA reduces the sensitivity of the parameterized LLMs over the hyperparameters. Our thorough empirical study overmined the superiority of MonteCLoRA over the contemporary low-rank adaptation method in terms of performance and stability. Although the current work acknowledges the effectiveness of MonteCLoRA low-rank parameterization regimes, the applicability of our method extends
26

---

Published in Transactions on Machine Learning Research (08/2025)
far beyond this. MonteCLoRA can be equally effective with the pre-trained LLM parameters. However, keeping the rising computation cost of fine-tuning M.L.s, it is computationally more sensible to utilize MonteCLoRA with low-rank parameters rather than high-rank dense parameters. MonteCLoRA can also offer great flexibility where it can be used to ensure robustness even during pro-training of LLMs.
Acknowledgment
Т. Chakraborty acknowledges the support of the J.P. Morgan AI Faculty Research Award and Rajiy Khemani Young Faculty Chair Professorship in Artificial Intelligence. This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorgan Chasc & Co and its affiliates (*J.P. Morgan") and is not product of the Research Department of J.P. Morgan. J.P. Morgan makes representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
References
Ebtesam Almazrouci, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Méronane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. XrXiv preprint Xrx:2:31 1. 1.86677, 2023.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7:8, 2009.
Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint ar Xrx::22455.099 09673, 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfong Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL
11641
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432 7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan WU, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1-45, 2024.
Hugh Chen, Scott Lundberg, and Su-In Lee. Checkpoint ensembles: Ensemble methods from single training process. orXiv preprint arXiv: 770.00389, 2017.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantamu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie
Achiam,
27

---

Published in Transactions on Machine Learning Research (08/2025)
Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
Christopher Clark, Kenton Lee, Ming-We Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. aiv preprint orXiv:1905.1004/ 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint wrXiv:1803.05557 2018.
Karl Cobbe, Vineet, Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Malthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:2110.. 14168, 2021.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177 190. Springer, 2005.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34:20089-200 20103, 2021.
Zhijie Deng, Feng Zhou, and Jun Zhu. Accelerated linearized laplace approximation for bayesian deep learning. Advances in Neural Information Processing Systems, 35:2695-2777 2708 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprint arxi::1810.04880 2018.
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4133-41445 4445, 2023.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third International Workshop Paraphrasing (IWP2005), 2005.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050 1059. PMLR, 2016.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Lau- rence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Mucnnighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https://zenodo. pngg/records/1200600
Saced Ghadimi and Guanghui Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming, 2013. URL httpss///arxii org/ http://rwww...oraaa//////./../ http://b//11155555// 5549
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrosing, pp. 1 1-9, 2007.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archic Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruy Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes,
28

---

Published in Transactions on Machine Learning Research (08/2025)
Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iiyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evti- nov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfong Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Tyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathicu Rita, Maya Pavlova, Mclanic Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Peng- wei Li, Petar Vasie, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Sil- veira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit, Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hos- seini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujiwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ra- manathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitncy Meers, Xavier Martinct, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annic Dong, Annie Franco, Amuj Goyal, Aparajita Saraf, Arka- bandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assal Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Lconhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Ed- ward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Estoban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Flo- rez, Gabriella. Schwarz Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Tbrahim Damlaj, Igor Molybog, Igor Tufanov, This Leontiadis, Irina-Flena Veliche, Itai Gat, Jake Weiss- man, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- Baptiste Gaya, Jeff' Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manay Avalani, Manish Bhatt, Martynas Mankus,
29

---

Published in Transactions on Machine Learning Research (08/2025)
Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Moham- mad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptey, Ning Dong, Nor- man Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parckh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zryag- ina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datla, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswany, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sueha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Slephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xisojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosubrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024a. URL https://arxiv: wwaeriivi..//aiiee /a////bb//24000/ 21783
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.21788 2024b.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Sapek- tor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, pp. 783-794, 2006.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024.
URL atttp:://arxxi.orr//aa..s 2
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiiient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arxiv preprint arXiv:1801.0614 2018
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL httpps://pperrviice www.:e/weeeee.orrrrrreeeeeeeeeee
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Wei Lee. LLM-adapters: An adapter family for parameter-efficien fine-tuning of large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization, 2019. URL https://arrx: sttps://aaddv.rr///////BB org//bs//803..
05407
30

---

Published in Transactions on Machine Learning Research (08/2025)
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXie:2310.06822 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXie preprint arXiv:2001.0883 2020.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arx::33112.611 2013.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficiet prompt tuning, 2021. URL http://///axxio.g//abb////////// 2000022220099911
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. arXiv preprint Xrx:222200. 01:::220000613,, 2022.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepscok-v3 technical report. arXiv preprint ar arXiv:24 12. 19437, 2024a.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameteo-ccffcieet fine-tuning is better and cheaper than in-context learning, 2022a.
URL https://arxiv.co hwtp:///ggg/bb/////aaaa/e
Haokun Liu, Derck Tam, Mohammed Muqcoth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameeerr-ffleient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-19995 2022b.
Shih-Yang Liu, Chien-Ya Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint orXbv:2402.0933 09.05333, 2024b.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Associalion for Compu- tational Linguistics (Volume 2: Short Papers), pp. 61-68, Dublin, Ireland, May 2022c. Association for .wc.loog.....ggg// or2/2022.
Computational Linguistics. doi: 10. /120:///11122222clsoott.t. URL
acl acl-short.8
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907. 11692, 2019.
Todor Mihaylov, Peter Clark, Tushar Khot,, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint orXii:18099 02999 2018.
Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, José Miguel Hernández-Lobato, Aliak- sandr Hubin, Alexander Immer, Theofanis Karaletsos, Mohammad Emtiyaz Khan, Agustinus Kristiadi, Yingzhen Li, Stephan Mandt, Christopher Nemeth, Michael A Osborne, Tim G. J. Rudner, David Rügamer, Yee Whye Tch, Max Welling, Andrew Gordon Wilson, and Ruqi Zhang. Position: Bayesian deep learning is needed in the age of large-scale AI. In Ruslan Salakhuidinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st Inter- national Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 99556-39586. PMLR, 21-27 Jul 2024a. URL html
mlr, ppwps//235/pppaaaa.seee
31

---

Published in Transactions on Machine Learning Research (08/2025)
Theodore Papamarkou, Maria Skoularidon, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent. Fortuin, Philipp Hennig, José Miguel Hernández-Tobato Aliak- sandr Hubin, Alexander Immer, Theofanis Karaletsos, Mohammad Emtiyaz Khan, Agustinus Kristiadi, Yingzhen Li, Stephan Mandt, Christopher Nemeth, Michael A. Osborne, Tim G. J. Rudner, David Rügamer, Yee Whye Teh, Max Welling, Andrew Gordon Wilson, and Ruqi Zhang. Position: Bayesian doep learning is needed in the age of large-scale ai, 2024b. URL https:///axxi http://aaxxvvi.r..a////// hb//ab////24/ 2ttp://4200000999
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Tryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.0024" 2020.
Mohammad Taher Pilehvar and Jose Gamacho-Colladen Wie: the word-in-context dataset for evaluating context-sensitive meaning representations. arXiv preprint Xtv: 09121, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):11-67 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9))99 106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense
reasoning about social interactions. aixie preprint orXiv:1904.000 09728, 2019.
Joram Soch, The Book of Statistical Proofs, Maja, Pietro Monticone, Thomas J. Faulkenberry, Alex Kipnis, Kenneth Petrykowski, Carsten Allefeld, Heiner Atze, Adam Knapp, Ciarán D. McInerney, Lo4ding00, and amvosk. Btaaarro//ooo//tattoooofoffoo StaattroofBookkgiiiii io: StatProofBook 2023. Zenodo, Jamuary 2024. doi: 10.5281 2tt/////eoooo100000
URL https: //op::://doi... org//o.../.... 1ee1111.221///oooee 10495684
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193 24205, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2902. (ete2.11991 13971, 2023.
Christophe Tribes, Sacha Benarroch-Lelong, Peng Lu, and Ivan Kobyzev. Hyperparameter optimization for large language model instruction-tuning arXiv preprint arXiw:2312.0000 00949, 2023.
Mojtaba Valipour, Mchdi Rezagholizadch, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaplation. arXiv preprint arXiv:2210.0755 07.05558, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Alex Wang, Amanproct Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. ar preprinl arXiv:1804.0744 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding sys- tems. Advances in neural information processing systems, 32, 2019.
Hao Wang and Yan Yeung. A survey on bayesian deep learning. ACM computing surveys (csur), 53(5): 1-37, 2020.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Trans- actions of the Association for Computational Linguistics, 7:625-641 2019.
32

---

Published in Transactions on Machine Learning Research (08/2025)
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss-instruct, 2024. URL /ttp://aiii..rr/aab/2//..2.2..
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generaliza- tion. Advances in neural information processing systems, 33:4697-4708, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-arl natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL aclweb httg//anthooogy/220..... e000220.mmmlmmmmmmmmm emnlp-demos. 6
Xingyu Xie, Kuangyu Ding, Shuicheng Yan, Kim-Chuan Toh, and Tianwen Wei. Optimization hyper- parameter laws for large language models. arXiv preprint arXiv:2409.04 04777, 2024.
Adam X. Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL http:://paaaaeeeeeeeeeeeerrrorrr met//forun/iddFFUyyyy
https://openrevier
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg, Bitfit: Simple parameter-effiient fine-tuning for transformer- based masked language-models. arXiv preprint arXiv:2106. 10199, 2021.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parfmeter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.
Shujian Zhang, Xinjie Fan, Bo Chen, and Mingyuan Zhou. Bayesian attention belief networks. In Interna- tional Conference Machine Learning, pp. 12413 12426. PMLR, 2021.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Be- ichen Zhang, Junjic Zhang, Zican Dong, al. survey of large language models. arXiv preprint Xiv:v:2303. 18223, 2023.
33

---

Published in Transactions on Machine Learning Research (08/2025)
A Additional Proofs
A.1 Proof of Lemma 3.1
Let X, Y E R" be arbitrary points. By the Mean Value Theorem for vector-valued functions, there exists a point & lying on the line segment connecting X and Y such that
- VJXY) = -x
(15)
Taking the norm on both sides, we have,
[VJ(X- VJ(Y)I = (X -
(16)
Using the submultiplicative property of norms, specifically the operator norm for matrices and the Euclidean norm for vectors, we obtain
(X -
[(( || YII.
(17)
where (E)I denotes the operator norm (spectral norm) of the Hessian matrix (E). Since HE) is
symmetric (because is twice continuously differentiable), its operator norm equals its largest eigenvalue in absolute value
||( A-aammax(ee (H))))
(18)
Therefore, we have,
[[ VY(Y)] & Amax x |X - -Yll.
(19)
Since & lies on the line segment between X and Y, and X and are arbitrary in R", we can take the supremum over all such &
sup Amax (H(E)) || YIl.
(20)
EER"
Thus, the Lipschitz constant L of the gradient VJ is given by
L sup Amaa (H(X).
(21)
X E
A.2 Proof of Lemma 3.2
The smoothed loss function J(0) can be expanded as,
J(0) = / 110
The gradient of J(() is,
VoJ(0) VoJ E)dr.
The Hessian of J(0) is,
(10)=vaj(e) / VoL( 7)N(y;0 E)dy.
So, the Hessian is the expectation of the Hessian at 0, i.e.,
#( = ( 7))
Since H()) is symmetric and positive semi-definite (for convex L), so is H + 7) for all y. The expected Hessian (0) is convex combination (integral) of Hessians points + 7. Therefore, (()) is also symmetric and positive semi-definite. Let be an eigenvector of (0) with eigenvalue X, then,
A()v Ao.
34

---

Published in Transactions on Machine Learning Research (08/2025)
Using the fact that vrv is scalar, have,
v TH(00 v' [ + +
Since E is linear, we have,
V H 7)0
= E
V. vLv
Let us define
T H(O + 7)2
S(0+7)
v
Therefore,
A = Since H( + +) is positive semi-definite, its eigenvalues S(0+ 2 0. Let Amax be the maximum eigenvalue of H0) over all f. Then, for all 7
+ E,5
S(0+ S Amax.
Taking expectation we get,
A E,|S(( + +)) S Amax-
Hence,
T (0)0
Amax (A(()) =sup
A Amax-
v/0
0 0
B Datasets
B.1 Natural Language Understanding
The General Language Understanding Evaluation (GLUE) Wang al. 2018 and SuperGLUE bench-
marks (Wang et al. 2019) evaluate the language understanding capabilities of LLMs. All the datasets in GLUE and SuperGLUE are obtained from https://gluebeucchig
com/ and https://superr
htuebenchmark.com/ respectively. We elaborate on the GLUE and SuperGLUE tasks as follows:
CoLA /Warstadt et al. 2019 or The Corpus of Linguistic Acceptability, comprises acceptability judgments from books and journal articles. The task is indicate the grammatical correct of the given sentence as "acceptable" or "unacceptable" using the Matthews correlation coefficient the evaluation metric.
MRPC (Dolan & Brockett, 2005), or Microsoft Research Paraphrase Corpus, comprises pairs of sentences extracted from online news sources. The objective is to predict whether the provided pair of sentences are paraphrases of each other or not.
Dagan et al. 2005: Haim et al. 2006 Giampiccolo et al, 2007 Bentivogli et al. 2009 or Recognizing Textual Entailment, are formed by combining a series of annual textual entailment challenges. The task comprises categorizing whether the two sentences entail each other.
RTE
BoolQ (Clark et al. 2019). Boolean Questions, involves binary inquiries sourced from the Google search engine. These questions are combined with pertinent paragraphs extracted from Wikipedia articles, ensuring that the provided paragraphs contain accurate answers to the queries.
WiC Pilehhvrr Camacho- Camacho-Collados: 2018 or Word-in-Context, a word sense disambiguation task that involves binary classification of sentence pairs. Within this task, two text snippets are presented, each featuring word with multiple potential meanings. The objective is determine whether the specified word holds the same meaning in both sentences.
35

---

Published in Transactions on Machine Learning Research (08/2025)
B.2 Commonsense Reasoning
PiQA Bisk et al. 2019). or Physical Interaction: Question Answering, is a reasoning benchmark based on physical interactions in everyday situations. This benchmark comprises questions with two choices, out of which one option is correct based on realistic understanding of the world.
SiQA (Sap al. 2019) Social Interaction Question Answer, is reasoning benchmark based testing social commonsense intelligence. SiQA consists of multiple choice questions, with each question describing social scenario, followed by a query with multiple options, one of which is the correct scenario representing the protagonist's likely intention.
Winogrande (Sakaguchi et al. 2021 a commonsense reasoning task inspired by the WSC ct al. 2012). The goal of the task to choose the correct choice given two choices.
(Levesque
ARC-easy and ARC-challenge (Clark et al. 2018) comprise the AT2 Reasoning Challenge (ARC) parti- tioned into easy (ARC-e) and challenging set (ARC-c). The datasets consist of grade-school science questions in multiple choice format.
OpenBookQA (Mihaylov et al. 2018 contains elementary-level science questions in the form of multiple choices requiring additional commonsense knowlodge.
B.3 GSM8k and Humaneval
GSM8k (Cobbe et al. 2021 is a benchmark designed to evaluate arithmetic and mathematical reasoning in a step-by-step setting. It consists of grade-school level word problems requiring multi-step numerical reasoning. We use the official lest split to evaluate model performance on this task. Following prior work, we report accuracy using the ezact match (EM) metric, which measures the proportion of generated answers that exactly match the reference answer.
To account for variations in output formatting and answer representation, we apply two post-processing filters on model generations: strict-match and flexible-ezttee extract. The strict-match filter expects the model's final output to contain only the correct answer without any surrounding context. In contrast, the flexible- extract filter employs regular expressions to parse and extract the final answer from longer outputs that may include chains of thought, explanations, or extra tokens. Reporting both versions allows us to separately quantify model's reasoning accuracy and its ability to follow formatting instructions precisely.
Evaa (Chen et al. 2021 is widely used benchmark for evaluating code generation through func- tional correctness. Each problem in HumanEval consists of a prompt and hidden unit tests that validate correctness. We evaluate using the pass@k @k metric, where k denotes the number of independently sampled completions per problem. A problem is considered passed if at least one of the A completions passes all unit tests. We report results for & = 2 and 4, following standard protocol. The final pass@k score corresponds to the fraction of problems in the dataset for which at least one generated solution functionally correct.
To create lightweight yet representative training set. for code generation, we curate a subset. from the Magicoder-OSS. Instruct-75K -rstrodde--SSS--nsttttt-------- corpus (Wei et al., 2024). We filter for Python problems whose tokenized input length does not exceed 512 tokens and randomly sample 10,000 such instances.
C Distributions and Reparameterization for Differentiable Backpropagation
Reparameterization trick (Kingma 2013 is often used to make sampling from a probability distribution
differentiable, where the distribution parameters are unknown and learnable parameters of the model. Our method is heavily dependent on reparameterized sampling from different distributions, their correctness and their differentiability. Here we describe the reparameterization techniques for the different probability ions used in the paper.
36

---

Published in Transactions on Machine Learning Research (08/2025)
C.1 Multivariate Gaussian
We denole normally distributed random vector as ~ (H, E). The probability density function of the Normal distribution is given by
fx(x) = (2T))/2/2111 exp [1 1 --2(x- 1i) T (x
x> 12)
where,
(x (1) the deviation of the random vector from the mean.
Σ is the inverse of the covariance matrix.
Σ denotes the determinant of E.
Reparameterizee Sampling and Proof of Differentiability Let a ~ (H, Σ) be a random vector sampled from a multivariate normal distribution, where f IR the mean vector, and E Rdxd the covariance matrix. In the reparamcterization trick, we express the sample x as,
pa Le,
where ~ A( I) is a standard normal vector, and L is the Cholesky decomposition of the covariance matrix Σ, such that Σ = LL". We will prove that the reparameterized transformation = f + Le is differentiable with respect to both l and Σ. The mean vector μe appears linearly in the transformation
x = p Le.
Since the transformation is linear with respect to the derivative of æ with respect to l is,
dx I
0p
where I is the identity matrix. Hence, æ is differentiable with respect to
The covariance matrix Σ enters the transformation through Cholesky decomposition Σ = LL". The Cholesky decomposition a differentiable function of as long as is positive definite. Thus, the entrics of the matrix L are dilferentiable functions the entries of Σ. Since the transformation æ = / + Le depends linearly on L, can apply the chain rule to differentiate æ with respect E. Let fEE E) g()) where g is the Cholesky factor L. The derivative of with respect to E is
dx 0Σ
dx dL
0L a
Since is a linear transformation of L, and L is differentiable with respect to E, is differentiable with respect to Σ.
C.2 Wishart Distribution
We denote Wishart-distributeed random matrix W ~ W,(V,,) v). The probability density function of the Wishart distribution is given by,
W (u---11) 11/2 exp
tr (v-'w))
Sw(W) W)
2"p/2 /
Reparamcterized Sampling and Proof of Differentiability
Wishart distribution the conjugate prior of the precision matrix (inverse covariance-matris) of multivari- ate Gaussian distribution with unknown variance. For sampling Σ Wm (V, n) with V E mm being the
37

---

Published in Transactions on Machine Learning Research (08/2025)
scale matrix and degrees of freedom, we first calculate Cholesky decomposition LL" V, with lower triangular matrix L. As V is diagonal, L can be directly calculated as
VV
Next, we sample Σ ~ (I,,n)) n). The reparametrized variance matrix is E LÉLT. Let E ~ W,(. n)) be a sample from the Wishart distribution, where V E IPXP is a posilive-definite scale matrix, and n is the degrees freedom.
The Cholesky dccompasition relates V and L through the equation V LL". The elements of V are quadratie functions of the elements of L,
min(6d)
Vij
Σ
This relationship is differentiable, and the partial derivatives of Eag with respect Vu are well-defned. Thus, L is differentiable function of V. The reparameterized Wishart sample is given by,
Σ = -.LL
Using the chain rule, we compute the derivative of E with respect to V as,
σΣ
2Σ OL
àL ov
ov
The differentiation of Σ with respect to V proceeds as follows. Using the chain rule, we first differentiate Σ with respect to L, and then differentiate L with respect to V
σΣ
2Σ aL
OV
OL
ov
Since Σ = LÊLT, the derivative with respect to is,
OΣ 0
OL Ol (LEL") = ELT -
From the above, we can conclude that the reparamterised fromulation of Wishart Distribution retains its differentiability with respect to its parameters
C.3 Dirichlet Distribution
We denote Dirichlet-distributee random vector as 0 ~ Dir(a), where 0 = (01,02,... 02,..,.... (e)) and ΣK 0, The probability density function of the Dirichlet distribution is given by:
K
fo(0) = B(a)
II
8 a0-1
where
0, 2 0 for all i 1,..., K, and ΣK| 0 1.
B(a) is the multivariate beta function, defined as
k T(a:)
B(a)
r EK
38

---

Published in Transactions on Machine Learning Research (08/2025)
r()) is the Gamma function.
Reparamcterized Sampling and Proof of Differentiability
Suppose we want to sample π ~ Dir(a)), where a = (ai....,,ar' .x)) is the concentration parameter. We first sample K independent Gamma-diste distaa-dstributed variables yi Gamma(a;, 1), for I...., K. To ensure differentiability of the sampling process, we apply the reparameterization trick by expressing each y, as,
yi a, (- log
where e, Uniform(O, 1) an independent uniform random variable. Finally, the reparameterized Dirichlet sample p obtained follows.
We sample K independent Gamma-distributed variables y/ Gamma(a;, 1), for i 1,..., K. The repa- rameterization of the Gamma distribution is given by,
y/ g = a, (- (=log(G)),
where 6 Uniform(0, 1). Since the transformation g(a,, e,) is differentiable with respect ag, we have,
dy: log(())
da;
Thus, Yr is differentiable with respect to a; After sampling Ut, we normalize the variables to obtain the Dirichlet sample,
2 4
The normalization is smooth function of the yı8, so the resulting p; is differentiable long as Σ 1/ > 0 (which holds because each yi 0). The derivative of pt with respect to yi is,
Op:
Σド زo - yt
Oy:
K (Σ
U/)
Since y is differentiable with respect to oi, the sample p (p1,P2,'' pK3 is differentiable with respect. to
D Calculation of KL Divergence Losses
For calculating the KL Divergence of the distributions used in this paper, we refer to Soch et al. 2024 The detailed derivations can also be found in Section 1 of the supplementary material.
Multivariate Normal. The KI. divergence between N(1,, Σ1) and Q ~ NN Ez) given by,
KL[P|[Q| 1 (12 u1) EE' - p1) tr Σn In
Σ1
Σg
Wishart Distribution. The KL divergence between P ~ (p (V m1) and Q ~ W/ (2,, is given by,
KL[P[[Q] 2 n2 (In V2 In [V1) + nitr V2 Vi) + 2ln
U1 r
141 (ni 12) (m) - 71p
2
Dirichlet Distribution. The KL. divergence between P Dir(on) from Q Dir(az) is given by,
r -i=1 a1,
[(a2.) ((a1.)
+ Σ (ai, azd) p(a1o) - 16 Σ a.. ]
[L[P in r
+
In
> 02,1
39

---

Cholesky Decomposition the Scale Matrix. The time complexity of computing the Cholesky decomposition for p x p matrix typically Ogr²). However, since we are using diagonal prior, the time complexity reduces (pp) or O(r), we are only sampling for LoRA A, and the prior is Published in Transactions on Machine Learning Research (08/2025)
E Complexity Analysis of Sampling
Training MonteCLoRA involves sampling from three different distributions. Therefore, essential to analyze the complexity of sampling from these distributions.
E.1 Sampling from the Wishart Distribution
2. Sampling from the Standard Wishart Distribution Σ ~~ Wp(,, k). We generate k independent samples from a multivariate normal distribution (N Ip), with each sample involving values. Generating the covariance matrix for each sample takes Opp²) time. Therefore, the total time complexity for generating samples is Okp"). In our context, this corresponds to O(nn7²).
Reparameterization
". The reparameterization step involves matrix multiplication and thus has a time complexity of O(p²). In our case, it is (r))).
The total time complexity is O(r ((nr).. Since r < min(mm oout), the complexity heavily influenced by Mim- mitigate this problem, we reduce the degrees of freedom to O(r). In practice, we found little difference between two degrees of freedom; therefore, we use the Wishart prior with degrees of freedom r.
E.2 Sampling from the Dirichlet Distribution
The time complexity of sampling from the Dirichlet distribution using the reparameteriiztiin trick can be broken down into the following steps
Reparameterization via Gamma Distribution. Sampling from a Gamma distribution with shape parameter a; and scale parameter 1 has time complexity ((1) for each variable yt. Sampling N independent Gamma-distributed variables takes ((N) time.
2. Normalization Step. After generating Gamma-cistribbueed variables ye, normalize them to produce the distbietdd sample p = (PI....,PN) Calculating the denominator involves a summation taking O(N) time, and normalizing each variable takes O(1) time. Therefore, the total time complexity for the normalization step is O(N).
Thus, the total time complexity of sampling from Dirichlet distribution is O(N).
E.3
Sampling from a Multivariate Normal Distribution
The time complexity of sampling from a multivariate normal distribution using the reparameterization trick is as follows
1. Cholesky Decomposition of the Covariance Matrix. The time complexily for computing the Cholesky decomposition of a d x d covariance matrix Σ is (dd)) Since we use a diagonal covariance matrix, the complexity reduces to O(d). For us, it is (rr).
Sampling from the Standard Normal Distribution. The time complexity for generating a vector of standard normal variables Odd). For our case, this is O(r).
3. Reparameterization
/ Le. The matrix-vector multiplication Le involves multiplying a d d matrix by dx1 1 vector, which has time complexity of Odd?). For MonteCLORA, this is O(r2).
Thus, sampling one random variable from this multivariate normal distribution has a time complexity of (rr2). Since we need to sample nia random variables for a weight matrix, the total time complexity for sampling is ))
40

---

Published in Transactions on Machine Learning Research (08/2025)
E.4 Total Sampling Complexity for the Entire Model
Combining the time complexilies for the three sampling steps mentioned above, the time complexity for sampling in one layer is,
O(r + nin x² + N + Ninr))
which simplifies to
O(n N),
assuming that < max( Rout). Let Min be the largest imput dimension for any layer in the model, and let the number of MonteCLoRA-emhancee layers be Tmme- The total time complexity for sampling across all layers is,
OLLmo + N)).
Since nin typically constant for most models, the total time complexity primarily governed by the number of MonteCLoRA layers, Lme: and the LoRA rank,
41


Reviews:
[{"review_id": "WVBXT111d5", "reviewer": "Reviewer_rGiY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- **Innovative Approach**: MonteCLORA blends Bayesian inference with Monte Carlo methods, offering a new way to tackle hyperparameter sensitivity in fine-tuning.  \n- **Solid Theoretical Grounding**: Detailed mathematical analysis backs up its claims about stability, giving it a strong foundation.  \n- **Strong Empirical Results**: Across various tasks and models, it consistently improves accuracy, robustness, and convergence—like a 53% lower performance spread in NLG tasks with LLaMA-1-7B.  \n\n## Weaknesses  \n- **Increased Computational Cost**: Monte Carlo sampling pushes training time complexity to O(n_in r² + N), which might hinder scalability for big models or limited resources.  \n- **Implementation Complexity**: The Bayesian setup could be tricky for users who aren’t familiar with probabilistic methods, even with the theory explained.  \n- **New Hyperparameters**: MonteCLORA brings in its own parameters, like the number of Monte Carlo samples, which still need some tweaking.\n- **Limited Generalizability Testing**: While effective on tested models and tasks, the method lacks evaluation across a broader range of LLM architectures or domains (e.g., beyond HumanEval for code generation). \n  - *Update*: Given the inclusion of multiple additional experiments, this concern appears to be sufficiently addressed. Consider removing this point to reflect the updated scope of the evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces MonteCLORA, a Bayesian approach to improve the efficiency and robustness of fine-tuning large language models (LLMs) using low-rank adaptation. Its key contributions are:  \n- **Theoretical Analysis**: It explores how hyperparameters influence LLM fine-tuning, focusing on the sensitivity of low-rank methods like LoRA to learning rates and batch sizes. Mathematical proofs tie these choices to optimization stability.  \n- **Proposed Method**: MonteCLORA leverages Monte Carlo estimation to learn posterior distributions of low-rank parameters, modeled as mixtures of multivariate Gaussian distributions with Wishart priors. This boosts stability while adding just O(r) extra parameters.  \n- **Empirical Validation**: The method was tested on five natural language understanding tasks and six generation tasks with models like RoBERTa-base and LLaMA-1-7B. It achieved up to 3.8% better accuracy, 8.6% more robustness, and a 62% smaller performance spread with LLaMA-3.2-3B-Instruct.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Quantify Computational Trade-offs**: Add a clear breakdown of how Monte Carlo sampling increases training time—think benchmarks like hours per epoch—and highlight cases where the gains outweigh the costs.  \n2. **Outline Limitations**: Add a subsection to discuss limitations of this work.\n3. **Outline Future Directions**: Suggest next steps, like making sampling more efficient, trying MonteCLORA in pre-training, or scaling it to bigger models.  \n4. **Figure 1 Fix**: Move the numbers above or below the boxes—right now, they overlap with lines, making it hard to read.  \n5. **Figure 3 Caption**: Add a quick summary of each step in the caption so readers can follow the process without flipping back to the text. \n6. **Lemma 3.1 Clarification**: Define \\(n\\) upfront—it’s not clear what it stands for yet.   \n7. **Lemma 3.2 Notation**: Shift the tilde to \\(\\tilde{J}(\\theta)\\) for clearer notation.  \n8. **Lemma 3.3 Reference**: Fix the reference to “Lemma 3”—it’s probably meant to be Lemma 3.2? \n9.  **Lemma 4.1 Assumption**: Spell out the assumption behind the norm (e.g., is \\(X \\geq 0\\) required?) to make the step airtight.  \n10. **Lemma 4.5 Detail**: Explain where \\(\\sum = 1\\) comes from—it’s not obvious without more context.  \n11. **Consistent Color Coding**: Stick to the same colors for each method across all figures (e.g., MonteCLORA in red) to make them easier to track.  \n12. **Comparison Tables/Figures**: Add arrows (↑/↓) in all tables and all figures to show whether higher or lower values are better, like in Table 5, for quicker understanding.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rlj79cU9iF", "reviewer": "Reviewer_2dxC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The authors recognize the issue of instability in parameter-efficient finetuning methods like LoRA and propose a Bayesian approach to solve the problem. \n\n- The authors provide good analysis of the various properties of the Bayesian estimates and also ablation studies. \n\n- For weaknesses, one main issue is the empirical results do not seem to support the conclusions. For example in Figure 4, LoRA and MonteCLoRA have similar accuracies, but the spread of accuracy is actually larger for MonteCLoRA in 4 out of 5 datasets. The same is true of other tables comparing LoRA and MonteCLoRA like Table 5 and Table 6. The results on Llama2-7B seems to be better in terms of reducing estimation variance, but the accuracy is no better. Also, why do the authors not evaluate on the full GLUE dataset? \n\n- In this approach we still need to choose the hyperparameters for LoRA like rank r and \\alpha, learning rate and batch size. In a fully Bayesian approach we should be able to infer from data. \n\n- There are also many clarity issues: \ni. In Figure 1: is the instability shown due to choice of different random seeds or different hyperparameters? It is not detailed there. \nii. Algorithm 1 is not clear. What does the last step of updating the weight matrix mean? Is it done by SGD? And shouldn't we be updating the trainable parameters of the different distributions? \niii. The scale factor \\epsilon in Lemma 4.1 does not seem to be defined anywhere and only appears once in Algorithm 1 but not explained. What is it?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a Bayesian-style PEFT method called MonteCLoRA. They propose Bayesian-style priors and sampling for the low-rank parameters used in LoRA and prove various properties of the estimation. Empirical evaluations show similar performance to LoRA and reduced variance in parameter estimates in some cases.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please address the weaknesses mentioned above.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h1BKoKW0iI", "reviewer": "Reviewer_QNMP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The authors provide a theoretical argument for their proposed method's robustness and that the final model output by MonteCLoRA is an unbiased estimator of the original model output\n- The authors do not provide experimental results on more common NLG benchmarks such as GSM8k and HumanEval.\n- The proposed method is ~2× slower than LoRA", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors devise a method to improve the robustness/variance of the LoRA fine-tuning process\n- The authors present evidence of MonteCLoRA's performance on the GLUE benchmark (ROBERTa) as well as various NLG tasks (Llama7b).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The way the introduction is worded may lead the reader to believe that the robustness of finetuning with LoRA is being addressed, as it is worse than that of full fine-tuning, which is not the case as shown in Fig. 1. I would like for MonteCLoRA to be presented in Fig. 1 as well and the motivation be a bit more clear.\n- I would like to see a comparison with alternative PEFT methods such as IA3 and Prompt Tuning", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)