Title: Training-Free Bayesianization for Low-Rank Adapters of Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Uncertainty Estimation

Bayesian Deep Learning
Abstract: Estimating the uncertainty of responses from Large Language Models (LLMs) remains a critical challenge. While recent Bayesian methods have demonstrated effectiveness in quantifying uncertainty through low-rank weight updates, they typically require complex fine-tuning or post-training procedures. In this paper, we propose **T**raining-**F**ree **B**ayesianization (**TFB**), a simple yet theoretically grounded framework that efficiently transforms trained low-rank adapters into Bayesian ones without additional training. TFB systematically searches for the maximally acceptable level of variance in the weight posterior, constrained within a family of low-rank isotropic Gaussian distributions. Our theoretical analysis shows that under mild conditions, this search process is equivalent to KL-regularized variational optimization, a generalized form of variational inference. Through comprehensive experiments, we show that TFB achieves superior uncertainty estimation and generalization compared to existing methods while eliminating the need for complex Bayesianization training procedures.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Training-Free Bayesianization (TFB), which makes pre-trained LoRA adapters into Bayesian adapters. It provides a practical way of doing uncertainty quantification in LLMs, and the paper shows that TFB's variance maximization is equivalent to generalized variational inference with some mild conditions. The paper also shows superiority in empirical results.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- This paper brings a novel way to change pretrained LoRA adapters into a Bayesian setting\n- It also provides theoretical grounds for showing the equivalence between TFB's variance maximization and generalized variational inference with mild conditions\n- It covers a decent amount of dataset/benchmark models to compare in the empirical analysis\n\nWeakness\n- Some of the arguments are not clear, and the presentation could be improved. Also, it is a little bit confusing why they call this Bayesianization at the first place.\n- Please refer to the questions for more details\n\nQUESTIONS:\n- line 21: \"its internal internal\" is it a typo?\n- line 86: When you denote matrix as $\\boldsymbol{X}=\\left[\\boldsymbol{x}_1, \\cdots, \\boldsymbol{x}_n\\right] \\in \\mathbb{R}^{m \\times n}$ with comma separted, I don't think your definition for notation of the vector, $\\operatorname{vec}(\\boldsymbol{X})=\\left[\\boldsymbol{x}_1^{\\top}, \\boldsymbol{x}_2^{\\top}, \\cdots, \\boldsymbol{x}_n^{\\top}\\right]^{\\top} \\in \\mathbb{R}^{(m n) \\times 1}$, could be confusing\n- line 93: What is \"the\" LoRA layer you are referring to here? I think it would be better to mention what that layer is.\nIs there any difference between \"LoRA\", \"LoRA adapter\", and \"low-rank adapter\"? The authors seem to use different expressions that refer to the same thing.\n-  line 100: What is the \"prior\" here? Are $B, \\theta$ observations? It is hard to understand what \"Bayesianized low-rank adapter's posterior\" is. Why do you call it Bayesian at all?\nAlso, I guess since the authors used $q$ to denote the approximate posterior(?) of $\\operatorname{vec}(\\boldsymbol{W})$, should I also understand $q\\left(A_{i j}\\right)$ to be the approximate posterior of $A_{i j}$?\n- line 126: Is it correct that the mean of the posterior distribution of a random variable is the random variable itself? ( $\\boldsymbol{M}=\\boldsymbol{A}^{\\prime}$ )\n- line 186: What is $P_{\\boldsymbol{\\theta}}(y \\mid \\boldsymbol{x})$ and what is $P\\left(\\boldsymbol{y} \\mid \\boldsymbol{x}, \\boldsymbol{W}_n\\right)$? Is the latter a true probability density function?\n- line 187: Does $\\theta$ include $B$ as well? or is it just $\\{M, \\Omega\\}$ as it was first defined in line 101\n- line 188: It is not valid to justify the selection of such a small number of Monte Carlo samples ( $N=10$ ) by referring to previous work. Do the authors acknowledge possible errors due to the gap between $P_{\\boldsymbol{\\theta}}(y \\mid \\boldsymbol{x})$ and the empirical average?\n- line 210: Missing \\prime for A on the right-hand side? $q\\left(\\boldsymbol{A}^{\\prime} \\mid \\boldsymbol{\\theta}\\right)=\\prod_{i j} \\mathcal{N}\\left(A_{i j} \\mid M_{i j}, \\Omega_{i j}^2\\right)$\n- In table 1, is the standard deviation calculated with $N=10$ samples?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers uncertainty estimation for Large Language Models (LLMs), specifically low-rank adapters for LLMs. It offers a simple approach to do with low-rank matrix found by an existed low-rank adapter, then treating it as matrix of isotropic Gaussians with the same variance. This variance is optimised (as an example binary search is used in the paper). Then predictions are averaged with weights sampled with these Gaussians. Despite simplicity of this approach it is shown to be theoretically grounded and equivalent to generalised variational inference.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n*Quality*: The main claims of the paper are supported with good quality experiments and also grounded theoretically linking the proposed approach with generalised variational inference.  \n\n*Clarity*: The paper is well written and easy to follow, after reading it though I have one confusion, please see below. \n\n*Significance*: With the spread of LLM usage uncertainty estimation is very important yet it still remains an open problem, therefore this paper aiming to address it can have a significant impact. \n\n*Originality*: The idea appears to be novel.\n\n**Weaknesses:**\n\n*Clarity and originality*: The paper emphasises that it offers a training-free method, which makes it unique amongst existing methods. However, I am finding myself confused why it is claimed to be training-free. It still requires to find the variance. I can get that on a higher level it is not training, but from the practical point of view, isn’t it just the same as training? \n\n*Originality*: The paper is generally well put into the existing literature but after a quick search I have found this work, for example, Onal, Emre, et al. \"Gaussian stochastic weight averaging for Bayesian low-rank adaptation of large language models.\" arXiv preprint arXiv:2405.03425 (2024) - that seems to be related. \n\n\nOther comments/suggestions:\n1. Line 241. Missing reference for Meta-Llama-3.1-8B\n2. Line 257. The acronym PEFT is not introduced (only introduced later in the Appendix)\n3. Figure 1 is unclear. It has a lot of elements that needs to be described. For example, what fire symbols mean, why $W_0$ is faded for TFB but still contributes to $z$, what different colours mean, etc.\n4. Lemma D.1 is placed in a weird place, namely before the theorem, while the proof of the theorem says “We have the *following* lemma” as if it should be after this text. Moreover, as the reader reads the text and encounters the lemma first, it stands there very confusingly without a proof or reference. \n5. Figure 2. In the first two plots apparently red and blue lines are inseparable. Maybe making one of the lines dashed would be better for visibility. At least text explanation is needed. \n \n \nMinor:\n1. Lines 344-345. “Beyesianization” -> ‘Bayesianization”\n\nQUESTIONS:\nFrom the practical point of view, why do you claim TFB to be training free if it still involves optimisation of the variance (which requires a training dataset)?\n\nLIMITATIONS:\nThe authors discuss both the limitations of their work and societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a training-free approach for Bayesianizing low-rank adapters (LoRA) for large language models (LLM). Bayesianizing low-rank adapters is crucial in estimating uncertainty in LLM responses, therefore of utmost practical importance. The proposed method, TFB, models the posterior over LoRA weights using low-rank isotropic Gaussian distributions, and automatically selects the maximum variance that preserves task performance using an anchor dataset. The authors prove that TFB is theoretically equivalent to generalized variational inference under mild assumptions. Empirical results across multiple LLM backbones and datasets show that TFB improves uncertainty calibration (lower ECE, NLL) while preserving accuracy and significantly reducing complexity compared to prior Bayesian LoRA methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. The overall writing of the paper is very good, quite easy to follow.\n2. TFB works on trained LoRA adapters without retraining or any gradient updates, making it highly practical and scalable.\n3. The theoretical results provided in the paper are solid and relevant.\n4. Experimental setup is pretty rigorous. TFB is shown to work well across different LLMs (LLaMA, Mistral) and LoRA variants (VeRA, PiSSA).\n\n**Weaknesses**\n\n1. Asymptotic inference-time complexity of TFB is not provided in the paper.\n2. TFB is sensitive to tolerance level $\\epsilon$. Some experiments should be conducted to understand the extent of the sensitivity.\n3. TFB requires access to an anchor dataset to search for posterior variance, which might not always be available in zero-shot settings. Also, robustness of TFB across different anchor dataset should be studied.\n4. Restricting to isotropic low-rank Gaussians simplifies inference but might limit modeling capacity compared to full-covariance Bayesian methods. Can we see some ablation to understand the trade-off?\n5. I am not fully convinced with the performance of TFB. Although its obvious that TFB improves uncertainty estimation (NLL and ECE) over MLE, MAP and BloB-mean, it seems to be performing poorly than LAP and BloB in many instances (as reported in Table 1). Statistical significance tests should be conducted.\n6. Can you compare your method with MonteCLoRA [1]? MonteCLoRA seems to be another posterior estimation method, works reliably well on LoRA. It can work in both ad-hoc and post-hoc settings, making it a strong competitor of TFB.\n\n**References**\n\n[1] https://arxiv.org/pdf/2411.04358\n\nQUESTIONS:\n1. Can you clarify what you mean by \"more compact\" (line 107, 130)? \n2. Can you move limitations to the main text?\n3. How do we select an appropriate anchor dataset for a given model and downstream task?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to “bayesianize” lora finetuning of LLMs by using as the variational posterior the lora mean and a grid searched variance term on the training data. The loss used to search the variance term is the variance that keeps the maximum functional change in loss underneath a pre-specified threshold. Experiments are done on a pretty wide variety of LLM style benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-\tI appreciate the simplicity of the approach, as well as the toy demonstration in Figure 2 in the appendix. \n\n-\tI like the ablation studies with other versions of the approach; indeed, it seems that the structure you chose is probably the best tradeoff performance / compute wise.\n\n-\tI also like that you can just tune the variance on a pretrained Lora model, thereby getting this for a few passes through the finetuning data.\n\nWeaknesses:\n\n-\tChoosing the prior mean the same as the posterior mean in Thm 4.2 is definitely putting the cart before the horse. Under those conditions, pretty much anything is variational inference. The overselling of this result is one reason why i'm voting weak accept currently; this is a pretty restrictive set of assumptions (not a \"relatively weak\" set).\n\n-\tNo comparison to something like simply turning up the generation temperature to increase output uncertainty and then using many samples in the Table 1 benchmarks. \n\n-\tI appreciate the attempt to phrase the method as Bayesian, but that’s really a strong overstatement. The “meta” prior here is “any weight distribution that doesn’t change the function too much”, which is distinctly not Bayesian, but more a post facto way of justifying some sort of uncertainty quantification.\n\nQUESTIONS:\nGenerically how is epsilon chosen here as I imagine that’s a clearly important hyperparameter? If this has to be chosen on a reference task, how do we ensure that this reference task represents the downstream usage tasks?\n\nWhat is the inference computational cost as compared to standard LLM inference?\n\n-\tIt’s not super clear to me what calibration error on textual generation / a text bot should actually mean. I think I understand when and how accuracy will be better (even from just the variational mean), e.g. we get more questions right.\n\nWhy is the Laplace approximation often better than MAP / MLE in Table 1?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}