Training-Free Bayesianization for Low-Rank Adapters of Large Language Models
Authors: Haizhou Shi
Yibin Wang
Ligong Han
Huan Zhang
Hao Wang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Uncertainty Estimation
Bayesian Deep Learning
Primary area: probabilistic_methods
TL;DR: A Training-Free Bayesianization approach is proposed for LLM adapters that achieves better uncertainty estimation.

Abstract: Estimating the uncertainty of responses from Large Language Models (LLMs) remains a critical challenge. While recent Bayesian methods have demonstrated effectiveness in quantifying uncertainty through low-rank weight updates, they typically require complex fine-tuning or post-training procedures. In this paper, we propose **T**raining-**F**ree **B**ayesianization (**TFB**), a simple yet theoretically grounded framework that efficiently transforms trained low-rank adapters into Bayesian ones without additional training. TFB systematically searches for the maximally acceptable level of variance in the weight posterior, constrained within a family of low-rank isotropic Gaussian distributions. Our theoretical analysis shows that under mild conditions, this search process is equivalent to KL-regularized variational optimization, a generalized form of variational inference. Through comprehensive experiments, we show that TFB achieves superior uncertainty estimation and generalization compared to existing methods while eliminating the need for complex Bayesianization training procedures.

Training-Free Bayesianization for Low-Rank Adapters of Large Language Models
Haizhou Shi
Yibin Wang +2
Ligong Han 3
Huan Zhang
Hao Wang +1
Abstract
Estimating the uncertainty responses from Large Language Models (LLMs) remains a critical challenge. While recent Bayesian methods have demonstrated effectiveness in quantifying uncertainty through low-rank weight updates, they typically require complex fine-tuning or post-training procedures. In this paper, we propose Training-Free Free Bayesianization (TFB), simple yet theoretically grounded framework that efficiently transforms trained low-rank adapters into Bayesian ones without additional training. TFB systematically searches for the maximally acceptable level of variance in the weight posterior, constrained within fam- ily of low-rank isotropic Gaussian distributions. Our theoretical analysis shows that under mild conditions, this search process is equivalent KL-regularized variational optimization, generalized form of variational inference. Through comprehensive experiments, we show that TFB achieves superior uncertainty estimation and generalization compared to existing methods while eliminating the need for complex Bayesianization training procedures. Code is available https: /ttp:://iiiiii github. com/Wang ML- ML-LMLLLLBBBB Std-Labbbayesiie -pett.
Introduction
Despite recent advances in Large Language Models (LLMs) showing great capacity for generating responsive answers human instructions [5, 64, 63, 44, 9, 54, 55, 49, 8, 48], the reliability of such large models remains critical concern [59, 58], as untruthful yet confident answers could cause sigmificant damage individuals and society [17, 47, 66, 28]. The accurate estimation of uncertainty in LLMs has thus emerged as an urgent challenge. Current approaches mainly follow two paths: one focuses on directly asking the model elicit its internal internal (verbalized) uncertainty [65, 52, 28], while the other employs complex fine-tuning techniques [28, 67, 62].
Both approaches suffer from inherent limitations. Verbalized uncertainty, while simple implement, remains controversial in terms of its empirical reliability and theoretical soundness [27, 38]. On the other hand, low-rank adapters (LoRA [23]), which offer a parameter-efficient way to adapt LLMs by adding small set of low-rank weight matrices, have emerged as a promising direction for fine-tuning models. However, while LoRA efficiently adapts large models new tasks, it does not itself provide mechanism for principled uncertainty estimation. In response, recent Bayesianization attempts [67, 62], integrate Bayesian methods with LoRA. but they still require complex training procedures and sophisticated hyperparameter tuning, limiting their practicality. These constraints motivate the following research question:
Can we "Bayesianize" LLM low-rank adapters in a theoretically sound yet empirically simple way?
In this paper, we diverge from conventional fine-tuning and post-training approaches. Instead, develop Training-Free Bayesianization (TFB) technique applicable to any given low-rank LLM adapter. TFB constrains the family of full-weight approximate posteriors produced by LoRA adapters to low-rank isotropic Gaussian distributions. Given a trained LoRA adapter, it systematically searches for the maximally acceptable variance of the variational distribution of the weight posterior, without
'Equal Contribution, Rutgers University. University of Illinois Urbana-Chane Urbana-Champaign (UIUC). "Red Hat AI Innovation. "Carrespondence to: Haizhou Shi chaizhou.shi rdu>, Hao Wang <hw488 @ c..ru.gers.dd..
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

the need for complex fine-tuning procedures, TFB's search range and stopping criteria can be determined using any in-distribution "anchor dataset," e.g., a small subset of the training dataset. Note that (1) this eliminates the need for an additional calibration or validation dataset; (2) this flexibility extends to both supervised and unsupervised data, even regardless of whether it was used in the original LoRA training.
Despite its simplicity, we theoretically demonstrate that, TFB's process of finding the maximal variance of the low-rank isotropic Gaussian posterior is equivalent to generalized variational inference, under mild conditions.
We verify TFB's 's effectiveness through extensive empirical evaluation across various settings, datasets, LLM backbones, LoRA weights. and LoRA variants. Our comprehensive experiments demonstrate that this novel training-freet free Bayesianization framework consistently achieves superior generalization and more accurate uncertainty estimation. To summarize, the main contributions of this paper are:
We propose Training-Free Bayesianization (TFB), the first framework to transform trained LoRAs into Bayesian ones without re-training, continued training, gradient estimation. We establish theoretical connections between TFB and generalized variational inference, proving their equivalence under mild conditions.
We develop an efficient implementation of TFB requiring only an anchor dataset for search, making it widely applicable across different application scenarios.
Through comprehensive experiments, we demonstrate that TFB consistently improves uncer- tainty estimation for fff-the-shelf LoRA adapters, and overall surpasses the state-of-the-art counterparts of Bayesian LoRA.
2
Related Work
LLM Uncertainty Estimation. To estimate the uncertainty of LLMs, the models are often em- ployed to generate and evaluate their own uncertainty [40, 27 However, such approaches typically rely task-specific labels and require additional training. Semantic entropy [38] leverages the invariance of language stemming from shared meanings to estimate uncertainty, while mutual infor- mation is used to compute lower bound on model uncertainty by sampling from the model's output distribution [66]. Despite their contributions, these methods fail to accurately capture true model uncertainty, they do not model the probability distribution over the LLM parameters [25, 1, 15].
Bayesian Low-Rank Adaptation. The Bayesian framework provides a powerful approach for capturing and estimating uncertainty during fine-tuning by defining prior distributions and approx- imating posterior distributions over the model parameters [46, 21, 14, 56). Recent research has explored combining Bayesian methods with LoRA to mitigate the additional computational overhead associated with modeling parameter distributions across the entire parameter space. Yang et al. [67] applies Kronecker-factorizee Laplace approximation to fine-tuned LoRA parameters. More recently, BLoB [62] advances the field by simultancously estimating both the mean and covariance of LLM parameters within single fine-tuning stage. Our proposed training frree Bayesianization represents a significant departure from these existing methods. Unlike approaches that require re-training [1 [14, 61, 4, 62] rely continued training and gradient estimation [67], method achieves uncertainty estimation without any additional training steps, substantially improving the simplicity and efficiency for Bayesian learning of LLMs.
3 Training-Free Bayesianization (TFB)
This section introduces our Training- Free Bayesianization (TFB). Sec. 3.1 introduces the problem setup. Sec. 3.2 and Sec. 3.3 present the two key parts of TFB: low-rank Gaussian variational distribu- tion family and a novel approach for converting deterministic weights to probabilistic distributions without training. The complete algorithmic implementation is provided in Sec. 3.4, with theoretical foundations addressed in a separate section (Sec. 4).
Notation. Scalars, vectors, and matrices are denoted by lowercase letters, lowercase boldface letters, and uppercase boldface letters, respectively. For a matrix = | ... ", .r. E R we use veo( (X) = [ [ .... "..".,]]'""" [ [mmm)x to denote vectorization. 6 and o denote the Kronecker and element- product, respectivly.. We use O,, gaxa to denote zero matrix.

---

2
+
Premmined Weights
B
Pretrained Weights
eppy Pretranned
M,
W3 g
SuPgl A
Wa mmnen
Wa grann
A
A
R2
BLoB
(b) LoRA
(c) Training Free Bayesianiationn (TFB)
Figure I: Overview of Training-Fri Free Bayesianization (TFB, Ours, right), as well as comparison with existing methods such as LoRA (middle) and BLoB (left).
3.1 Preliminaries
Low-Rank Adaptation (LoRA). Given pre-trained neural network layer with weight matrix
Low-Rank Adaptation (LoRA) [23] confines weight updates low-rank subspace during fine- tuning, expressing the update as A AW BA, where AW E BR B E Rmxx, and A E TR"x. For input h and output of the LoRA layer, the forward pass computation is then given by:
Wo.
2 Woh + AWh Woh + BAh.
(1)
LoRA Bayesianization with Low-Rank Gaussian Distribution. BLoB [62], pioneering work
in low-rank Bayesianization for LLMs, empirically demonstrates that modeling A's elements with independent Gaussian variables suffices for effective uncertainty estimation in LoRA. Specifically, the probability density of each element of A follows ((A,)) ((Ag, S vi E [r]. [m]. where matrices M and Ω sharing the dimensions of A, represent the mean and standard deviation of the random variable A, respectively. This formulation is equivalent to approximating the Bayesianized low-rank adapter's posterior in the full-weight space of W with a low-rank degenerate distribution:
g(vec( W) B, 0) = (Veee ()) Eg);
where 0 = [2) denotes the set of parameters for modeling A's posterior distribution, Myg vec( W is its mean, and M, I @ B] [
J J its low-rank degenerate covariance matrix. In this paper, we adopt similar approach for modeling the variational distribution of the weight posterior, focusing exclusively on Bayesianizing the weight update matrix A.
3.2 TFB's Variational Low-Rank Isotropic Gaussians
Variational Distribution Family. In TFB, constrain the variational distributions of the weight posterior to a more compact family of Gaussians than BLoB: specifically, we employ full-space isotropic Gaussian distributions projected onto the low-rank space:
q(vec( (W) B, 0) -N(vee (W) proj(o?
where By is defined as in Eqn. 2. Here, of E mm represents full-rank isotropic covariance matrix with standard deviation og. and proj(-) denotes linear projection operator that maps the full-space covariance matrix onto the low-rank space (see Proposition D.1.1 for details).
TFB Generalized Variational Inference. The choice of low-rank isotropic Gaussian approx- imate posteriors serves both theoretical and empirical purposes: it provides single-parameter family that enables converting the generalized variational inference into a variance maximization problem (more details in Sec. 3.3, Theorem 4.2, and Appendix F.1), and empirically outperforms alternative distribution families (Sec. 5.4). Below, present a practically efficient implementation for Bayesianizing LoRA under the constraint specified in Eqn. 3, with detailed theoretical analysis provided in Theorem 4.1.
TFB in Practice. Consider a LoRA layer with weight updates E mxx A E Tx and a standard deviation scale og > O. We begin by computing the compact Singular Value Decomposi- tion (SVD) [33] of B:
B = U diag (aaag(d) V
"proj(:) only depends the rank of the trained LoRA.
3

---

where U E m and V E Rexr are orthonormal matrices, and d = [d1, d2, d is the vector consisting of singular values with all positive entries". We then transform the original weight matrices {B, A} into an equivalent pair
BB' = U diag( d), A' VTA),
(5)
maintaining the equality AW BA B'A'. Following BLoB's Asymmetric Bayesianization scheme, we define the variational distribution for A' uuing the mean matrix M A'' and the standard deviation matrix Z E Rrx*, such that
(A,,)) = (A', May, Ogy), ), Vi E [r]. E [n].
(6)
Unlike BLoB, our Ω is not freely parameterized but instead derived from projecting the full-space matrix o,d I onto the low-rank weight space:
Si sv/d.,
Vi E [r]. Vj [m],
(7)
where d is defined in Eqn. 4. This solution can be expressed compactly as Ω ['=/a,... 0d/d], comprising n repeated vectors. To summarize, our TFB
takes input trained LoRA matrix pair {B U diag(d)V .A} and predetermined
standard deviation Ty. and
outputs "Bayesianized" LoRA adapter (B', A'), where B' U diag(d), and A' becomes distribution ((A)) IILE())JE( N (Aj Nzy), with M = V A, and
Ω = (c./d,..., cu/d]-
Note that the formulation in Eqn. significantly improves memory efficiency during inference, reducing the storage for standard deviation parameters from O(rn) Orr). While alternative parameterization approaches are possible, they must capable of generating the low-rank isotropic Gaussian noises demonstrated in Theorem 4.1. We have selected the current method (implemen- tation) to ensure maximum compatibility with existing codebases [62]. In TFB, we use a single dg shared across all LoRA layers.
3.3 TFB as Variance Maximization
The previous section presents straightforward Bayesianization scheme for predetermined value of 0g. In this section, we describe practical method for determining Tq-
General Bayesianization Framework. Consider an in-distribution "anchor" dataset D, an associated evaluation metric I, and a performance change tolerance e. TFB determines o, by solving a constrained optimization problem:
max 09 s.t. |(D|B', ( ((D
where I(D)B, A) and ((D)'', M, M,,(),)) Eg [(D
B', M + E) denote the pre- and Bayesianaanniioo performance, respectively. This optimization maximizes the noise scale dg applied to model weights M while ensuring that the resulting performance change remains within an acceptable threshold e.
Anchor Dataset D and Evaluation Metric 1. Our general TFB framework accommodates various choices of anchor dataset D and evaluation metric based on practical requirements. Below, we consider two key scenarios (with N being slightly overloaded in its notation).
For supervised dataset D [nn yn ) The Negative-Log Likelihood (NLL) serves as a natural
evaluation metric in Eqn. (D(() - En= log (e (n). theoretically corresponds to minimizing the KL-regularized variational objective (more details in Sec. 4). The anchor dataset D can be either the original training set used for the LoRA model or an independent calibration dataset, as commonly employed in calibration- accuracy or FI score are also readily applicable. In our experimental setup. to ensure fair comparisons across uncertainty estimation baselines, use the original training data D (maintaining the same information access as baselines) and employ NLL as the evaluation metric. Additional results with accuracy as / can be found in Appendix F.5.
methods [16, 70]. Alternative evaluation metrics such as
By stating d > 0, we assume B has the full column rank r, which usually holds for LLM adaptation.

---

For unsupervised dataset D = {an In One approach is to generate pseudo-labels y using the model before Bayesianization, effectively converting the problem to the supervised case with D = {ær.: Fr | n=1- TFB can also directly incorporate purely unsupervised metrics such as the expected embedding norm lemb (D)0) = EgzDD |||emb 0) |||, where we are only concerned with properties of the representations themselves rather than any supervised signal. Hence our TFB offers substantially more flexibility compared to pure calibration methods, which typically rely a labeled unseen calibration dataset. As a general framework, TFB also supports alternative evaluation metrics and statistical measures specifically designed for unsupervised data.
Performance Change Tolerance c. The selection of performance change tolerance is critical in TFB. While our experiments demonstrate that a fixed relative change rate, i.c., 5/po 0.3% for NLL and t/po 1% for accuracy, where po denotes the pre-Bayssianization performance, can achieve effective uncertainty estimation across various datasets and LoRA checkpoints, an adaptive can further improve the performance of TFB. Users can determine the appropriate value for by considering multiple factors simultaneously, among which the most important is the given LoRA checkpoint. For instance, an overfitted LoRA can typically accommodate larger tolerance c when using the training dataset (or its subset) as the anchor dataset. Additional properties of the data, model, and adaptation tasks can inform the choice of e as well.
3.4 TFB: Final Algorithm
Automatically Determining og. Our final algorithm, presented in Al- gorithm and Fig. 1, employs binary search to determine the optimal og within an initial range I mmadl After identifying the optimal og- we Bayesianize all LoRA layers using this value.
Final TFB Algorithm:
Prediction. For prediction, we average multiple outputs produced by samples from TFB's posterior:
Po (x) = [ey(wo) W)]
Σ
n=1 Plylæ, Wn), Wa ~ o ( 0).
where q( (WWe ()) denotes the variational distribution defined in Eqn. 3, and we set the number of test-time samples to N = 10, following BLoB's protocol [62].
Remark TFB's Efficiency. While TFB with binary search is efficient in terms of both time and memory (Appendix 5.3), and yields near-optimal solution of a. more efficient parallel searching technique can be applied in practice. For instance, in Appendix F.10, we conduct grid search across 8 different 0, values in parallel, construct an approximate function (p) through piecewise linear interpolation of the observed performance, and estimate og (poo e), where po denotes the model's 's performance before TFB.
4 Theoretical Analysis
In this section, we discuss our theoretical analysis, with complete proofs provided in Appendix D. First, we demonstrate that our TFB's Bayesianization scheme, defined in Equations 4, 5, and 7, projects a full-rank isotropic Gaussian distribution onto the low-rank space. We then prove that Eqn. 8 is equivalent to generalized variational inference for LLMs' weights under specific, achievable conditions, offering solid theoretical grounding for TFB.
Assumption 4.1. The evaluation metric (p RR+, + R4, the Negative Log Likelihood (NLL) evaluated on the data distribution DD for the variational standard deviation og:
lD(0,) -. (o P(y| W))-
(10)
Furthermore, we assume lp is locally convex, i.e., 3co > S.f. (a,) O. Yo, E (0,
Remark. The local convexity of the loss function is not unrealistic [43]. For instance, a local minimum Wa of a twice-differentaabe loss function I will imply the local convexity around Wa, as assumed in Laplace Approximation [53, 6].
Theorem 4.1 (Equivalent Variational Distribution of the Full Weight W in TFB), With the pre- trained weight matrix Wo E x the low-rank weight update matrix (B' RRxr A' RR transformed from the given matrices ( (B,A) A} following Eqn. 4 and 5, suppose that the variational
'While traditional search algorithms require monotonicity within the search range to guarantee optimal solutions, empirically a optr-aptimal a, sufficient for effective uncertainty estimation.
5

---

distribution of A' is Gaussian ( = -II,, Mig. where M = [M = A.y) E R"*"" is its mean and S2 [2n E Rx is the standard deviation calculated as in Eqn. 7. The equivalent variational distribution ( W) (.) defined on the full weight W is
(Wre(())lg)) = ~N(vec Mlg. Eg),
where
My voc(Wo + B'M).
(I1)
Σg og In @
0m-r
Theorem 4.1 establishes that for any given dg. our algorithm for regrouping B, A and computing the standard deviation matrix Z successfully constrains the corresponding full-weight variational distributions to the family of low-rank isotropic Gaussian distributions. This lays the foundation for the equivalence between our TFB and generalized variational inference approximate the posterior distribution of LLM parameters (details in Theorem 4.2).
While alternative families of Gaussian distributions parameterized by single scale Og possible, our empirical results demonstrate that our approach achieves superior performance (Sec. 5.4).
Theorem 4.2 (TFB as Generalized Variational Inference). Suppose the evaluation metric ()(og)) defined following Assumption 4. I is locally convex within the range of 0g E [0, c)) Suppose the approximate distribution W given og is defined following Theorem 4.1. Suppose we have the prior distribution P(yec(W) = N [vee ()) Pp, Ep), where Mp Mr = vee (Wo B' B'M), and Ep = I with op > ep. Then for VA > 0, ΣE, s.t. the following two optimization problems (i) Generalized Variational Inference [7, 22, 30, 34]
min
(o) + XKLLg (g P(W)),
(12)
dg
and (ii) Training-Free Free Bayesianization (TFB)
max 0q
(13)
s.1.
lT)(ag)
ST,
are equivalent. i.e., the two optimization problems have the same optimal solution, where the regularization coefficient of the KL-divergence.
This theorem provides the primary theoretical foundation for TFB. It demonstrates that under specific conditions- namely, local convexity within [0, <p) and prior standard deviation Sp > 60 maximizing the scale og of the standard deviation matrix is equivalent to generalized variational inference [34], which approximates the posterior distribution of L.LM parameters. Notably, when À = 1/|p) is set to the reciprocal of the dataset size, generalized variational inference reduces to variational inference.
Remark. TFB maintains theoretical soundness (through its equivalence to variational optimization) while offering practical simplicity, as it eliminates the need to explicitly specify the prior distribution standard deviation v. The condition Gp > is naturally satisfied by common choices such as the standard normal distribution (σp = 1) or uniform distribution (op - +00).
5
Experiments
We evaluate TFB through comprehensive experiments.
5.1
Settings
Models, Datasets, and Evaluation. We use the latest open-source Neta-Llama- 3. 1-88 primary LLM backbone while also providing additional results on other recent LLM architec- tures in Sec. 5.5, including 11 2- Neta-Llama 1.lama-3-88 3-88, and Mistral-7B-vo from the Llama [12] and Mistral [26] families.
1.------------....... as our
For in-distribution experiments, we evaluate model performance on six commonsense reasoning tasks: Winogrande-Sminn Small (WG-S) and Benggrande-Mediun (WG-M) [50], ARC-Challenge (ARC-C)
and ARC-Easy (ARC-E) [11], Open Book Question Answering (OBQA) [42], and BoolQ [10].
Furthermore, we use models fine-tuned on OBQA [42] to evaluate their generalization ability on out- of-distribution datasets: college-level chemistry (Chem) and physics (Phy) subsets of MMLU [20].
Label spaces and prompt templates are detailed in Appendix E.

---

Table 1: Performance of different methods applied to LoRA on Llama3.1-8B pre-trained weights, where Accuracy (ACC) and Expected Calibration Error (ECE) are reported in percentages. "TF?" denotes whether a method is Training-Fres -Feee. The evaluation is done across six common-sense reasoning tasks with a shared hyper-paramettr setting after fine-tuning of 5 epochs. We use N = 10 samples during inference in all sampling- methods including BLoB [62] and TFB. Rows with shading indicate training-free Bayesianization methods that use a pre-trained LoRA as their mean. Cells highlighted in BLUE indicate improved performance achieved by TFB compared to the weight mean. "f" and "4" indicate that higher and lower values are preferred, respectively. Boldface and underlining denote the best and the second-best performance, respectively.
Out-of-Distribution Datasets (OBQA-X)
Metric Method
TF
In-Distribation Dutusets
Small Shijt
Lorge Shift
WG-5
ARC-C
ARC-E
WG-M
OBQA
BoolQ
ARC-C
ARC-E
Clemn
Pby
MCD
X
28.03.041
81.641134
91.37,008
13.18
87.20 10
89.931106
81.42,11
17.27
47,92
46.53 k.0
ENS
X
78.82265
82,55
91.84:406
83.992074
87.37
90.50
79.62:951
86.56:000
49.65+132
44 14.44
LAP
76.05=mm
79.95e04:
90.73=t0
52,83+aas
87.90=220
89.364902
81.08e120
87:21=13)
18.26e393
18:18:150
MoelsCLoRA
69 69.20eeus
78.38
90.79=000
74.79
84.13-6.11
89.174000
79.63-981
86.58-202
50,00=104
42.00=241
DLoB
76.45 X
82.32
91.14
82.01
87.57
89.65
79.75
87.13:m0m
42.71
44.79::m4
ACC (1)
MLB
77 57.87
81. 15
67.66
82.30.05)
87.90
41.48-241
86.83
45.833005
17.36:117
TFB (Ours)
77 77
82.53
82.53,995
88.53
89.75
79.76.111
85.52.n
44.333-105
37.00..16
MAP
76. 76.90=m07
81.08+241
91.61
82.59
85.73e0.19
90.09
79.98
86.58=077
43.405405
38.5 38.54-140
TFB (Ours)
1674
$2,80
91.39,007
12.64.055
86.00...6
89.96.14
30.61
86.30..o
45.33
35.67.4m
BLoB- TFB (Ours)
77.72.001
82.60
91.64
83.92
88.00.cam
19.86
82.06.111
88.54
39.93
39
77.81a0
83.33:a1
91.76anu8
83.81a00
87.80
16
82.93-154
67.64:551
39.674733
37.55
MCD
X
16.13
13.69a1.1
6.73
13.05aus
9.76
7.95
13:63=11
9:27-am
30.91
33.08:140
ENS
14.72=m17
13.45eL1
6.59-t45
11.17a092
8.17
7.35+035
11.37=18
7:21-11
18.92es33
26. 26.80-123
LAP
BP
4.18:am
9,2611es
5.27
3,50
8.93
um
7,83214
7.80:11
14,49.am
13.17:224
MRR
X
18.29.0m
12.22a015
7:23
15.972045
9.792am
7:09
10.652051
18
10.
9,934622
5,4+un
2,70-13:
4.28
2.91=am
2.58+025
5.61+a4
2.48-640
16.676017
12.78=418
ECE (L)
MLE
17.02=046
16.35
7,00=t.5
13.83aas5
9,77=081
8.69
14,452219
10.78:000
52.164250
38.41-44
IFB (Our)
:0:9
p1.63
5.14::14
1000
7.20:041
7.39
6,54
5.69
14.63
19:68
MAP
2877
15.77
6.62
14.26
12.19
8.40
16.46:044
11 11:36
34,79
38. 18
TFB (Ours)
gows
11,27g
5.76..0
10.97.114
9,70gam
6.862
18:25:44
9:22.001
27.21
15
X
15
43201s
12
9.3721m
6.44
6.26,g0
11:22,41
14
6. 6.34.977
26.655336
25, 40
I7B (Ours)
8 16snus
6,484035
2.44:130
3.833804
2.67=au8
3,10a009
6.69
3.61
18.45e635
20.53
X
1.83
1.99
0.45.co cmm
0.64
0.62
149.mm
1,03
2.02..15
ENS
X
D. 0.75
0.80
0.38
0.55gn22
0.45:am
0:420005
0.72
0.44.0m
1,40a011
1.50ga15
LAP
BP
1.56
1188100
1.04ccm
1000
0.94.000
0.43.0m
1,171nm
T. 1111gm
1.27
1,28
MonieCLoRA
0.82=002
0.71:00
0.74g0m
0.55=am
0.36.002
0.6849.9
0.49-200
1444003
1.44
BLoB
X
0.58:z00)
0.5
0.23-am
0.43aux
0.34
0.26:
0.56eacs
0.35:
1.34
1.35
NLL.)) MLE
0.88
1.20ssus:
0.46=tas
0.68
0.61:ame
0.52
1.07aaze
0.72:ttw
2.25
TFB (Ours
1.68
1085
0.33.cm
0.46.0ml
0:42
0.661483
1.39
1.9r
1.12
0.46.cm
0.74
0.79.am
0.52.om1
1.191a01
1.17
2.32
17B (Ours)
0.77:000
0,80sn
0,38
0,574035
0.61::m
0.40:200
0.966924
0.66:2006
1.699015
2. 2.12200e
BLoB-Meuu
0. 0.74
0.73
0.29
0.47
0.37:010
0.32.moz
0.67
1,53 p03
1.54
IFB (Ours)
0.55
0.53
0.23
0.40:am
0.33:x1
0.27:00
0.52
0.35
1.36
1.4eam
To assess uncertainty estimation, we measure Expected Calibration Error (ECE [45]) and Negative Log-Likelihood (NLL) on the test dataset. We also report Accuracy (ACC) to ensure models maintain strong performance. Additional evaluation details are provided in Appendix E.2.
Baselines. We compare TFB with state-of-the the-t-rrttttttt uncertainty estimation methods for LoRA-adapted
LLMs, including ensemble-based method: Deep Ensemble (ENS) [39, 4, 61], variational inference methods: Monte-Card Cante-Carlo Dropout (MCD) [14], Monte Carlo-enhanced LoRA (MonteCLoRA) [51],
Bayesian LoRA by Backprop (BLoB) [62], and post-training method: Laplace-LoRA (LAP) [67].
For reference, we also include two standard Eirameter-Efficieer Fine-Tuning (PEFT) baselines: Maximum Likelihood Estimation (MLE) [23] and Maximum A Posteriori (MAP). All baselines are implemented following the protocols established in BLoB, detailed in Appendix E.4.
TFB Implementation. TFB can be directly applied to trained LoRA adapters without additional training. As indicated by the "TF?" column in Table 1, TFB is Training-Fres Free and requires only LLM inference (V), while the other methods need full retraining (X) or gradient estimation with Backpropagation (BP). We evaluate TFB on three off-the-shell LoRA checkpoints: MLE, MAP, and the mean component of BLoB (obtained by discarding BLoB's 's standard deviation matrix Z) More details are included in Appendix E.4.
7

---

Table 2: A comparison of running time and maximum GPU memory cost between TFB and BLoB during the process of Bayesianizationn. The experiments are conducted on a single NVIDIA A100 GPU. The subscripts in the table calculate the relative cost of method compared to that of LoRA, a non-Bayesian baseline method. RED and GREEN represent worse and better efficiency, respectivley. Note that varying batch sizes do not impact the performance of TFB, Algorithm is independent of gradient and batch size
Method
Butch Size
WC-S
ARC-C
WG-M
Time
(MB)
Time
Mom. (MB)
Man. (MB)
Mess (MB)
Mem.
LoKA
338
12,894
652
19,762
1,238
18,640
1,339
13.164
2,692
17,206
6,489
29,450
BLoB
571 (1.0))
13,194 iuibo
685
21,736
1,360 cLin 20,700
1,476
(1.t0)
13,194 (ome
3,257 (1200) 18,046 (it
7,251
6.12
30,578 (104)
TPB (Qurs)
1,203 Sai
10,372 misi)
1,257
11 11,066
1,246
11,200
1,237
10,344 a.7k)
1,238
w450) 10,376
1,522
16,340 nsia
TFB (Ouna
622 (1960)
10,566 susin
751
15, 15,236 m.
702
12,596
634
10,662 mare)
642
(o1 12,116
1,015 0.00
22,14 (055
TFB (Ouns
12
446 (13t0 12,064
599
18,204
540
0--00 14,310
441
(032)) 11,370 (u8
487 (uRo 13,410.87700 908 0.13
25,220 (a50
5.2 TFB Improves Accuracy and Uncertainty Estimation across Distributional Shifts
Table I shows results on comprehensive metrics for various methods applied to LoRA on Llama3.1-81 pre-trained weights. More empirical results on Llama2-7B can be found Appendix F.10.
In-Distribution Results. The addition of TFB maintains competitive accuracy while substantially improving model calibration across in-distribution datasets. For ECE, TFB yields notable improve- ments when applied to different base methods: MLE+TFB reduces ECE to 5. 5.14% on ARC-E (from 7.00%); similarly MAP+TFB and BLo reduce ECE to 9. 9.70% on OBQA (from 12.19%) and 3.83% on WG-M (from 9.37%), respectively. For NLL, TFB consistently produces better- calibrated predictions, with ARC-E (from 0 0.29), 0.33 on OBQA (from 0.37), and 0.27 on BoolQ (from 0.32). These improve- ments in both ECE and NLL demonstrate TFB's effecctiveness in enhancing model calibration while preserving accuracy on in-distribution tasks.
achieving strong performance across datasets: 0.23 on
Out-of-Distriuution Results. For diittiidiitiittiion datasets, which represent more challenging evaluation scenario, TFB continues to show benefits, though the performance gaps are generally smaller. In both Small Shift and Large Shift scenarios, TFB-enhanced methods maintain relatively strong performance, particularly in the Small Shift cases (ARC-C and ARC-B). However, there's a noticeable performance drop in the Large Shift scenarios (Chem and Phy). which is expected given the significant domain difference. Even in these challenging cases, TFB- TFB-enhanced methods tend to maintain better calibration (lower ECE scores) compared to their base counterparts, suggesting improved reliability in out-of-distributte settings.
5.3 Computational Efficiency of TFB
We compare the computational efficiency of TFB and BLoB during the process of Bayesianization [62] in Table 2. We also report the computational cost of the standard LoRA fine-tuming as reference. All three methods are evaluated on the configurations detailed in Appendix E.4. For LoRA and BLoB, the evaluation of running time and maximum GPU memory is based on fine-tuning for 5 epochs. TFB uses a fixed number of 500 training examples to search for og across all datasets, and performs binary search for at most 5 rounds (sequentially).
As shown in the table, TFB can be slower on small datasets (e.g., WG-S with ~600 samples, nearly 3x slower than BLoB under the same batch size). However, on larger datasets (c.g., BoolQ with 1000000 samples), TFB is up 5x faster while using only half the GPU memory. Since TFB avoids gradient estimation, memory use is substantially reduced, allowing larger batch sizes; increasing from 4 to 12 already yields lower time and memory than BLoB on most datasets. Remarkably, TFB is even more efficient than standard LoRA fine-tuming thanks to its training-free free nature. Importantly, our efficiency results cover the entire process of searching a, whereas baselines report only successful runs and exclude hyperparameter luning costs, biasing the comparison in their favor. Despite this, TFB still achieves superior time and memory efficiency, highlighting the advantages of its training-free approach and flexibility to trade off speed and memory under resource constraints.
8

---

Table 3: Performance of TFB with different variational distribution families applied to BLoB- Mean on Llama3.1-88 pre-trained weights. FR: Full-rank isotropic Gaussian noises are applied to AW; C-STD: Standard deviation matrix S [$2. 0gg is constant. The evaluation protocol strietly follows Table I. "Rk.": Average ranking of each method when compared to all other approaches on in-distribution datasets. "1" and "J" indicate that higher and lower values are preferred, respectively. Boldface and underlining denote the best and the second-best performance, respectively.
Out-of-DDstribution Datasets (OBQA-H) -X)
In-Distribation Dalasets
Metric Method
Small Shift
Large Shift
WG-S
ARC-C
ARC-E
WG-M
OBQA
BuulQ
RL. (1)
ARC-C
ARC-E
Chemm
Phy
BLoB-Mean
77.22
91662a0
83.92-04
98.00-1::
86
82.06-1.1
88.54:4.31
30.9345.20
39 39.93+-02
ACC (1)
+ 17B (C-S (C-STD)
17B (FRR)
76.35=cre
75.57
83, 33,20+0.33
20:200
91,334000
51.383
81.79
82 19
58
88.73-44
28.20
89 30.65
89 4946:2
3.00
2,83
81,33
81.73-3m
88.06-005
88.18420
43.00:141
42.00
41.331544
39 39.33
17D (Final)
77.81
83.33
91.76:n.4
83.81enm
$7,80
90.11=aa
1.67
82,93-154
87.64
39.67
37.33
BLoB-
15/43a0us
12.4|+13
9.37etus
6.Meaus
6.26=229
11,22
6.34
26.654:06
25 25.40+.aa
BCE (L)
+ 17B (FRR)
+ TFR (C-STTD)
1042262
9.23_.00
7,452000
6.98
2.0
2.94.om7
1.86in
4.362nss
3/70
3.17,001
3.62=211
2.82
2.67
1,83
7.19: 10
4.89
2.76:436
3.20-1m
17.78
18.27
19.14:4m
19.45
TFB (Final)
8.16
6.48
1636
2.44
2.67
3.10
1.50
6.09 143
18:451675
20:53
BLaB-Mean
0.74
0.73
0.29
0.47
0.37
0.32,303
3.67
0.67asm
0.39.am
1,53
1.54
NLL ())
+ 178 (C-STD)
(FR)
0.57-4m
0.60-801
0.51:4uc
0.53
0.23
0.22:xum
0.43
0.43
0.33
0.33:am
0.27a381
0.26-181
1.33
0.57a504
0.56=220
0.34
0.33-eac
1.34:00
1.34.um
1.41:00
1,42
+ 170 (Final)
0.55:140
0.53=0m
0.23
0.40
0.33
0.27e301
1.50
0.52=aae
0.35
1.36211
1.468011
5.4
TFB Beyond the Low-Rank Isotropic Gaussians
In this section, consider two simple TFB variants with other families of Gaussians for modeling the variational distributions of W: (i) Full-Rank Isotropic Gaussian (FR, Σg = o2 and (ii) Constant Low-Rank Standard Deviation (C-STD, S = [a, = og). Similar to our final TFB, both distributions controlled by a single og parameter and fit the maximal variance search in Eqn. 8. For fair comparison, we adopt the same optimal o search protocol as described in Sec. 5. Table shows the performances of TFB and its variants applied to the mean of BLoB (more in Table 11 of Appendix F.6).
These results show that our final TFB outperforms both variants FR and C-STD across multiple metries on in- distribution datasets, with notable improvements in cal- ibration (ECE reduced by up to 15.77%) and accuracy (e.g.. 77.81% on WG-S). While C-STD shows better NLL scores, the improvement comes at the cost of signifi- cantly degraded overall performance partommaaanpptiicllaaaryl in ac- curacy, where it performs the worst-makinng it imprac- tical for real-world applications. Although our final TFB maintains strong performance on datasets with smaller distributional shifts, its advantages diminish on datasets with larger shifts in the domains of Physics and Chemistry.
Table 4: Performance of different LLM backbones on the combined dataset.
Method
ACC
ECE
NLL
Lluma27B
81.41:444
4.50=a337
0.43:am
+ TFB (Ours)
81 32snn
1.24::33
0.43:am
Lhuma-8B
86.93
4,28
0.34:am
(Ours)
86
1.64:nsi
0.34
Llbma3 1-8B
86. 70.70
4.74.008
0.35.4
+ TFB (Ours)
45.45
1.05 1006
0.34:4
Misral-78-v0.3
5.05
0.35
TFB (Ours)
86.64
1.68
0.33
Advantages of Final TFB's Variational Low-Rank Isotropic Gaussians. Compared to TFB (FR) and TFB (C-STD), TFB (Final) offers additional advantages. It is computationally more efficient than FR with noise complexity of (rr) versus O (mn). Furthermore, unlike C-STD whose variational distributions vary with different but equivalent LoRA matrix pairs (see Appendix F66 for details), TFB (Final) produces consistent Bayesianization for all equivalent LoRAs satisfying BA = AW.
5.5 TFB Beyond the Llama3.1-88 Backhone
We conduct comprehensive experiments across multiple LLM backbones to validate our approach. Our experiments span several models from the Llama family [55, 12], including 1lama-2- ---aa-22-bbbi hf, Meta-Llama-33 -3--88 and Meta-Llam -3. 1-8. While we initially considered L1ama-3. 2BB we ultimately discarded its results due poor adaptation performance with the smaller model architecture. We also extend our analysis to include Mistral-7B-vol [26].
Following commonsense-1 1inee-1700 17k [24, 60], we combine the 6 reasoning sub-tasks from the main experiments with one shared label space (Combined) and train the base LoRA adapters with MLE. Break-down statistics of each sub-dataset are available in Appendix F.7. Table 4 shows the results, demonstrating TFB's effectiveness: it dramatically reduces ECE across all models (e.g., from 4.74% to 1.05% for Llama3, 1-8B) while maintaining strong ACC and NLL scores.

---

5.6 TFB Beyond the Naive LoRA
Our proposed TFB is general, compatible with various LoRA-based methods that use different initialization strategies [41], parameter sharing schemes [35], and optimization approaches [68]-
Table 5: Performance of different LoRA-like PEFTs on the combined dataset.
Method
ACC
ECE
NLL
We evaluate TFB on two representative LoRA vari- ants (see Appendix E.8 for details):
LoRA
86.70100s
4.74
0.35
TFB (Ours)
86,45
1.05
0.34
VeRA
84.93+0.00
5.1 3.1.5
0.39+00:
VeRA [35]: Uses shared low-rank matrices B and A across layers, with the layer-specific trainable scalar vector d and the bias vector b. PiSSA [41]: Employs an alternative initializa- tion while maintaining LoRA's training process.
TFB (Ours)
84,28
1,44:0.44
0,38
PISSA
86.831051
4.261.014
0.35
TFB (Ours)
86.61 #043
1.17=0.22
0.33
Table 5 shows the results, demonstrating TFB's broad applicability: it substantially reduces ECE across all LoRA-like PEFT methods (e.g., from 5.11 to 1.44 for VeRA). Importantly, it maintains strong ACC and NLL with minimal performance degradation, validating the effectiveness of low-rank isotropic Gaussian distributions for variational inference of LLMs.
5.7 Improving Inference-Tine Time Efficiency of TFB
Inspired by Harrison et al. [19], in this section, we study the compactibility of TFB and Last-Layer Bayesianiza- tion to speed up inference. Specifi- cally, employ the same standard deviation of as identified through the full-model TFB, but apply the Bayesianization to the last layer only. By limiting inference to sampling from the variational distribution of the last layer, the outputs of earlier layers can be reused, signifi- cantly improving inference speed. Although restricting Bayesianization the last layer sacrifices some richness of the variational distribution family, this approach allows for higher number of posterior samples, resulting in more accurate approximations.
Table 6: Performance of Last-Layer TFB (LL TFB) applied to the combined dataset.
Method
#Sample (N)
ACC
ECE ())
NLL ())
Time (s)
MLE
86.70.o08
4.74
0.35
118
+ TFB
10
86.45
1.05
0.34
1,114
TFB
86.53.o.12
1.14
0.34
LL TFB
100
86.75-9.4
0.92:6.10
0.33
924
Table shows the results. When only N = 10 samples are used, the uncertainty estimation of last-layer Bayesianization performs worse than that of full-model Bayesianization. Nevertheless, the improved posterior estimation with N = 100 samples enables last-layer Bayesianization to achieve better performance. For comprehensive analysis of its performance, please refer Appendix F.9.
Conclusion
In this paper, we introduce Training-Free Bayesianization (TFB), novel framework that transforms trained LoRA adapters into Bayesian ones without additional training. By systematically searching for the maximally acceptable variance in the weight posterior within family of low-rank isotropic Gaussian distributions, TFB provides a practical solution to uncertainty estimation in LLMs. Our theoretical analysis shows that TFB's variance maximization process is equivalent to generalized variational inference under mild conditions. Our empirical results verify its superior performance across various settings and model configurations. Our framework's simplicity and effectiveness, requiring only an anchor dataset for search, makes it widely applicable across different domains. As LLMs continue to evolve, TFB represents significant step toward more reliable and uncertainty- aware Al systems, paving the way for future research in adaptive and trustworthy machine learning. For Limitations, please refer to Appendix B.
Acknowledgement
We thank all reviewers, AC, and SAC for their valuable comments. HW is supported by Amazon Faculty Research Award, Microsoft AT & Society Fellowship, NSF CAREER Award IIS-2340125, NIH grant R0I R01CA297832, and NSF grant IIS-2127918.
10

---

References
[1] Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, I.., Ghavamzadeh, M., Fieguth,
P., Cao, X., Khosravi, A., Acharya, U. R., et al. review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion, 76:243-299 292,229,00 2021.
[2] Achiam, J., Adler, S., Agarwal, S., Ahmad, I.., Akkaya, I., Aleman, F. L.., Almeida, D., Altenschmidr, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[3] Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, 7., et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[4] Balabanov, O. and Linander, H. Uncertainty quantification in fine-tuned Ilms using lora ensembles. arXiv preprint arXiv:2402. 0202402..22260 12264, 2024.
[5] Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et Pythia: suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
[6] Bishop, C. M. Pattern recognition and machine learning, Springer google schola, 2:1 122-1128. 2006.
[7] Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. In International conference machine learning, pp. 1613-1622. PMLR, 2015.
[8] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:7897-19 2020.
[9] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
[10] Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T.. Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Burstein, J., Doran, C.. and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2924 2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N199 1300. URL htttps://acaanaaaaaat /N19-1300.
org
[11] Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018
[12] Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., al. The llama herd of models. arXiv preprint arXiv:2407.21783 2024.
[13] Fan, X., Zhang, S., Chen, B., and Zhou, M. Bayesian attention modules. Advances in Neural Information Processing Systems, 33: 33:16362-16377, 2020.
[14) Gal, Y. and Ghahramani, Z. Dropout a bayesian approximation: Representing model uncertainty in deep learning. international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
[15] Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 15:1511-1589, 2023.
[16] Guo, C., Pleiss, G., Sun, and Weinberger, K. Q. On calibration of modern neural networks. In International conference machine learning, pp. 1321-1330. PMLR, 2017.

---

[17] Gupta, N., Narasimhan, H., Jitkrittum, W., Rawat, A. S., Menon, A. K., and Kumar, S. Language model cascades: Token-level uncertainty and beyond. arXiv preprint arXiv:2404.10136 36, 2024.
[18] Gustafsson, K., Danelljan, M., and Schon, T. B. Evaluating scalable bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 318-319, 2020.
[19] Harrison, J., Willes, J., and Snoek, J. Variational bayesian last layers. In The Twelfth Interna- lional Conference on Learning Representations, 2024.
[20] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Mea- suring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[21] Hermández-Lobato, J. M. and Adams, R. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International conference on machine learning. pp. 1861-1869.
PMLR, 2015.
[22] Higgins, I., Matthey, L., Pal, A., Burgess, C. P., Glorot, X., Botvinick, M. M., Mohamed, S., and Lerchner, A. vat:-vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
[23] Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: attpps//oppereviiet
[24] Hu, Z., Lan, Y., Wang, L., Xu, W., Lim, E.PP., Lee, R. K.-W., Bing, L., and Poria, S. Llm- adapters: An adapter family for parameter-efficien fine-tuning of large language models. arXiv preprint 2rXiv:2300.009333 2023.
[25] Hüllermeier, E. and Waegeman, W. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 1 11033)4577-066 2021.
[26] Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, d. I., Bressand, F. Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b, arXiv preprint arXiv:2310.0683) 2023.
[27] Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield- Dodds, Z., DasSarma, N., Tran-Johnson. E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L.,, Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, Language models (mostly) know what they know, 2022.
[28] Kapoor, S., Gruver, N., Roberts, M., Collins, K., Pal, A., Bhatt, U., Weller, A., Dooley, S., Goldblum, M., and Wilson, A. G. Large language models must be taught to know what they don't know, arXiv preprint arXiv:2406. 243399999,,, 2024.
[29] Karush, W. Minima of functions of several variables with inequalities as side constraints. M. Sc. Dissertation. Dept. of Mathematccs, Univ. of Chicago, 1939.
[30] Khan, M., Niclsen, D., Tangkaratt, V., Lin, W., Gal, Y., and Srivastava, A. Fast and scalable bayesian deep learning by weight-perturbation in adam. In International conference machine learning, pp. 2611 2611-2660 1620. PMLR, 2018.
[31] Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[32] Kjeldsen, T. H. contextualized historical analysis of the kuhn-tucker theorem in nonlinear programming: the impact of world war ii. Historia mathematica, 27(4):331-36 2000.
[33] Klema, V. and Laub, The singular value decomposition: Its computation and some applica- tions. IEEE Transactions automatic control, 1642):164-176, 1980.
12

---

[34] Knoblauch, J., Jewson, J., and Damoulas, T. Generalized variational inference: Three arguments for deriving new posteriors. arXiv preprint
2019.
[35] Kopiczko, D.J., Blankevoort, T., and Asano, Y. M. Vera: Vector-based random matrix adaptation. arXiv preprini arXiv:2310.11445. 2023.
[36] Krishnan, R., Esposito, P., and Subedar, M. Bayesian-torch: Bayesian neural network layers for uncertainty estimation. https: http://gttttbttt/g com/ IntelLabs aattellaas/bbyesiann bbwpsbaaaayesrrtittie January wwtphhuu...ommntteeeeeeeeeeeeeee
2022. URL https::///ti.oooooo org http:/11..222/eeo//ee 5908307.
[37] Kuhn, H. W. and Tucker, A. W. Nonlinear programming. In Berkeley Symposium Math- ematical Statistics and Probability, pp. 481 482- Berkeley: University of California Press, 1951.
[38] Kuhn, L.,, Gal, Y., and Farquhar, S. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023.
[39] Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[40] Lin, S., Hilton, J., and Evans, O. Teaching models to express their uncertainty in words. Trunsactions on Machine Learning Research, 2022.
[41] Meng, F., Wang, Z., and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint 7xXi:2444.022445 2024.
[42] Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 21.22381-299 Brussels, Belgium, October-Novemtn 2018. Association for Computational Linguistics. doi: 10. 10.1188////D1.... 1260. URL battps://aclantiiie .org/Die 1260.
[43] Milne. T. Piecewise strong convexity of neural networks. In Wallach, H. Larochelle, H., Beygelzimer, A., d'Alché-Buc, F. Fox, E., and Garnett, R. (eds.), Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: http:ps:////ceeeeepeeeee www.eediiinnnnnnnns....ee wwwrips.cc/paper.... wc/paper_fileeeeeeeeeeeeeeeeeeee wtt.fffies/aape//1////leeee $118:0009003ddd11119000000662229 Paper pdf.
[44] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint a202::2202. 2283, 2022.
[45] Naeini, M. P., Cooper, G., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
[46] Neal, R. M. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
[47] Nikitin, A., Kossen, J., Gal, Y., and Marttinen, P. Kernel language entropy: Fine-grained uncertainty quantification for Ilms from semantic similarities. arXiv preprint arXiv:4405.20003, 2024.
[48] OpenAl. Introducing chatgpt. [online]. available: 2022.
chatt.
[49] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 18):9, 2019.
[50] Sakaguchi, K., Bras, R. L.,, Bhagavatula, C., and Choi, Y. Winogrande: adversarial winograd schema challenge at scale. Commun, ACM, 6409):99-106, aug 2021. ISSN 0001-0782, doi: 10.1145/3474381. URL https:///oi...... org/ 10. 1145/3474381.
13

---

[51] Sengupta, A., Seth, V., Pathak, A., Verma, A., Raman, N., Gopalakrishnan. S., Chatterjee, N., and Chakraborty, Robust and efficient fine-tuning of Ilms with bayesian reparameterizztioo of low-rank adaptation. arXiv preprint arXiv:241 1.04358, 2024.
[52] Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H.,, Finn, C., and Manning, C. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language mod- els fine-tuned with human feedback. In Bouamor, H., Pino, and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433-5442, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.118/// emnlp-main.330, URL https: http///aaaaaatlttoaaeeeee org/2023...22 emnlp-main 330.
[53] Tierney, L. and Kadane, J. B. Accurate approximations for posterior moments and marginal densities, Journal of the american statistical association, 8 (3333):822-86 1986.
[54] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[55] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 230.00080,,,8 2023.
[56] Wang, H. and Yeung. D.-Y. Towards bayesian deep learning: framework and some existing methods. IEEE Transactions on Knowledge and Data Engineering, 28(12)3335-3408, 2016.
[57] Wang, H., Shi, X., and Yeung, D.-Y. Paturee- networks: A class of probabilistic neural networks. Advances in neural information processing systems, 29, 2016.
[58] Wang, H., Tan, S., Hong, Z., Zhang, D., and Wang. H. Variational language concepts for interpreting foundation language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 8645- 8671, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: /tt/://///22244..nnnn--mmmmmmm// fii.diingg---el.55........ URL https: https:///ccaanmtt //clantthooooooggooooooo.o org/2024. findin gs emnlemmmmmm 505/-
10.18653/v1/2024
[59] Wang, H., Tan, S., and Wang, H. Probabilistic conceptual explainers: trustworthy conceptual explanations for vision foundation models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24 24. JMLR.org, 2024.
[60] Wang, H., Xiao, Z., Li, Y., Wang, S., Chen, G., and Chen, Y. Milora: Harnessing minor singular components for parameter-efficien Ilm finetuning. arXiv preprint arXiv:2406.09044, 2024.
[61] Wang, X., Aitchison, L., and Rudolph, M. Lora ensembles for large language model fine-tuning, 2023.
[62] Wang, Y., Shi, H., Han, L., Metaxas, D., and Wang, H. Blob: Bayesian low-rank adaptation by backpropagation for large language models. arXiv preprint arXiv:2406. 1675, 2024.
[63] Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01 105.01652, 2021.
[64] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. arXiv preprint 0xXiv:2206.07682, 2022.
[65] Xiong, M., Hu, 7.,, Lu, X., Li, Y., J., He, an empirical evaluation of confidence elicitation in lmm.. arXiv preprint arXiv:2306. 13063, 2023.
and Hooi, B. Can lmms express their uncertainty?
(66] Yadkori, A., Kuzborskij, I., György, A., and Szepesvári, To believe or not to believe your Ilm. arXiv preprint arXiv:2406. 24X::24000022443, 2024.
14

---

[67] Yang. A. X., Robeyns, M., Wang, X., and Aitchison, L. Bayesian low-rank adaptation for large language models. arXiv preprint arXiv:2308. 1..3008.31111 13111, 2023.
[68] Zhang, Q. Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, arXiv preprini arXiv:2303.10512, 2023.
[69] Zhang, S., Fan, X., Chen, B., and Zhou, M. Bayesian attention belief networks. International Conference Machine Learning, pp. 12413-1222 12426. PMLR, 2021.
[70] Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp. 12697-12700 PMLR, 2021.
15

---

Appendix
In Appendix A, we present the full algorithmii description of TFB. In Appendix B, we present the limitations of TFB. Next, in Appendix C, we present more detailed introduction to recent advances of Bayesian low-Rank Adaptation. In Appendix D, we provide detailed proof' for all theorems presented in the main paper. In Appendix E,, we describe our experimental methodology. Finally, in Appendix F, we present additional empirical results, including:
visual study demonstrating TFB functions general Bayesian Neural Network (Ap- pendix F.I),
test-time sample size analysis (Appendix F.2),
anchor dataset size analysis (Appendix F.3),
TFB with unlabeled test data the anchor data (Appendix F.4),
TFB with non-NLL evaluation metrics (Appendix F.5),
TFB with other ainle-parameter variational distribution families (Appendix F.6),
TFB with other LLM backbones than Llama3. 1-8B (Appendix F.7),
with other PEFT methods than LoRA (Appendix F.8),
improving inference-time efficiency of TFB with last-layer Bayesianization (Appendix F.9),
and
the full results on a widely used LLM architecture Llama2-7B (Appendix F.10).
A TFB: Algorithm
Algorithm Training-Fri Free Bayesianization (TFB)
input D: Anchor Dataset;
BB (B:A): Low-Rank Component;
1: Model Evaluation Metric;
e: Performance Change Tolerance;
[Tamin Amaal: search range of aq.
1: Evaluate the original performance: po ())
2: Singular Value Decomposition on B:
U, diag(d). V + SVD(B).
Eqn.
3: Get an equivalent pair of the low-rank component:
B' +U U diag(d); A' < VTA. A.
Eqn. 5.
4: while 0g not converged do
5: Bg + (a mmuu/2.
6: Calculate the standard deviation matrix Ω for 4':
2i = 0o/d..
Eqn.
7:
Evaluate the performance:
p 1(D B', A' 22)
8:
if Pol then
9:
Amin f 0q-
10: else
11: o Umax ogg
12:
end if
13:
end while
output (B', A', R): Bayesianized Low-Rank Adapter.
B Broader Impact and Limitations
Broader Impact. This research advances methods for making large language models more trustwor- thy and reliable through improved uncertainty estimation. While we focus on language models, the fundamental principles of our framework can enhance uncertainty quantification across the broader machine learning field. This wider applicability creates opportunities for improving model reliability and safety across diverse applications. To the best of our knowledge, there are no ethical or other concerns that need to be addressed.
16

---

Limitations. TFB is subject to several limitations. First, our approach relies on the availability of an anchor dataset for determining search range and stopping criteria. Although this dataset doesn't t require supervision or prior use in LoRA training. its quality and representativeness could impact the effectiveness of uncertainty estimation. Second, by constraining the family of full-weight posteriors to low-rank isotropic Gaussian distributions, TFB may not capture more complex uncertainty patterns that could be present the data. At first blush, this seems to imply a trade-off between computational efficiency and model expressiveness. However, in practice, the trade-off may not be necessary as TFB can often enjoy both computational efficiency and model expressiveness, getting the best of both worlds. Given TFB's proven effectiveness and inherent simplicity, recommend implementing TFB as the initial approach when developing reliable LLMs with existing LoRA adapters. If TFB fails to meet specific requirements, practitioners can then consider alternative expensive Bayesian methods. Finally, while we have demonstrated the effectiveness of TFB in various settings, its performance in more complex generation tasks requires further investigation. Future work could explore extending the framework to handle more sophisticated language generation scenarios and broader applications.
C Related Work
Bayesian Low-Rank Adaptation. The Bayesian framework provides a powerful approach for capturing and estimating uncertainty by defining prior distributions and approximating posterior distributions over the parameter space [46, 21, 14, 56, 18]. However, modeling parameter distri- butions across the entire parameter space during fine-tuning introduces significant computational overhead [13, 69]. To address this challenge, recent research has explored combining Bayesian methods with Carameter-Efficiert Fine-Tuning (PEFT) techniques improve the efficiency of uncer- tainty estimation. Several notable approaches have emerged in this direction. Wang et al. [61] and Balabanov & Linander [4] demonstrate improved performance by training multiple LoRA modules and ensemble their predictions during inference. Taking a different approach, Yang et al. [67] ap- plies a Frroneckerffcttorizee Laplace approximation fine-tuned LoRA parameters. More recently, BLoB [62] advances the field by simultaneously estimating both the mean and covariance of LLM parameters within a single fine-tuning stage, leading substantial performance improvements. Our proposed training-frve free Bayesianization represents sigmificant departure from these existing methods. Unlike approaches that require re-training [14, 61, 4, 62] or rely on continued training and gradient estimation [67]. our method achieves uncertainty estimation without any additional training steps, substantially improving the simplicity and efficiency for Bayesian learning of L.LMs.
D Proof of Theorems
Theorem 4.1 (Equivalent Variational Distribution of the Full Weight W in TFB). With the pre-
trained weight matrix Wo E Rmmx, the low-rank weight update matrix (B' E RRuxrr A' E B transformed from the given matrices (B, A} following Eqn. 4 and 5, suppose that the variational distribution of A' is Gaussian q(A'(0) = II ((Ag)) Mig. 22z)), where M = [Ma = A'yl E R
is its mean and Ω [2y E R is the standard deviation calculated as in Eqn. 7. The equivalent variational distribution q W) (o,)) defined on the full weight matrix W is
g(vec(W) = =N(veo( W) W)lgg Eg),
where
My vec(Wo + +B'M).
(14)
I,
Σg og 8
0 0m-r
Proof. We have the following lemma from BLoB that calculates the covariance matrix of a given low-rank Bayesianization scheme (B,,,S) S2 [62].
Lemma D.I. With the pre-trained weight matrix Wo E R.mx and the low-rank weight update matrix B Boxr suppose that the variational distribution of the other low-rank update matrix A E xx is Gaussian with A(A = {M, s2)) = -Ie N [Aij] P2y), where M = [Ma]] E R and
Ω [Ryy] E grxn are its mean and standard deviation, respectively. The equivalent variational
17

---

distribution defined on the full weight matrix W is given by q(voc(W) 0) (vecc(WW W) My: Eg);
where
My vec WW + BM), Eg = In @ Blddiagveec())))] @ B
(15)
Based on the assumption outlined in Eqn. 4, 5, and 7, we have the following properties about B', M, and Z of TFB:
U Ω -1///, ... .////,
(16)
(17)
where UU = I,
(18)
0m-r
now can be easily shown that the covariance matrix of TFB is:
Dg In B"]]diag(ccc(DD3 [ @ B
(19)
I B I diag(1/ ( B
(20)
In & B (taa((o,/d) 1
(21)
In @ IU diag d) diaglo, d)² diag( d) T U
(22)
I,
o In
(23)
which proves that q(veo(()) is low-rank isotropic Gaussian distribution.
Proposition D.1.1. The function proj(-) defined in Eqn. projects the full-dimensional isotropic Gaussian the low-rank subspace of LORA. can be formulated as
proj(o I Imn) = Ploy mmn)
(24)
I
where
P
In 6
(25)
0 0m-r
Proof. By Theorem 4.1, have
Ir
PI
In
0m-r
(26)
I,
Hence it is trivial to have P In
X
0 0m-r
Theorem 4.2 (TFB as Generalized Variational Inference). Suppose the evaluation metric (o(a))) g) defined following Assumption is locally convex within the range of 0g E 6)) Suppose the approximate distribution W given og is defined following Theorem 4.1. Suppose have the prior distribution P(vec(W)) = (vec () Hpp Ep), where μp = M4 = vec(Wo
M) and Ep I²II with op > Cu- Then for VX > 36, s.t. the following two optimization problems (i) Generalized Variational Inference [7, 22, 30]
min lo(0,) + XKL[g(Wlo,)) || P(W)),
(27)
and (ii) Training-Fre Free Bayesianization (TFB)
max 0g
(28)
s.t.
lo( SE.
are equivalent, i.e., the two optimization problems have the same optimal solution, where the regularization coefficient of the KL-divergence.
18

---

Proof. First we prove the KL divergence term is convex w.r.t. Gg. For two Gaussian distributions and whose covariance matrices En E Raxd and Dp E Rddd are both full-rank, with their means as My E R² and My E R we have their KL-divergence as
KL
log
E E
d tr ( + (Mg Up) E (Fg 0
12p)
(29)
For TFB, avoid unbounded KL divergence, we project the original assumed Gaussian prior into the same low-rank sub-space of the posterior q. We summarize the prior and variational distribution of the posterior as follows:
Ir
O 0m-r) 1)
((voc(W))og) =N vvcc(W) = vec W. + B'M), En o 8
(30)
I,
(())) = (vec(W)) vec W. + B'M), Ep = o o @
Om-r
Substituting Eqn. 30 back into Eqn. 29, we have
o (o) (op))] 2 log 1+ log(og) +
(31)
which is convex w.r.t. Bg and the global minimum of KL achieved when og - -p-
With 0g S cp. the convexity of two terms (KL and lp) holds. Hence we show by the Karush- Kuhn-Tucke theorem [32, 29, 37] that, for any given À there exists E such that the following two optimization problems are equivalent:
1. Minimization of generalized variational inference in the Lagrange-form optimization
min dy
([L]gvee(W))loz) P(vec(W))op) ID(())
(32)
2. The constrained-fomy optimization correpsoiingg to Eqn. 32
min KLL((voc(W)logg) P(vce (a)
(33)
s.t. 1p(a,)) Eτ.
Since the KL term is monotonically decreasing when 0g E [0, ap). and due to the fact that op > Ep. the optimization in Eqn. 33 equivalent our final Training-Fres FFre Bayesianization (TFB):
max 0g s.t. lo(0g) Sτ.
(34)
E
Implementation Details
E.1
Datasets
We provide details of the datasets used in this work, as shown in Table 7. The combined dataset consisting of the six commonsense reasoning tasks contains the label set of "[A, B, C, D, E, True, False]".
Table Dataset Statistics. The size of the Anchor Set D is used in Table 3 and 14.
WG-S
ARC-C
ARC-E
WG-M
OBQA
BoolQ
Combined
Size of Label Space
2
Training
640
1,119
2,251
2.258
4,957
20.652
Size of Anchor Set D
500 (78%)
500 (45%)
500 (22%)
500 (22%)
500 (10%)
500 (5%)
500 (2%)
1,267
3,270
19

---

E.2
Evaluation Metrics
Negative Log-Likelihood (NLL) and Expected Calibration Error (ECE [45]) are key metrics for uncertainty estimation. For a model Pe and test dataset (In, Bo BoJa=1 NLL penalizes models that assign low probabilities to correct labels, and is defined as:
NLL
log Po(yn).
(35)
ECE measures the alignment between model confidence and accuracy by binning predictions:
M Σ m=1
ECE
Bmnl aaco( - conf(Bm)).
(36)
where ace( Bm) = 1/1 EieB,. 1 vs) is the average accuracy and conf Bm = 1/I.... Eien.. P(Ti) is the average confidence in bin Bm We use bin size Bml = 15 throughout
this paper.
E.3 Searched Bg of TFB
We report the searched o using Algorithm in Table 8, where the reported values are the mean values of three random seeds.
Table 8: Searched og of TFB using Algorithm 1.
Base Model
WG-S
ARC-C
ARC-E
WG-M
OBQA
BoolQ
MLE
0.004500
0.003917
0.004500
0.004354
0.003771
0.004063
MAP
0.004500
0.003479
0,003188
0,004208
0.003917
0.005083
BLoB-Mean
0.005813
0.005229
0.005229
0.006250
0.006250
0.005958
E.4
Bayesianization (Training)
Shared Configuration. We report the mean and standard deviation of all experimental results calculated over three random seeds. For all training processes in experiments, we employ the AdamW optimizer. The learning rate follows a linear decay schedule with a warmup ratio of 0.06 and maximum value of 2e 4. The batch size is set to 4, and the maximum sentence length is limited to 300 tokens. The LoRA configuration includes LoRA a = 16 and LoRA = 8. PiSSA [41] follows the exact same configuration the LoRA's. For VeRA [35]. due its characteristic of shared weights across different layers which enables higher-rank setting with the same memory efficiency, set its rank to 256 and learning to 5e 3 for the MILE training the combined dataset.
Baseline Configuration. The baseline configuration mainly follows BLoB [62]. MLE follows the standard LoRA implementation. For MAP, we implement it with weight decay rate of 1e 5. MCD consists of an ensemble of 10 LoRAs with a dropout rate of p = 0.1. For ENS, we fine-tune LoRAs independently and combine them by averaging their logits during evaluation. We implement LAP and apply it to MAP checkpoints. For BBB and BLoB, we use the default settings from Bayesian-Tore library [36], applying Bayesianization only to the A matrix. During training, the number of samples is set to K = for both BBB and BLoB. At test time, we use N 10 samples, matching the configuration of TFB.
TFB Configuration. We randomly sample unlabeled training data points to construct the anchor dataset D = (æx, De (E[M) where Dj is the pseudo-label generated by the given LoRA adapter before Bayesianization; the anchor dataset size M 500 is fixed for all the datasets. We use NLL as the metric / and set the performance change tolerance to 0.3% of relative performance change for all the datasets. To determine the optimal o we perform a 5-step binary search with the initial range of [0.001, 0.015) using Algorithm Similar to the other baseline methods, the final results of TFB are reported averages across three random seeds using oz
20

---

0.1. Alfr (f
dg 6.3. Ellfes 4[
166
T78
21 STD
180
-08
22 - 0.6, AlMue -AYmm 7.46
Oe- 2.0.r 1 13.07
de 1.5, Aill - 41ymm - 160.65
200
100
Figure 2: Uncertainty estimation of TFB on toy regression task. The true cubic function y = 2 (GREEN) and noisy training samples (GRAY) are shown alongside predictions from deterministic MLP baseline (RED) and our TFB (BLUE). The blue shaded region represents #1 standard deviation from TFB predictions with og = 1.0. TFB effectively captures predictive uncertainty, showing low variance in data-dense regions (r E [0, 2)) and increasing uncertainty for inputs outside the training distribution (æ & |-4, 4]), with uncertainty proportional to distance from the training domain.
F Additional Experimental Results
We present additional experimental results this section. Due space constraints (and large table size), we defer several detailed tables to the end of this section rather than presenting them alongside the corresponding analyses.
F.1 TFB as General Bayesian Neural Network: A Visual Study
We demonstrate that TFB functions as a general Bayesian Neural Network (beyond its application to the LoRA adapters of LLMs) by evaluating its uncertainty estimation capabilities on a toy regression dataset, as illustrated in Fig. 2. We follow the regression task framework [21, 57] with the following specifications:
Input Features: {æ, ~ U[-4, 4]]re/20) uniformly sampled from the interval [-4, 4).
True Function: d (plotted in GREEN).
Noisy Labels:
A 1). representing the true function with added Gaussian noise. These data points appear GRAY crosses.
xx + 9 E; where
A two-layer MLP with 16 hidden neurons fit to the sampled dataset, providing a deterministic baseline without uncertainty quantification. The MLE predictions are shown in RED.
MLE Baseline:
MLE Training Configuration: Adam optimizer [31] with learning rate 0.1, trained for 1000 steps until convergence.
TFB Implementation: We Bayesianize the MLE bascline using TFB with full-rank noise, defining the variational distribution as q( W Wo, o,) = II N(W,, WWo.g. o For in- ference, we draw N average prediction as a BLUE curve. The shaded region represents +1 standard deviation of the sampled predictions. We evaluate multiple settings ag E 0.3, 0.6, 1.0. 1.5] and report the average squared predictive difference between TFB and MLE in the figure.
samples from the variational weight distribution and plot
21

---

ACC (ia) vs. #Samples
ECE (1) vs. #Samplcs
NLL V5. #Samplcs
18
16-
14
3.0
10x
32
0.7
0.6
#Sampies
Figure 3: Performance of test-time scaling for TFB with varying numbers of samples N. The shaded region denotes the standard deviation across three random seeds. While larger N generally yields better performance, our choice of N 10 provides a favorable trade-off between accuracy and efficiency at test time.
As Fig. 2 demonstrates. with an appropriately calibrated standard deviation of the approximate posterior (o,, 1.0 this case), TFB effectively captures predictive uncertainty: in regions with dense data sampling (æ [0, 2)). TFB produces low predictive uncertainty; conversely, for inputs outside the training domain (a: é [-4, 4)), the predictive uncertainty increases proportionally with distance from the training distribution.
F.2 Test-Time Sample Size Analysis
Increasing the number of samples generally yields more accurate estimates of the expected output, thereby improving model performance in terms of ECE and NLL. We ran additional experiments and report TFB's performance for sample sizes ranging from N (reduced to BLoB-Mean) to N 160 on the WG-S dataset [50] in Fig. 3. Each experiment is repeated three times with different random seeds on Llama3.1-8B.
As shown Fig. 3, increasing the number of samples at test time generally improves the alignment between the Monte Carlo estimates and the expectations in theory, leading to better uncertainty estimation. Our choice of = 10 samples in the paper balances performance and test-time efficiency: it achieves significant improvement in ECE and NLL while introducing acceptable computational overhead. The performance gap between this setting and extremely large is also mild.
F.3 Anchor Dataset Size Analysis
We present the performance of BLoB-Mean + TFB on the ARC-Easy dataset with varying anchor dataset sizes ranging from 100 to 2000 in Figure 4. Initially, we hypothesized a negative correlation between the performance variance and the anchor dataset size. However, as shown in the figure, across experiments with three different random seeds, neither the performance variance nor the average performance exhibits significant correlation with the anchor dataset size. This suggests that TFB is robust to the size of the anchor datasct.
Accuracy 1561
ECE (%)
NLL
12.0
22.6
0.24
3
12.0
0.1
0.22
0.2
1030
1030
1750 2202
750
L:oo
Figure 4: Performance of BLoB-Mean + TFB with different size of anchor dataset on the ARC- Easy dataset. The shaded area represents the standard deviation of results across three random seeds, indicating that TFB is not sensitive to the size of the anchor dataset.
22

---

F.4 TFB Beyond Training Data as Anchor Dataset
In this section, we designate a portion of the unlabeled testing dataset as the anchor dataset, simulating scenario where partial user input is accessible. The anchor dataset size remains fixed at 500 across all datasets. As illustrated Table 10, the performance variation across different data sources for the anchor dataset is minimal, indicating that the choice of data source for the anchor dataset has negligible impact.
F.5 TFB Beyond the NLL Metric
We report the additional results of TFB when using Accuracy (ACC) as the evaluation metries / in Table 9. our implementation, we adopt the change ratio of classification results as an accuracy-based evaluation metric (lAcc) for the unsupervised anchor dataset. Comparing the two evaluation metrics (lAcc vs (NLL) in Table 9, we observe comparable performance across datasets. In some cases, accuracy-basc based evaluation (lace)
yields slightly better results. For instance, BLoB-Mean+Hi achieves lower ECE on several datasets when using lace. However, we adopt NLL as the primary evaluation metric in Table since it better aligns with our theoretical framework in Theorem 4.2.
F.6
TFB Beyond the Low-Rank Isotropic Gaussians
In Sec. 5.4, we compare TFB with two alternative Gaussian distribution families that are controlled by a single parameter .g:
Full-Rank Isotropic Gaussian (FR): given Tg. the FR's variational distribution of the weight
matrix q(vec (W)) AV(eec (W) pag Eg) where Py Wo + BA (same as TFB) and
Eg = is full-rank.
Constant Standard Deviation Matrix (C-STD): given Tg. the C-STD's variational distribution of the weight matrix q(vec( ()) = (Veco W Mag Eg) where M4 = Wo + BA (same as TFB) and Σn o In [BB]]
C-STD's covariance matrix Σg derived through Lemma D.1:
Σg I, @ [l (1)) B
(37)
= I X B |
I, @ B
(38)
o I B ]
(39)
BB
(40)
This depends on B and thus varies for equivalent LoRA parameterizations B, A the same AW.
We report the additional results comparing TFB with other approximate families Gaussians (FR and C-STD as discussed in Sec. 5.4) when using Accuracy as the evaluation metrics | in Table 11. When the evaluation metric is set to Accuracy, the advantage of TFB becomes more significant compared to the results shown in Table TFB with low-rank isotropic Gaussian as the variational distribution demonstrates superior calibration performance compared to both FR and C-STD variants while maintaining competitive accuracy. For ECF, TFB achieves better results across most datasets, with notable improvements on in-distribution tasks: 8. 8.78% on WG-S (vs. 12.06% for FR and 1 11.61% 11 for C-STD) and I. 1.28% on BoolQ (vs. 3.26% for FR and 2.65% for C-STD). Similarly for NLL, TFB consistently outperforms or matches the baseline variants, particularly on WG-S (0.55 vs. 0.63 for FR and 0.61 for C-STD) while preserving comparable accuracy scores. These results suggest that TFB's approach to variance modeling is more effective than both full-rank isotropic and constant standard deviation alternatives.
F.7 TFB Beyond the Llama3.1-8B Backbone
We report the detailed performance of TFB applied to various LLM backbones in Table 12. While the baseline MLE is typically trained for 2 epochs (shown with each backbone name), we also report results with reduced training (1 epoch) for comparison. Although training with fewer steps (early stopping) can effectively reduce model overconfidence, it typically leads to performance degradation.
23

---

The results demonstrate that TFB consistently improves model calibration across different backbones while maintaining competitive accuracy. Specifically, for Llama2-7B, TFB reduces the ECE from 4.50% to 1.24% on the combined dataset while preserving the accuracy (81. (81.32% vs 81.41%) Sim- ilar improvements are observed with Llama3-8B, 1-8B, and Mistral-7B-v0.3. where TFB achieves better calibration than both the full training and early stopping baselines without sacrificing performance, suggesting its effectiveness as a general approach for enhancing LLM calibration.
F.8 TFB Beyond the Naive LoRA
We report the detailed performance of TFB applied to various LoRA variants in Table 13. The baseline models are trained for 2 epochs using pre-trained Llama3. 1-8B on the concatenated dataset of six commonsense reasoning tasks. Specifically, we consider the two LoRA variants:
VeRA [35]:
Uses shared low-rank matrices B and A across layers, with layer-specific trainable scalar vector and bias vector b. Concretely, the parameterization of VeRA's updated weight matrix W is modeled as:
W Wo AW Wo [ding(b)]][diaa((d))
(41)
Hence after the fine-tuning of VeRA, we can easily regroup the weight matrices into (B' [diag(b)) B [ddiag((d))) d)), A' A}, and apply the TFB Bayesianization scheme illustrated in Algorithm
PiSSA [41]: Employs an alternative initialization scheme while maintaining LoRA's param- eterization and training procedure. Hence the TFB process for PiSSA trivial.
The results in Table 13 show that TFB consistently improves calibration across different LoRA variants while preserving model performance. Notably, when applied to the standard LoRA, TFB significantly reduces the ECE from 4.74% to 1.05% on the combined dataset with minimal impact accuracy (86.45% vs 86.70%) Similar improvements are observed with VeRA and PiSSA variants, where TFB achieves better calibration (reducing ECE to 1.44% and 1.17% respectively) while maintaining comparable accuracy levels. These results demonstrate that TFB can effectively enhance model calibration across different LoRA architectures without compromising their performance.
F9 Improving the Inference-Tinne Efficiency of TFB
We report the detailed performance of last-layer TFB (LL TFB) in Table 14. As indicated in the table, with only N 10 samples, last-layer Bayesianization provides a less effective uncertainty estimation compared to full-model Bayesianization. However, increasing the number of samples to N = 100 significantly enhances the posterior estimation, allowing last-layer Bayesianization to achieve better accuracy. This improvement further allows it to outperform the full-model Bayesian- ization in terms of NLL across most datasets.
F.10 Additional Results on Llama2-7B
We report the detailed performance of TFB applied to the Llama2-7B pre-trained weights in Table 15. The performance change tolerance e is set adaptively to either 1 1% or 0.5%, depending on the checkpoint's overfitting characteristies. To determine the optimal o7, conduct parallel experiments with eight values of a, E [0.01, 0.015, 0.02, 0.025, 0.03, 0.035, 0.04, 0.05] using single random seed. We construct approximate function (p) through piecewise linear interpolation of
the observed performance and estimate of R (po - e). Similar to other baseline methods, the final results of TFB are reported averages across three random seeds using oz
In n--iitribution (IND) Results. We observed several key patterns from the IND Datasets results. For example, the MLE baseline shows relatively strong accuracy but suffers from high ECE values (e.g., 29.83% on WG-S), indicating significant overconfidence. This aligns with the common challenge of LLM overconfidence during conventional fine-tuning.
TFB applied to BLoB-Mean demonstrates strong overall performance across the IND datasets, achieving the highest accuracy on several datasets (69. 94% on WG-S, 70.72% on ARC-C, and 86. 74% on ARC-E). More importantly, it achieves this while maintaining lower ECE values compared to methods like MCD and ENS, suggesting better calibrated predictions. The method also shows strong
24

---

NLL performance, with values consistently among the lowest across datasets (0.62 for WG-S, 0.86 for ARC-C).
In summary, TFB consistently enhances the performance of baseline methods (MLE, MAP, and BLoB-Mean) across different evaluation scenarios, with notable improvements in both accuracy and calibration metrics. The improvements are particularly evident in the significant ECE reductions (e.g., from 29.83% 16.26% for MLE on WG-S) while maintaining improving accuracy, with the most substantial gains observed when TFB is combined with BLoB-Mean, achieving both the highest accuracy and lowest ECE values across most datasets.
Out-of-Distribution (OOD) Results. The OOD evaluation reveals interesting patterns across both smaller and larger distribution shifts. For smaller shifts (ARC-C and ARC-E), BLoB -Menn with TFB maintains strong performance, achieving 70.38% and 80. 16% accuracy respectively, while keeping ECE values low (12.28% and 8.07%). This suggests robust generalization under moderate distribution shifts.
For larger shifts (Chem and Phy datasets), we more significant performance degradation across all methods, as expected. However, BLoB-Mean with TFB still maintains competitive per- formance, achieving 42.67% % accuracy Chem and 30.67% on Phy. while maintaining reasonable calibration metrics. The method's s NLL values (1.35 and 1.46 respectively) remain competitive with other approaches, indicating relatively well-calibrated uncertainty estimates even under substantial distribution shifts.
Notable is the consistently strong performance of the BLoB variants (both w/ and w/o TFB) across different metrics and datasets, suggesting that this approach offers a robust framework for both in-distribution and out- out-of-distribution scenarios. The results demonstrate that the method successfully balances the trade-off between accuracy and calibration, particularly evident the out-of-distribution scenarios where maintaining both aspects becomes more challenging.
25

---

Table 9: Performance of different methods applied to LoRA on Llama3.1-8B pre-trained weights, where Accuracy (ACC) and Expected Calibration Error (ECE) are reported in percentages. "TF?" denotes whether a method is Training-Frr Free. The evaluation is done across six common-senss reasoning tasks with a shared hyper-paramete setting after fine-tuning of 5 epochs We sample N ==11 10 during inference in all sampling-based based methods including BLoB [62] and TFB. Rows with shading indicate training-free Bayesianization methods that use a pre-trained LoRA as their mean. For TFB, we randomly sample a subset of the training data without labels as the anchor dataset D. For accuracy-based evaluation (lAcc), we set the performance drop tolerance to e 1%. For NLL loss (INLI.,, we use the same settings as in Table 1. "f" and "L" indicate that higher and lower values are preferred, respectively. Boldface and underlining denote the best and the second-best performance, respectively.
Out-of-Distribation Datasets (OBQA-t)
In-Distribation Datusets
Metric Method
TF?
Small Skiff
Large Shijt
WG-S
ARC-C
ARC-E
WG-M
OBQA
BoolQ
ARC-C
ARC-E
Chem
Phy
MCD
78.03400
81 81.643139
91. 37,37
83.18
87 10
89 19.93
81 42
87.27
47.92+225
46.53=+a
ENS
78:82:65
82.55an
91
83.99.am1
87. 87.37
90.50.a11
79.623957
86.56gam
49.65:8m1
44 44:4
LAP
76.05e002
79.95404:
90.734008
82.83e0s5
87.90=000
89.36-051
81,08=1.0
87.21s120
18,264332
18
MLE
17.87.00
81.08
91.
82.30
87.90.pvr
89.58.436
81.48
86.83,aa7
42. 177
IFB (ace)
76.40:a:s
82.00
91 91:39
82.37
88.07:302
89.66:439
10.66:11
6,44g134
23.592124
26.90=427
TFB
82.533100
91.33amm
82.53.05
88.53
89.75.0
79.76.131
85.522a56
44.331405
16
ACC (1)
MAP
76.90
81.08-244
91.61
82,59=015
85.734009
90.09-922 25
79.98-4077
86.58
43,40
TFB (ace)
76.00=134
82.53
91
11:39:mmm
12.19
86.13
89.84
79.73s1.s
86.
18:
42.67s530
36.00:136
TFB (SLL))
1om
82.80
91.30
91.39
82.64.09
86.00
89.96
1126
86, 10,30,2:
45.33
16.45
82.32
91.141154
82.01
87.57
89.65:411
19.75
42.71
44.79.sm
HI HL
77
82.601000
91
83.92.a
88.00.0a0
89
89.86.a
K2.06.115
88.54
39.933500
39.93.mm
TTB (ace)
75. 28e0ss
82.80
81.84
88,00=006
89
35
82,934131
86.97
36,00#556
36.00-372
IFB (bar.)
77.81
63.33
91.76au
23.8
87.80
90.11:023
82.93:1.4
87.64:831
39.67an.2
37.33
MCD
16. 13anss
13.69
6.73.67
13.05
9.76.m71
77
13.63a1
9,27
30.91
33.0821.0
ENS
14.72star
13.45a119
6.59-044
11.17205:
8.17=0a
7.35
11.37a132
7,21etu
18.92s60
26.80=122
LAP
4.18
9,26a308
5.27=031
3,505a7
8.93-03
1,93aam
7.83
7.80a199
14.495037
13.17
MLE
17.02.004
7.00
13,83
9.77_pa
8.69
14.451219
10.78.050
32.461220
38
TFB (sce)
8.77
9.9729
4:32.00
6,10in
5.96208
5.96in
10.6621m
6.44g 1
23.592i21
26-90:177
TFB
120981007
11.63,nes
5.14_o1
1070
7.20_om
1.39
6.54
6.
14.63
19.68
ECB (L)
MAP
18.71
15.77
6.62
14.262090
12.19a03
8,40
16,46304
11,36295
34.792574
38, 18
TFB (lace)
1188
8.61:293
5.19:0as
7.51
8.72=623
5.55eas4
12.19:1a9
8.08
27.76:544
31.91=1m
TFB )
14s95
11927
5.76_
10.97
9.70_am
6.86103
13.25
9,22
27:21
BLOB
9.
5,41a1
2.70=000
4,288164
291-pm
2.588
5.61:6m
2.48:443
16.67an37
12.78-418
BLoB-Mean
15-43=a1s
12.11=152
4.1==2m
9.37
6:Maous
6.268929
11.22+00
6.34
26.65+305
25.40=544
17B (ace)
3,04=112
6.7621.4
4.81=1s
7.426124
5.26-001
3.2269:5
6.55g104
5.54
17.00s471
16.65
(bar.)
8.16
6.48
2.44
3.83504
2.67-au8
3.10
6.69
18.45
20.53
MCD
0.83atm
0.992n
0.45.0m
0.642am
0.62.0m
0.49g000
1 1.08gom
0.6122m
1.91
25
ENS
0.75=tic
0.80aan1
0.38=0m
0.55+uue
0.45=066
0.126916
0.72=t07
0.44
1,40=015
1.50
LAP
0.56
1.18
1.4:com
0.5
0.94.cmm
0.43
1.17.mm
1.11gare
1.27am
1.28
MLE
D.88:cas
1.20
0.46
0.68
0.61:006
0.52
1.07:wm
0.72
1.91
2.25
TFB (sce)
0.581am
0.731ns
0:29_0m
0.461am
0.41.mm
0.361at
0.79g00s
0.491am
1.52.0s
1
(NUL))
0.68=500
0,85+am
0.33-00
0.534am
0.46-004
0.42+1.0
0.66:4002
0.44=2.0
1.39-m1
1.49-306
NLL
MAP
0.99
1.12
0.46-cu
0.74
0.79-xoz
0.52
1.19:0044
0.83:aae
1.97
2.32
TFB (ace))
0.65=cxe
0.65:aas
0.35-om
0.4844054
0.54=004
0.35eaue
0.88:200
0.60e118
1.58:012
1.95ea1
TFB (oL.)
0.77
0.80
0.38_
0.57
0.61.omm
0.40
0.96
0.66
1.6
2.12.014
HL.8
0.58
0.51
0.2
0.43am
0.3400
0.2614m
0.56-1m
0.35
1,35 10
BLoB Mean
0.74=cx
0.73
0.29-0m
0.476am
0.37=002
0.32
0.67-2007
0.39
1 153-n13
1.54
TFB (aoc)
0.52
0.52
0.24-cmm
0.45
0.35:mm
0.28:41e
0.55:mm
a.37
1.4 1.38
IFB (1xL.)
0.55=cmm
0.53
0.23-1uu
0.40
0.33-bm2
0.27eam
0.52-x08
0.35=ax2
1.36am13
1.462n
26

---

Table 10: Performance of different methods applied to LoRA on Llama3.1-8B pre-trained weights, where Accuracy (ACC) and Expected Calibration Error (ECE) are reported in percentages. (Puni) and (Pus)) denote the anchor dataset is randomly sampled from the training dataset and the testing dataset, respectively. "TF?" denotes whether a method is Training-Free Free. The evaluation is done across six common-sense reasoning tasks with a shared
saper-parameter setting after fine- tuning of 5 epochs We sample N 10 during inference in all campling-bssed methods including BLoB [62] and TFB. Rows with shading indicate training- free Bayesianization methods that use a pre-trained LoRA as their mean. "T" and "L" indicate that higher and lower values are preferred. respectively. Boldface and underlining denote the best and the second-best performance, respectively.
Out-of Distiif-Distrbbttion Datasets (OBQA xX)
In- Disttribbbuion Datasets
Metric Method
TF?
Small Shift
Large Shift
WG-8
ARC-C
ARC-E
WG-M
OBQA
BoolQ
ARC-C
ARC-E
Chem
Phy
MCD
X
78.03-061
$1.61
91.37eas
83. 18asss
$7,20 17,20
89.93-
81.4=-13
57.27
47.92=215
46.53=1.2
ENS
78.82
82.55 m04
91.84
8399
90.50
86,56110
49165
44 1w
LAP
BP
76.05-to2
79.95
90.73
82.83
87.90au3:
89.36
81.08:120
87.21
48.26=112
46.18:130
MLB
77.87
81 n
91.67
82 M0aes
87 27.90
89
28
86883n007
45.832085
42 16
TFB (Duas)
77 27
82.531100
91.38
82.53100
89.75
79.76.11/
85.52,005
44,331110
37:00_216
TFD (Pan.
77:07:04
82.27=059
91.33
82.51
89.60=a
81
86.674m7
47.335
40.33-220
ACC MAP
76.90=uon
81,08:224
91.61=044
82 22.59
85.73
90.09
79.98=
36.58
43.40z45s
38. 38.54:3.40
TFD (Paus)
76 76.43-0722
82 62.80=144:
91.39
82.64
$6.00
89.96=9.11
80.61=12
$6.30
45,334-237
35.67=44
133 (De))
76 06:2::m
82.40.1.2
91.642074
82.29
a6.07
89.99...1
-0.85:112
86,30
44.672170
36.00:43z
BI .B
X
82.32
91.
11.14
X2.01
20%
87.57.991
19
29.75.04s
87.13
42.711371
44.79_6m
BLoB-Mcan
X
77.72-012
82. 20:60
91.64
K3.92=04
S8.00203)
89
05
82
06.06-11
88.541001
39.93=520
39.93-4m
TFB (Deas)
77 77.81 2n0
83,33:0
4
83.8
87 87.80
10-11-00
82.93:144
87.64
39.678733
37 33
IFD (Pan)
77.65-005
$3,20e1.3
91.94:11e
83.41s00
87.7340x
89.7
82.93-147
57.15
37.676450
37.00=1.4
MCD
X
16.13.001
13.69
6.73.mm
1300
9.76.171
17
13.63.118
9.27.am
30.91
33.08
ENS
X
14.72.0017
13.45
6.59
11.17.000
8.17
1.35
11.37
7.21au
18.92
26.80.123
LAP
BP
4.18
9.26,334
5,27.05)
8.93ga3
1,93
7:832100
7.80a1m
14.49:a5
13.17
MLB
17:02:0a5
16.35
7.00
13.83200
9.77gnm
8:001
14.452219
10.78,339
32.461220
IFD (Pane)
/
12.98:037
11,634088
5.14
10:01
7.20#047
7.39
6.54
5.69:114
14,63+14
19.68-127
1EB (Dac)
11:22:115
13.85
5.62:00e
9
7.40:10m
6.05
13.41am
2,98
29.72
36.22 3.:22:1..
ECB (1)
18.71
15.77
6.62:0044
24
12.
8.40
16.46:0a
11.36aus
34.79
IFE (Dan))
14.95 pos
11:27
5.761a6)
10.97 1119
9.70
6.861031
13:25
9.22
27:211:30
35.91
TFB (Daz)
13.08 1.:58
12,85
44
9.28-115
15
9.86
6.442037
11:07:15:
8.022100
24.512:76
33 25
9.93=622
5.41+u
2.70=0m
1.28=ts4
2.91
2.58e913
5.61=840
2.48:043
16.67
12.78-418
BLoB-Mean
X
15:43=0s
12,41=1552
9.37-135
6.M4mu5
6.262929
11:22
6.34
26.652306
25 25.40=5.40
IFB (Drns)
8. 16:244
6,48
2.44:
2.67:0m
3.10sa39
6.69-125
3.61
18.45
20.53:427
TFR (Dane)
1.77
6,92
109
2.94
1.69
1,48
1.1
3,56
17,15
16
MCD
X
1o83
1.99
0.45
0.64.cm
0.62100s
0,491000
1.05_am
1.11
2.02.
ENS
X
0.
D.75
0. 0.80
0.38:00
0.55=tm
0.452005
0.426905
0.72=6m
0.44:00
1,40
1.50 1.50=a1
LAP
BP
0.56
1.18au02
1.4
0,51=co
0,94
0.43
1.17smm
1.1
1,27
1,28=1ee
MLE
a.88
1.20am1
0.46:5004
0.68=tm
0,61a
0.52
0.72
2:25
TFE (Deae)
/
0,85 Lea
0.331:00
0.53.cm
0.461101
0.42
0.66_aie
0.44 Loot
1,39
1.49
IFE (D )
/
a.63
1.00
0.361:10
0.51
0.46:00s
0,36
0.83.0m
0.5l
1.59
1.94
NLL
MAP
1.12
0.46100
0.74gtm
0.79gmm
0.52gam
1.19_00
0.83
11
1.
IVD (Drne)
/
0.77=106
0.80a015
0,38+00
0.57=t
0.61:008
0:40
0.96:006
0.66:200
1.69
2.12
IHE (Da.)
/
0. 0.72
0.92
0
0.34::10
0.51
0.61
0.37
0.98
0.66
I. L.70
2
12
BLoB
X
0.58
0.51:
0,23-a0:
0.43=om
0.31amm
0,26:axs
0.56=tic
o.35
1.4
1.35=a10
BLoB-Mean
0.73
..29
0.47-cuo
0.37
0.32
0.67..om
p.5
1.54
TFB (Draa)
0.53:09t
0.23:aa
0.40_4m
0.33:
0.27
0252.mm
1.362n 13
1.46550
TFR (D,..
/
0.54 tm
0.54
0.24.am
0.41 .cm
0_34
1.27
0.54 cm
0.36 1oo
1,38
1.4
Table 11: Performance of TFB with accuracy-based evaluation metric (/=ACC) using different posterior families applied to the mean of BL.oB, based on Llama3.1-8B pre-trained weights. FR: Full-rank isotropic Gaussian noises are applied to AW: C-STD: Standard deviation matrix Ω = [2, = o,] is constant. The evaluation protocol strictly follows Table 1. 1." and "1" indicate that higher and lower values are preferred, respectively. Boldface and underlining denote the best and the second-best best performance, respectively (only for TFB variants).
Oft-of-Distribttion Datasets (OBQA-X)
In-Distribution Datasets
Metric
Method
Small Shift
Large Shift
WG-88
ARCAC
ARC-B
WG-M
OBQA
HoolQ
ARC-C
ARC-E
Chem
Phy
Mean
77.72=a11
$2.60ammn
91.6 91.64:03s
83.92:14
88.00st0m
89,86
82.06=115
88,54
39.93.33:
39.93
ACC (t)
TFB (C-ST)
78
46
82,
22.75
92.38:am
91,52s
83.33
83.44
88.40:016
38.37an00
90.03-011
90.21
82.27=:10
82.38
87,98
43,33-335
40.00
41.67441
11:33-133
IFB (Final)
75:28:033
82,
22,80
91.642013
81.84
58.00am1
89.60
82,93
86.97
36.00:335
36.00
BLoB Mean
15.43
12.41a152
4.91anms
9:37=133
6.44
6,261929
11.22
6.34
26.65:105
25.40
BCB (-)
7FB (C-STD)
11,19-13
14.07 2m33
5.74:124
9.85
3.21:0.4
4.17pmm
8.94.150
4.22:6.14
5.48
5.72
4,90+04)
8.85
9.29
4.1846722
6.131041
14.78
17.63-429
24.5
TFB (Final)
3.04:a11
6.76:147
4.81s1.
7:42:124
5.26
3.22
6.55
5.54
17.00:471
16.65
Mean
0.74g310
0.73
0:201133
047..0m
a.37.am
0.32
0.67-4m
0.39amm
1,532n
1554
NLL (4)
7FB (C-STD)
TFB (FR)
0.61
0.68.am
0.59
p.25
0,23
0.43.aa
0.46.410
0.34
0.32.
1.29
0.21 103t
0.68 010
0.61 501
0.37 100m
1,36
1.49.013
145.0m
7FB (Final)
0.52
0.52.am
0.24
0.45-4m
0.35.0m
0.28
0,55-440
0.37
1.38
1.41
27

---

Table 12: TFB Performances with various LLM backbones [54, 12, 26], where Accuracy (ACC) and Expected Calibration Error (ECE) are reported in percentages. The MLE training for each different backbone is conducted for 2 epochs on the concatenated dataset of six commonsense reasoning tasks, with a shared hyperparameter setting: "Fewer Epochs" represents training for epoch. We use N = 10 samples for TFB during inference and rows with shading indicate training- free Bayesianization methods that use pre-trained LoRA as their mean. "f" and "J" indicate that higher and lower values are preferred, respectively. Boldface and underlining denote the best and the second-bee best performance, respectively.
Datasets
Metrie
Method
WG-S
ARC-C
ARC-E
WG-M
OBQA
BoolQ
Combined
Llama2-7B
72.30
33.24+134
87.66:081
72.30=a00
83.27+155
87.84
$1.41:a..
Fewer Epochs
63.85.1366
69.231033
86.732009
63.85=10s
79.6721555
86.082023
1..8
TFB (Ours)
72s03
74.36
87.31
72.85 19.9
83.73
87.44.034
81.32.0.51
Llama3-8B
81.45:0000
84.95
92.631n9s
81.45
88.20
10.19...13
86.93:..m
Fewer Epochs
79.08=+118
82.72+0.39
92.22+0.53
79.08=1.11
86.07
14.94
+3.76
ACC
TFB (Ours)
59
14.73+168
92.98
-1.1
87.73+0644
89.75+0.10
86.61 +0.20
Llama3.1-4 1-BB
81.24:0015
82.72+0.19
92.
81.24:4.15
87.80=200
90.20
86.70
Fewer Epochs
78. 10
83.95+
91.17+1.17
78. 78.11=50 =
85.33++050
89. 00.35
84. 22
TFB (Ours)
80.66:107
82.5010
91.93:105
81.22
87.73+1.99
89.962023
26.45
Mistral-7B-vO..
82.45.1022
84.28
90.94
82.45 14.83
17.73
89.71
86.88
Fewer Epochs
79.72:1000
83.95
91.581043
79.72.s0.00
87.5 87.5
89.20:1020
85.71
TFB (Ours)
81.74
84.06=148
90.99:0.3
81.74=075
87.93=0442
89.71+0.22
86.64
1.lama-7BB
9.17+0.74
9.37
2.65=0.16
9. 9.174074
5.54
1.59
4.50+a37
Fewer Epochs
4.83
5.67
4,462002
4.83
=04
6.90
2.00
TFB (Ours)
5.44
6.06=1.54
3.83=004
5.50
3.87=111
2.51
1.24
Llama3-8B
8.49
6.76
2.57=0.84
8.49
3,84=0.37
1.88
20.4
Fewer Epochs
4.45+0.22
4.99=21M
2,83=0.58
4.45+0332
3.14=0.13
2.71
1.79+116
ECE (L)
TFB (Ours)
3,47+074
5.58
4.34
4.07
3.79-000
3.49=1442
1.64:2644
Llamn3. 1-8B
8.58 3.588
2..8
2.92:492
8.58
3.85
2.32.
4, 4,74
Fewer Epochs
4.7611091
1.29 1..9
3.11
4.76.099
3.99
3.02.0.59
1.45
TEB (Ours)
4.45
4.34
2.97=026
4.56
3.55
3.16::045
1.05
Mistral-7B-vO..
8.02#1.8
6.98=1.18
4.12=a.13
8.02+168
5.99=2.4s
3.17=0.55
5.05+aus
Fewer Epochs
5.72+201
4.74
2.52:4479
5.72+201
3.50=077
1.70=047
2.47+100
TFB (Ours)
4.47:2.0
4.72
2.62:0020
4.01
4. 4.
0.97
1.68
Llama2-7B
0.581001
0.69
0.35
0.581001
0.48
0.30
0.43
Fewer Epochs
0.6420.00
0.78=0.0
0.39
0.64
0.56=0o02
0.36=001
0.50=am1
TFB (Ours)
0.56:00.1
0.68
0.35
0
0.46=0.03
0.31
0.43-=4u10
Llama3-8B
0.482001
0.47=0.0
0.22
0.48
0.35
0.25
0.34:2000
Fewer Epochs
0.46110.11
2..8
0.22:
0.4611011
0.37
2.4
0.40
NLL (1)
TFB (Ours)
0.4410001
0.45
0.23
0.44
0.35
0.271000
0.34
Llama3. 1-BB
0.48+001
0.53
0.24=om
0.48
0.33
0.25
+.35
Fewer Epochs
0.48+0.00
0.45
0.23
0.48+n00
0.37=000
0.27=000
0.36=am0
TFB (Ours)
0.44
0.4620000
0.23=aam
0.44
0.33=
0.27=000
0.34
Mistral-7B- -B00,33
0.4611001
0.47.0.02
0.28
0.461004
0.36
0.261.0.01
0.35 1002
Fewer Epochs
0.4710.2
0.4610.01
0.25
0.40
0.35
0.26
0.35 10001
TFB (Ours)
0.42
0.43
0.26=0001
0.42::u02
0.33
0.26:0001
0.33
28

---

Table 13: Performance of TFB when applied to variants of LoRAs [23, 68, 35, 41], where Accuracy (ACC) and Expected Calibration Error (ECE) reported in percentages. The MLE training for each LoRA variant is conducted with pre-trained Llama3. 1-BB model for 2 epochs on the concatenated dataset of six commonsense reasoning tasks, with a shared hyperparameter setting. We set the number of samples to N = 10 for TFB during inference and rows with shading indicate training-free Bayesianization methods that use a pre- pre-trained LoRA as their mean. "f" and " indicate that higher and lower values are preferred, respectively. Boldface denotes the best performance.
Datasets
Metric
Method
WG-S
ARC-C
ARC-E
WG-M
OBQA
BoolQ
Combined
LoRA
81.24
82.72=019
92.11=115
81.24=au15
87.80=203
90.20=um1
86.70
+ TFB (Ours)
80.6610.70
82,50108:
91.931100
81.22.003
87.73.1129
89 89.96.1023
16.45
ACC (1)
VeRA
7..24
82.39
90.47=1117
18.
86.13:0023
89.27
84.93
TFB (Ours)
76.82#007
81,27=2234
90.35=a01
77.03=1104
86.07=0044
88. 88.99
84.28=0.48
PISSA
81.4511.45
83,95
92,22.054
81.45
88.40
10.40
90.0910.11
86.83
TFB (Ours)
80.77+1.42
82.94
92.40:0.66
81.32=078
88.
13::0:2
90:01
86.61 20043
LoRA
8.58=0.56
8.58=029
2.92:=n92
8.58+tn.6
3.85++1.15
2.32=
4.74=00.x
TFB (Ours)
4.45
1.29
2..26
4.56.00.8
3.55
3,1610.45
1.05
ECE (1)
VeRA
9.54:1047
7.26:12.2
3.72.1086
9.54.0047
1.41
2.2811040
5.1 1.0.5
TFB (Ours)
5.03=0.92
5.92=1.53
2.80
5.0940.7
3.31
1.78=00.0
1.44=144
PISSA
7.36=0.40
8.12=122
2.83
7,36#0.00
3.73
2.59#0030
4.26-1014
TFB (Ours)
4.591063
4.97-1063
2.71
4.37.10.22
2.96.10.16
1.41
1.17:0322
LoRA
1.481001
0.53
0.24.10.3
0.48
0.33
0.25
0.35
(Ours)
0.44:a.11
0.46::0.00
0.23
0.44:10.1
0.33
0.27:000
0.34::0.0
NLL ())
VeRA
0.54
0.53=000
0.29
0.54+0o0
0.41
0.27=aa.1
0.39
TFB (Ours)
0.51
0.27
0.50
0.281000
0.38
PISSA
0.47
0.49
0.23
0. 0.47.10.1
0.32.10.0
0.26110.00
0. 0.35
TFB (Ours)
0.44=0.01
0.46=au02
0.23-=n.01
0.44=
0.32++n.02
0.26=00.00
0.33-:1000
29

---

Table 14: Performance Last-Layer TFB (LL TFB) applied to LoRA on Llama3.1-88 pre-trained weights, where Accuracy (ACC) and Expected Calibration Error (ECE) are reported in percentages. The evaluation is done across six common-sense reasoning tasks with shared hyper-paramen setting after epochs. We sample N times during inference in the sampling- based methods. Rows with shading indicate training-fiee free Bayesianization methods that use pre-trained LoRA as their mean. "T" and "4" indicate that higher and lower values are preferred, respectively. Boldface and underlining denote the best and the second-best performance, respectively.
Datasets
Metric
Method
#Sample (N)
WG-S
ARC-C
ARC-E
WG-M
OBQA
BoolQ
MLE
77.87:2.54
81.08
91.67=033
82.3020.59
87.901087
89.58
TFB
77 17.44.0.30
82.53
91.33.0.7
12.5310.56
88.531057
89.75
TFB
76.96:2046
82.00
90.97=a34
82.67=0.49
87.80=107
89.6 12.6222111
77.3920.30
82.13-1022
91.33=037
82.61
87.80=091
89.66.20.2
MAP
76.9010.9
81.08.2248
91.611044
82.5910.29
85.7310019
90.09
TFB (Ours)
10
76.43
82.80+142
91. 91.39=037
86.00=a 82.64
89. 19.96=0.15
ACC (1)
LL TFB
10
76.35220.9
83.07
91 91.15=052
82.27
85.27 19
90.09
LL TFB
100
76.72.10m
83,07
91.151000
82.5310.39
85, 15.6010.16
90.02.0.14
BLoB
76.4510.37
82.32.1.15
91.141054
82.01
87.57.021
12.65..0 10.15
Bean-MMaa
72.72.0.12
82.6020x0
91.64=055
83.92:0.48
88.00=080
89 86.
TFB (Ours)
77.81
83.3310.19
91.76_2445
83.81
87.801011
90.11::.25
LL TFB
10
77.57110
82,801033
91.45
83,2310.57
88.33
89 19.85
LL TFB
100
77.60=0.22
83.33
91.39=000
83.6320.62
87.60=044
90.03=00m
MLE
17.0210.46
16.35
7,00
13,8310.6
9.77.0.81
8.691021
TFB (Ours)
10
12.98.1.37
11.63
5.14
10.01
7.2010.47
7,39
LL TFB
14.42:041
13.862045
6.922062
10.3220.0
8.5622006
7.52:2012
13.4510.00
13.17.002
6.841067
10.7610.88
1.68
7,4610.00
MAP
18.7110.74
15.77
6.621064
14.2610.92
12.191055
8,40
TFB (Ours)
10
14.95
11.27
5.76
10.97
9.70
6.864031
ECE (4)
LL TFB
10
16. 16.03:10.44
12.72.135
6.54
12.06
11. 11.36.104
7,51
LL TFB
100
15.56.00.9
12.84
6.38
11.80
11 11.22.1038
7,30
BLoB
9.93-1022
5.41
2.70
4.28::0.4
2.91
2.58
BLoB-Mean
15.43:.015
9.37
6.44.10.5
6,26
TFB (Ours)
8.16.10.45
1.48
2..4.10.50
3.83
2.67
3,10..099
LL TFB
10
9.68
7.201091
3,01
3.94
3.33
2.96
LL TFB
100
8.88
6.47+1.55
2.84
3,40
3.70=0.27
2.54
MLE
0.881004
1.2010.11
0,46
0.68 1000
0.60
0.52
TFB (Ours)
10
0.68
0.85
0.33
0.53=0001
0.46:00.4
0.42
LL TFB
0.70
0.9620.12
0.41 1:006
0.53
0.50
0. 0.42.1001
0.66.1002
1.84
0.39
0.53
0.49
0,40
MAP
0.99:1007
1.12:0.3
0.46
0.74::007
0.79
0.52
TFB (Ours)
10
0.77=auu5
0.80
0.38
0.61 #0.00
0.40=no1
(L)(()
LL TFB
10
0,80
0. 1.0.1
0,43
0.60.1005
0.6510..1
0.43
LL TFB
100
0.77:10.0
U. 0.861
0.41
0.57=1004
0.63
0.40
BLoB
0.58=a0m
0.51
0.23
0.43=a001
0.34
0,26=
BLoB-Mean
0.741002
0.731
0.2911003
0.47 1100
0.3710.00
0.32
TFB
0,55
0.53
0.23
0.40
0.3310.0
0.27
LL TFB
10
0.56
0.60=
0.26
0.4] =o.01
0.33
0.27 +000
LL TFB
100
0.53=
0.54+0004
0,24+0o1
0.39:=a.11
0.31
0.26=
30

---

Table 15: Performance of diffferent methods applied to LoRA on Llama2-7B pre-trained weights, where Accuracy (ACC) and Expected Calibration Error (ECE) are reported in percentages. "TF?" denotes whether a method is Fraining-Free The evaluation is done across six common-sense reasoning tasks with a shared during inference in all sampling-basee based methods including BLoB [62] and TFB. Rows with shading indicate training-free Bayesianization methods that use a pre-trained LoRA as their mean. For TFB, the anchor dataset 'D is set to randomly sampled subset of the original training set, the performance evaluation metric/ set to accuracy, and the performance drop tolerance is set adaptively to 1% or 0.5% based whether the given mean overfits. "I" and " indicate that higher and lower values preferred, respectively. Boldface and underlining denote the best and the second-bes performance, respectively.
setting after 5,000 gradient steps. We sample
10
Out-of- Out-of-Distributian Datasets (OBQA-XX)
Metric Method
TF?
In-Distribution Datasets
Small Shift
Large Shift
ARC-E
OBQA
BoolQ
Chem
Pby
MCD
X
69 46:0
68 69
46
76.45:a04
81.72a000 87. 2912 69.03am0
76.00s1 51 42.71=30:
29 47:17:
ENS
69.57 S7.
66. 66.20125
84.40:.11
75. 25.322331
81. 11.381091
87.09
67. 67,34103:
75, 78.18120
43.75_1m
30. 30.56_77
BBB
13:122
85.86
73.63224
82.06a009
67.2541.8
42.36=000
30.21-125
LAP
BP
69. 69.20-1.0
66.78su39
80.05ea11
75.55
82.12
86.95
69.14
74.94au
44.10=130
31.60-049
ACC
MLE
IPD (Ours)
68 99.
19.83eLie
69,
69.10s234
13:19
86,
85.65eart
86.21
74.53
75.95e334
81.52ss2s
82
92.80
$7.66-mus
56.53
69.934211
66 66.20m
75.12s00s
78.87e1
40.62-125
34.67=131
31.00=200
18.82-130
62:mmmm
m7.59
86.55
75.61.3m
11.38.nus
16.50.m4
75.47.0m
28.47.12
TFR (Ours)
69.17.3 1e
67.6821
m3
85.8613
75.872300
17.74
69
37.375
78.76183
34.31_.51
31:000.1m
BLoB
68. 80
67.591mm
86.371031
73.
73.261110
81.9911/4
16.58.10
67.71
76.37
44.79
1.7
31.60.2m
BLoB-Mcan
72.15=a.17
86. 31
15.47
36.88204
31.94-199
+ IFB (Ours)
70.72+2:5
86.74:a57
73.13
83.13=476
86.36am2
70.38:1.0
80.16eam
42.67=111
30.67=138
MCD
x
27.98
27.53
12.20
19.55e347
13.10amm
3.46:mss
19.54
15.32et
17.9
19.53:42)
ENS
28.52..0s
29.1612.7
12:57
20486
1.59
6.44
12.04_155
13
BBB
21:81
26.2331.4
12.28405
15
15.762471
11,38210
3.74
19,90ams
13.41208
13.67-12
10:40
IAP
BP
4.15
16.25
7:40
8.70
1:30
5884.64
8.51g1
10.76.241
13,91 jae
BCB
MLE
(Ours)
29. 59.83
36.26-933
29.001147
6.932144
13.12a) 1e
20.62g374
8.78
12.58goo
3.18
22
22.201nm
16.4710
4.64
15.87=511
21.72_a30
16.77-410
29 19.60.11
MAP
29.76-9a7
29.42aa0
12.07
23.07
13.26
3.16:
19.31
15.68
17.55
30.25-244
TFS (Ours)
11.72
6.07=13)
6.99
5:21
3.82
2.65=030
8,39
4.86812
16.11:92
16.35-200
BLoB
8.98:am
10.8
4.54:009
1.98:1
1.24.035
9.55
5,48
9.77
18.29
MB-MMann
20:5
9.64:005
11.58
8.65.0
2.88
14.01110
10:70,0 34
15
TFS (Ouns)
3.03
4.07 6.
1.37
12.281131
8:07
12
22.02 pmm
MCD
2.79
2.67
1.00a044
0.77=0m
0,31
1.08
0.88e
1.59
1.67=366
ENS
2.71
2.465022
1.25
0.865001
0.69
1,28:are
1.39
BBB
L.40:035
2.23
0.91
0.84
0.66-2ae
0.31=00
1.06100t
0.79
1.49saee
1.62-2w
LAP
BP
0.60
1.03 i0ot
p.57.am
0.52.cmm
0,31_mom poo
1m80
0.70
1m3
NEL.O)
MLE
3.17
1.17
0.95
0.73_cm
0.32
1.1610m
0.92
1,56
1.66
TFB (Durs)
0.98
0.48
0.50
0.54
0.302.00
0.87800
0,70
1.4622
1.44506
MAP
2.46.111
2.662011
0.00
1.62
0.75_cm
0.33gmm
1.10gmm
0.933100
1.65
IPS (Ours)
0.72891.
0.962ma
0.50
D. 0.55=100
0.53mt
0,30=600
0.87a002
0.71
14665000
1.425566
0.41:am:
0.54:am
0.49-am
0.31=00m
0.83:am:
0.60:am:
138
146
BLeB-Mean
0.79
1.27
0.57
0.60=s.e
0.56-240
0.32=001
0.89
0.67
L44sue
1. 1.53+scc
TEE (Ours)
p.62
0.86
0:42
0.56.:210
0.50.ccm
0.34
0.84
L33
1.46:506
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Every claim the Abstract and Introduction is supported either empirically as in Sec. theoretically in Sec. 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are addressed Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
the paper. The authors should use their best
will be specifically instructed
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification:
Yes, the assumptions are explicitly listed in Assumption 4.1, Sec. 4.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details of implementations of this paper and the baseline models are disclosed in Sec. 5 and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [No]
Justification: At this time, we do not provide code for our method. We will release the code once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines bettps://nips. cc/pu
blic/gun Ceriides/ooeeuiiii
onssiicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: /CodeSubmiiiiccec ceaps.cc/public/gut h commiissionn onssiicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our algorithm has only one parameter to optimized and is fully discussed in our algorithmic description and experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All of our experiments are repeated with 3 different random seeds, the mean and the standard deviation of which are reported in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
34

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our proposed method is training-free with only forward passes of models required, which does not rely on high-profile computational resources. We have detailed our computational resource type in Appendix 5.3.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: http///eerrppppeeeeeeeeeeee hctpuullce/chicccuueeeeeeeeeeeee
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: N/A.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: No licensing issues involved.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /ataestss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: For now there is new assets added by this submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing is involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No IRB is included.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
37

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLM usage for the core method is involved.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: / //ttc///eeeeeeeee///2/////////// for what should or should not be described.
38


Reviews:
[{"review_id": "YLK6Ma41lS", "reviewer": "Reviewer_BHzf", "summary": "This paper proposes to “bayesianize” lora finetuning of LLMs by using as the variational posterior the lora mean and a grid searched variance term on the training data. The loss used to search the variance term is the variance that keeps the maximum functional change in loss underneath a pre-specified threshold. Experiments are done on a pretty wide variety of LLM style benchmarks.", "questions": "Generically how is epsilon chosen here as I imagine that’s a clearly important hyperparameter? If this has to be chosen on a reference task, how do we ensure that this reference task represents the downstream usage tasks?\n\nWhat is the inference computational cost as compared to standard LLM inference?\n\n-\tIt’s not super clear to me what calibration error on textual generation / a text bot should actually mean. I think I understand when and how accuracy will be better (even from just the variational mean), e.g. we get more questions right.\n\nWhy is the Laplace approximation often better than MAP / MLE in Table 1?", "limitations": "n/a", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n-\tI appreciate the simplicity of the approach, as well as the toy demonstration in Figure 2 in the appendix. \n\n-\tI like the ablation studies with other versions of the approach; indeed, it seems that the structure you chose is probably the best tradeoff performance / compute wise.\n\n-\tI also like that you can just tune the variance on a pretrained Lora model, thereby getting this for a few passes through the finetuning data.\n\nWeaknesses:\n\n-\tChoosing the prior mean the same as the posterior mean in Thm 4.2 is definitely putting the cart before the horse. Under those conditions, pretty much anything is variational inference. The overselling of this result is one reason why i'm voting weak accept currently; this is a pretty restrictive set of assumptions (not a \"relatively weak\" set).\n\n-\tNo comparison to something like simply turning up the generation temperature to increase output uncertainty and then using many samples in the Table 1 benchmarks. \n\n-\tI appreciate the attempt to phrase the method as Bayesian, but that’s really a strong overstatement. The “meta” prior here is “any weight distribution that doesn’t change the function too much”, which is distinctly not Bayesian, but more a post facto way of justifying some sort of uncertainty quantification.", "was_revised": true, "final_justification": "I thank the authors for responding during the rebuttal period. Overall, this is a reasonable paper demonstrating that bayesian uncertainty quantification can improve llm benchmarks.", "rebuttal": "Thank you for your constructive feedback. \nWe are glad that you `\"appreciate the simplicity\"` of our approach and like our ablation studies, acknowledging that the structure we chose is `\"the best tradeoff performance / compute,\"` and that our experiments cover `\"a pretty wide variety of LLM style benchmarks.\"` Below we address your comments one by one in detail.\n\n**W1. Choosing the prior mean the same as the posterior mean in Thm 4.2 ... Under those conditions, pretty much anything is variational inference ... this is a pretty restrictive set of assumptions (not a \"relatively weak\" set).**\n\nThank you for mentioning this. The goal of our theoretical analysis is to identify the conditions under which the proposed TFB is equivalent to variational inference, thereby providing a solid theoretical foundation for our novel yet practically simple variance maximization framework.\n\nThe assumption that the posterior shares the same mean as the prior (i.e., the off-the-shelf LoRA adapter) reflects the practical design of our algorithm: in a training-free framework, estimating or modifying the parameter mean would be highly challenging. As noted in the `Remark` (lines 234–237), this assumption is also consistent with the core premise of other Bayesian methods, such as the Laplace Approximation, where the MAP point estimate is effectively used as the posterior mean.\n\n**W2. No comparison to something like simply turning up the generation temperature to increase output uncertainty and then using many samples in the Table 1 benchmarks.**\n\nThis is an interesting point. Following your suggestion, we have implemented this simple baseline, which is equivalent to MC Temperature Scaling (MCTS) [1], using the LoRA weights from BLoB-mean and MAP. To ensure a fair comparison, we strictly followed the hyperparameter settings in [1] and performed 10 samples at test time (same as TFB). Each experiment was repeated three times with different random seeds, and the results are reported as `mean (std)` in the table below across six datasets.\n\n**Table A. Performance of MC Temperature Scaling (MCTS) and TFB (Ours) applied to LoRA of BLoB-mean weights on Llama3.1-8B.** \n\n|Metric|Method|WG-S|ARC-C|ARC-E|WG-M|OBQA|BoolQ|\n|-|-|-|-|-|-|-|-|\n|**ACC($\\uparrow$)**|**MCTS**|77.71 (0.34)|**83.60 (0.98)**|**91.94 (0.31)**|83.28 (0.88)|**88.07 (0.62)**|90.06 (0.12)|\n||**TFB (Ours)**|**77.81 (0.36)**|83.33 (0.19)|91.76 (0.48)|**83.81 (0.39)**|87.80 (0.16)|**90.11 (0.28)**|\n|**ECE($\\downarrow$)**|**MCTS**|18.59 (0.35)|13.41 (1.19)|6.45 (0.09)|13.28 (0.74)|8.91 (0.49)|7.95 (0.05)|\n||**TFB (Ours)**|**8.16 (0.48)**|**6.48 (0.36)**|**2.44 (0.50)**|**3.83 (0.43)**|**2.67 (0.18)**|**3.10 (0.59)**|\n|**NLL($\\downarrow$)**|**MCTS**|0.97 (0.01)|0.81 (0.08)|0.37 (0.02)|0.62 (0.04)|0.46 (0.01)|0.42 (0.02)|\n||**TFB (Ours)**|**0.55 (0.01)**|**0.53 (0.04)**|**0.23 (0.02)**|**0.40 (0.01)**|**0.33 (0.02)**|**0.27 (0.01)**|\n\n**Table B. Performance of MC Temperature Scaling (MCTS) and TFB (Ours) applied to LoRA of MAP weights on Llama3.1-8B.**\n\n|Metric|Method|WG-S|ARC-C|ARC-E|WG-M|OBQA|BoolQ|\n|-|-|-|-|-|-|-|-|\n|**ACC($\\uparrow$)**|**MCTS**|**76.46 (1.33)**|82.38 (1.44)|**91.85 (0.59)**|**82.70 (0.58)**|85.73 (0.25)|**90.07 (0.18)**|\n||**TFB (Ours)**|76.43 (0.72)|**82.80 (1.42)**|91.39 (0.37)|82.64 (0.58)|**86.00 (0.16)**|89.96 (0.18)|\n|**ECE($\\downarrow$)**|**MCTS**|21.28 (0.99)|16.13 (1.60)|7.41 (0.67)|15.91 (0.62)|13.11 (0.15)|9.18 (0.21)|\n||**TFB (Ours)**|**14.95 (0.65)**|**11.27 (2.53)**|**5.76 (0.63)**|**10.97 (1.19)**|**9.70 (0.69)**|**6.86 (0.31)**|\n|**NLL($\\downarrow$)**|**MCTS**|1.30 (0.06)|1.40 (0.30)|0.59 (0.04)|0.95 (0.08)|0.98 (0.03)|0.66 (0.01)|\n||**TFB (Ours)**|**0.77 (0.05)**|**0.80 (0.15)**|**0.38 (0.03)**|**0.57 (0.05)**|**0.61 (0.03)**|**0.40 (0.01)**|\n\nWe can see that our TFB outperforms MCTS in most cases, especially in terms of ECE and NLL, which are the focus of our methods. This demonstrates TFB's effectiveness. We will include the results and discussion in the revision as suggested. \n\n**W3. I appreciate the attempt to phrase the method as Bayesian, but that’s really a strong overstatement. The “meta” prior here is “any weight distribution that doesn’t change the function too much”, which is distinctly not Bayesian, but more a post facto way of justifying some sort of uncertainty quantification.**\n\nThank you for your constructive feedback and your insightful suggestion on connecting TFB to uncertainty quantification methods.\nUnlike typical Bayesian methods that require **training from scratch**, an approach that can be prohibitively expensive for large language models (LLMs), we focus on the problem of Bayesian **fine-tuning**. In this context, it is natural to use the pretrained weights as the mean of both the prior and the posterior so that the \"weight distribution doesn't change the function too much,\" as you mentioned. \n\nTo further demonstrate the non-triviality of TFB’s approximate posterior weight distribution $q(W)$ (as defined in Eq. 15), we compute the Signal-to-Noise Ratio (SNR) of each layer:\n$$\\text{SNR}(\\mathcal{N}(\\mu_q, \\Sigma_q)) = \\frac{\\|\\|\\mu_q\\|\\|^2}{\\text{tr}(\\Sigma_q)}.$$\nA high SNR indicates that the distribution is dominated by the Gaussian mean, effectively nullifying the need for distribution modeling.\n\n\n**Table C. Square root of the Signal-to-Noise Ratio ($\\sqrt{\\text{SNR}}$) for the first 12 Query and Value layers of TFB on the WG-S dataset, applied to BLoB-Mean using Llama3.1-8B pre-trained weights.**\n\n| Layer |1|2|3|4|5|6| 7| 8| 9| 10| 11| 12|\n| - | - | - | - | - | - | - | - | - | - | - | - | - |\n| **Query** | 0.2468 | 0.2700| 0.2480 | 0.2600 | 0.2330 | 0.2291 | 0.3523| 0.2857 | 0.2511 | 0.2363 | 0.2697 | 0.2314 |\n| **Value** | 0.1062 | 0.1423 | 0.1238 | 0.1245 | 0.1612 | 0.1226 | 0.1325 | 0.1569 | 0.1867 | 0.1378 | 0.1416 | 0.1472 |\n\nAs shown in the tables above, the $\\sqrt{\\text{SNR}}$ values of the TFB layers, both the Query weights and the Value weights, are significantly less than 1. This highlights the non-trivial nature of the searched low-rank weight noises, indicating that the approximate posterior of TFB constitutes a true Bayesian neural network rather than a simple noise-injection-based uncertainty quantification method. \n\nWe will add this discussion to the revision.\n\n**Q1. Generically how is epsilon chosen here as I imagine that’s a clearly important hyperparameter? If this has to be chosen on a reference task, how do we ensure that this reference task represents the downstream usage tasks?**\n\nThank you for mentioning this. As noted in `lines 173–181,` the choice of $\\epsilon$ can be determined by considering various factors in real-world production environments. Scenarios such as medical diagnosis may require extremely strict standards; in such cases, $\\epsilon$ should be set to a smaller value. In our paper, we adopt a unified $\\epsilon$ value across all domains, which consistently yields universal improvements. \n\nMoreover, inspired by your comment, we conducted additional experiments to analyze the sensitivity of $\\epsilon$. Each experiment was repeated three times with different random seeds, and the `mean (std)` is reported in the table below. The results show that, within a reasonable range, variations in $\\epsilon$ have minimal impact on the final performance in terms of ECE and NLL, thereby demonstrating the robustness of our chosen $\\epsilon$ setting.\n\n**Table D. Sensitivity Analysis on the choice of $\\epsilon$.**\n\n|$\\epsilon$|0.001|0.002|0.003|0.004|0.005|\n|-|-|-|-|-|-|\n|**ACC($\\uparrow$)**|92.18 (0.26)|92.00 (0.30)|91.76 (0.48)|91.76 (0.34)|91.76 (0.34)|\n|**ECE($\\downarrow$)**|3.70 (0.50)|3.16 (0.85)|2.44 (0.50)|2.72 (0.55)|2.98 (0.66)|\n|**NLL($\\downarrow$)**|0.24 (0.01)|0.23 (0.01)|0.23 (0.02)|0.23 (0.01)|0.23 (0.01)|\n\n**Q2. What is the inference computational cost as compared to standard LLM inference?**\n\nAs a Bayesian method that relies on Monte Carlo sampling for expectation estimation, TFB is $N\\times$ more computationally expensive than standard LLM inference, where $N$ represents the number of samples during test time. While this is a general limitation shared by all Bayesian Model Averaging methods, and improving test-time efficiency is not the primary focus of TFB, we explicitly demonstrate in `Section 5.6` that applying Bayesianization only to the last layer can partially mitigate the inference-time computational overhead.\n\n**Q3. It’s not super clear to me what calibration error on textual generation / a text bot should actually mean.**\n\nThis is an excellent question. TFB serves as a simple, effective, and theoretically grounded Bayesian fine-tuning algorithm for different downstream tasks. The label spaces of these tasks (as detailed in `Appendix E.1` and `Table 6`) consist of fixed options (e.g., \"A, B, C, D, E\") or binary judgments (\"True\" and \"False\"). By indexing and normalizing the first token of each option, we transform the original short-form natural language generation output into a probability distribution, making the evaluation of calibration straightforward.\n\n**Q4. Why is the Laplace approximation often better than MAP / MLE in Table 1?**\n\nThanks for mentioning this. This outcome is actually expected. LAP approximates the weight posterior via Hessian estimation, transforming the point-estimate LLM into a Bayesian LLM through a post-training process. As a result, it yields more calibrated predictions than standard estimation methods, as evidenced by the improved NLL and ECE metrics. We will include the clarification above in the revision as suggested. \n\n**References**\n\n[1] Cecere, Nicola, et al. \"Monte Carlo Temperature: a robust sampling strategy for LLM's uncertainty quantification methods.\" arXiv preprint arXiv:2502.18389 (2025)."}, {"review_id": "zTDY1iMwMR", "reviewer": "Reviewer_y21w", "summary": "The paper proposes a training-free approach for Bayesianizing low-rank adapters (LoRA) for large language models (LLM). Bayesianizing low-rank adapters is crucial in estimating uncertainty in LLM responses, therefore of utmost practical importance. The proposed method, TFB, models the posterior over LoRA weights using low-rank isotropic Gaussian distributions, and automatically selects the maximum variance that preserves task performance using an anchor dataset. The authors prove that TFB is theoretically equivalent to generalized variational inference under mild assumptions. Empirical results across multiple LLM backbones and datasets show that TFB improves uncertainty calibration (lower ECE, NLL) while preserving accuracy and significantly reducing complexity compared to prior Bayesian LoRA methods.", "questions": "1. Can you clarify what you mean by \"more compact\" (line 107, 130)? \n2. Can you move limitations to the main text?\n3. How do we select an appropriate anchor dataset for a given model and downstream task?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n1. The overall writing of the paper is very good, quite easy to follow.\n2. TFB works on trained LoRA adapters without retraining or any gradient updates, making it highly practical and scalable.\n3. The theoretical results provided in the paper are solid and relevant.\n4. Experimental setup is pretty rigorous. TFB is shown to work well across different LLMs (LLaMA, Mistral) and LoRA variants (VeRA, PiSSA).\n\n**Weaknesses**\n\n1. Asymptotic inference-time complexity of TFB is not provided in the paper.\n2. TFB is sensitive to tolerance level $\\epsilon$. Some experiments should be conducted to understand the extent of the sensitivity.\n3. TFB requires access to an anchor dataset to search for posterior variance, which might not always be available in zero-shot settings. Also, robustness of TFB across different anchor dataset should be studied.\n4. Restricting to isotropic low-rank Gaussians simplifies inference but might limit modeling capacity compared to full-covariance Bayesian methods. Can we see some ablation to understand the trade-off?\n5. I am not fully convinced with the performance of TFB. Although its obvious that TFB improves uncertainty estimation (NLL and ECE) over MLE, MAP and BloB-mean, it seems to be performing poorly than LAP and BloB in many instances (as reported in Table 1). Statistical significance tests should be conducted.\n6. Can you compare your method with MonteCLoRA [1]? MonteCLoRA seems to be another posterior estimation method, works reliably well on LoRA. It can work in both ad-hoc and post-hoc settings, making it a strong competitor of TFB.\n\n**References**\n\n[1] https://arxiv.org/pdf/2411.04358", "was_revised": true, "final_justification": "The authors have provided additional experimental results, highlighting the effectiveness of their method against the baselines. The additional analyses results were also insightful.", "rebuttal": "Thank you for your constructive feedback. We are glad that you find our proposed TFB `\"highly practical and scalable,\"` our theoretical results `\"solid and relevant,\"` our experimental setup `\"rigorous,\"` showing TFB to `\"work well\"` across different LLMs and LoRA variants, and our paper's writing `\"good\"` and `\"easy to follow.\"` Below we address your comments one by one in detail.\n\n**W1. Asymptotic inference-time complexity of TFB is not provided in the paper.**\n\nFor each forward pass, TFB has the same computational complexity as low-rank adaptation (LoRA), which is minimal, particularly when the rank is low.\n\nThe inference-time complexity of TFB matches that of other Bayesian LLMs based on weight posterior approximation (e.g., BLoB, LoRA ensemble) and introduces an additional $O(N)$ factor compared to standard non-Bayesian counterparts, where $N$ is the number of Monte Carlo samples used at test time.\n\nRegarding the \"asymptotic analysis,\" could you please clarify whether you are referring to the asymptotic behavior as the number of samples increases during test time? If so, because TFB employs Bayesian Model Averaging (BMA) via simple Monte Carlo estimation, it is an unbiased estimator of the true expectation, and its asymptotic behavior follows directly from the Central Limit Theorem (CLT).\n\n\n**W2. TFB is sensitive to tolerance level. Some experiments should be conducted to understand the extent of the sensitivity.**\n\nThank you for mentioning this. Actually, TFB's effectiveness is **not heavily dependent** on $\\epsilon$: as demonstrated in `Table A`, we set $\\epsilon$ to a consistent value corresponding to a relative performance change of 0.3\\% across **all datasets.** This demonstrates TFB's insensitivity to $\\epsilon$ and its universal applicability, while suggesting that further improvements are possible by tailoring $\\epsilon$ to specific datasets.\n\nFurthermore, inspired by your comment, we conducted additional experiments to analyze the sensitivity of $\\epsilon$. Each experiment was repeated three times with different random seeds, and the `mean (std)` is reported in the table below. The findings show that, within a reasonable range, variations in $\\epsilon$ have minimal impact on the final performance in terms of ECE and NLL, thereby demonstrating the robustness of our chosen $\\epsilon$ setting.\n\n**Table A. Sensitivity Analysis on the choice of $\\epsilon$.**\n\n|$\\epsilon$|0.001|0.002|0.003|0.004|0.005|\n|-|-|-|-|-|-|\n|**ACC($\\uparrow$)**|92.18 (0.26)|92.00 (0.30)|91.76 (0.48)|91.76 (0.34)|91.76 (0.34)|\n|**ECE($\\downarrow$)**|3.70 (0.50)|3.16 (0.85)|2.44 (0.50)|2.72 (0.55)|2.98 (0.66)|\n|**NLL($\\downarrow$)**|0.24 (0.01)|0.23 (0.01)|0.23 (0.02)|0.23 (0.01)|0.23 (0.01)|\n\n**W3. TFB requires access to an anchor dataset to search for posterior variance, which might not always be available in zero-shot settings. Also, robustness of TFB across different anchor dataset should be studied.**\n\nThanks for mentioning this. Actually, in practice, a natural choice for the anchor dataset is the (subset of) training set itself, which is *usually available*. \n\nBesides, in this paper, we explicitly identify and address the most significant factors of an anchor dataset:\n\n+ **Training vs. Validation Set as Anchor:** We thoroughly analyze these choices in `Appendix F.4` and `Table 10`.\n+ **Anchor Dataset Size:**\n  + We intentionally use an extremely small anchor dataset (500 samples) throughout all experiments, creating a challenging scenario that disadvantages TFB compared to baselines.\n  + We thoroughly study the performance brought by different anchor dataset sizes in `Appendix F.3` and `Figure 3`, showing robust performance across different sizes.\n+ **Supervised vs. Unsupervised Anchor:** In `Section 3.3`, we choose the most restrictive setting where TFB operates without supervision, a scenario that existing calibration methods cannot handle.\n\nIt is arguably challenging for any Bayesian method to function in a true “zero-shot” scenario, since Bayesian inference fundamentally relies on updating beliefs based on observed data. Our TFB is no exception.\n\nWe will include the discussion above in the revision as suggested. \n\n\n**W4. Restricting to isotropic low-rank Gaussians simplifies inference but might limit modeling capacity compared to full-covariance Bayesian methods. Can we see some ablation to understand the trade-off?**\n\nThis is a good question. Modeling the approximate posterior as a Gaussian with **full covariance** is an appealing idea. However, it expands the distribution family to better approximate the true weight posterior; consequently, it imposes prohibitive computational and memory costs on the Bayesianization process, especially for LLMs.\n\nConsider a standard scenario where the pretrained weight matrix $W_0 \\in \\mathbb{R}^{4096\\times 4096}$ is paired with a pretrained rank-8 LoRA $\\{B\\in \\mathbb{R}^{4096\\times 8}, A\\in \\mathbb{R}^{8\\times 4096}\\}$. Modeling just one low-rank matrix with a full-covariance Gaussian requires $\\frac{1}{2}(8\\times 4096)^2$ parameters (accounting for symmetry), which is $32\\times$ more memory consumption than the non-Bayesian method. \n\nIn `Section 5.3 (TFB Beyond the Low-Rank Isotropic Gaussians)`, we investigate a **full-rank** Gaussian (FR) with a diagonal covariance matrix, an alternative to the **full-covariance** Gaussian. \nThe results demonstrate that our current design of low-rank isotropic Gaussians as approximate posterior achieves both greater parameter-efficiency and superior performance, attaining the best average rank on ACC and ECE, and the second-best on NLL.\nAs `Reviewer BHzf` also noted, this ablation study indicates TFB's choice of low-rank isotropic Gaussians represents `\"probably the best tradeoff performance / compute wise.\"`\n\n**W5. I am not fully convinced with the performance of TFB. Although its obvious that TFB improves uncertainty estimation (NLL and ECE) over MLE, MAP and BloB-mean, it seems to be performing poorly than LAP and BloB in many instances (as reported in Table 1). Statistical significance tests should be conducted.**\n\nThank you for mentioning this. In terms of the significance of TFB's performance, **the following table shows the rank (\"1\" being the best) of ECE and NLL from our main experiments (`Table 1`)** to provide a straightforward comparison between TFB and BLoB (the best-performing baseline). It shows TFB's overall superior performance compared to baselines.\n\n**Table B. Rank (Rk) comparison between TFB and the strongest baseline (BLoB).**\n\n|Metric|Method|WG-S|ARC-C|ARC-E|WG-M|OBQA|BoolQ|Avg. Rk.|\n|-|-|-|-|-|-|-|-|-|\n|**ECE**|**BLoB**|3|**1**|2|3|2|**2**|2.16|\n||**TFB**|**2**|2|**1**|**2**|**1**|3|**1.83**|\n|**NLL**|**BLoB**|3|**1**|**1**|2|2|**1**|1.67|\n||**TFB**|**1**|2|**1**|**1**|**1**|2|**1.33**|\n\nIt is also important to note that the primary purpose of TFB is **NOT** to propose a new state-of-the-art method that tops the current leaderboard. Rather, our empirical evaluation is designed mainly to demonstrate that TFB achieves comparable performance to the baselines in a `\"theoretically sound\"`, `\"empirically simple\"` and efficient way. Besides, TFB achieves the current performances **despite several inherent disadvantages:** \n+ it operates **without supervision**,\n+ uses an **extremely small** anchor dataset,\n+ requires **no gradient estimation**,\n+ and employs **unified settings** across all LoRA checkpoints (MLE, MAP, and BLoB-Mean) throughout our experiments.\n\n**W6. Can you compare your method with MonteCLoRA [1]? MonteCLoRA seems to be another posterior estimation method, works reliably well on LoRA. It can work in both ad-hoc and post-hoc settings, making it a strong competitor of TFB.**\n\nThank you for bringing this work to our attention. We will cite and discuss it in the revision. \n\nWe noticed that the current implementation of MonteCLoRA does not support decoder-only transformer architecture such as LlaMA. As noted in `Section 1, line 30`, reimplementing and tuning a Bayesian LLM requires careful consideration. Although we were unable to include MonteCLoRA’s results during the rebuttal phase, we will revisit this work and incorporate a discussion of the results in the revised version.\n\n\n**Q1. Can you clarify what you mean by \"more compact\" (line 107, 130)?**\n\nWe apologize for the confusion. By `\"compact\"`, we mean `\"parameter-efficient\"`. We will clarify this in the revision. \n\n**Q2. Can you move limitations to the main text?**\n\nThis is a good suggestion. We will move the `Limitations` to the main text in the revision. \n\n**Q3. How do we select an appropriate anchor dataset for a given model and downstream task?**\n\nPlease refer to our **Response to W3**."}, {"review_id": "vGDlNGY3g7", "reviewer": "Reviewer_3SzA", "summary": "The paper considers uncertainty estimation for Large Language Models (LLMs), specifically low-rank adapters for LLMs. It offers a simple approach to do with low-rank matrix found by an existed low-rank adapter, then treating it as matrix of isotropic Gaussians with the same variance. This variance is optimised (as an example binary search is used in the paper). Then predictions are averaged with weights sampled with these Gaussians. Despite simplicity of this approach it is shown to be theoretically grounded and equivalent to generalised variational inference.", "questions": "From the practical point of view, why do you claim TFB to be training free if it still involves optimisation of the variance (which requires a training dataset)?", "limitations": "The authors discuss both the limitations of their work and societal impact of their work.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n*Quality*: The main claims of the paper are supported with good quality experiments and also grounded theoretically linking the proposed approach with generalised variational inference.  \n\n*Clarity*: The paper is well written and easy to follow, after reading it though I have one confusion, please see below. \n\n*Significance*: With the spread of LLM usage uncertainty estimation is very important yet it still remains an open problem, therefore this paper aiming to address it can have a significant impact. \n\n*Originality*: The idea appears to be novel.\n\n**Weaknesses:**\n\n*Clarity and originality*: The paper emphasises that it offers a training-free method, which makes it unique amongst existing methods. However, I am finding myself confused why it is claimed to be training-free. It still requires to find the variance. I can get that on a higher level it is not training, but from the practical point of view, isn’t it just the same as training? \n\n*Originality*: The paper is generally well put into the existing literature but after a quick search I have found this work, for example, Onal, Emre, et al. \"Gaussian stochastic weight averaging for Bayesian low-rank adaptation of large language models.\" arXiv preprint arXiv:2405.03425 (2024) - that seems to be related. \n\n\nOther comments/suggestions:\n1. Line 241. Missing reference for Meta-Llama-3.1-8B\n2. Line 257. The acronym PEFT is not introduced (only introduced later in the Appendix)\n3. Figure 1 is unclear. It has a lot of elements that needs to be described. For example, what fire symbols mean, why $W_0$ is faded for TFB but still contributes to $z$, what different colours mean, etc.\n4. Lemma D.1 is placed in a weird place, namely before the theorem, while the proof of the theorem says “We have the *following* lemma” as if it should be after this text. Moreover, as the reader reads the text and encounters the lemma first, it stands there very confusingly without a proof or reference. \n5. Figure 2. In the first two plots apparently red and blue lines are inseparable. Maybe making one of the lines dashed would be better for visibility. At least text explanation is needed. \n \n \nMinor:\n1. Lines 344-345. “Beyesianization” -> ‘Bayesianization”", "was_revised": true, "final_justification": "After reading all other reviews and authors' responses I am still voting to accept the paper. I was quite impressed how many additional experimental results the authors' were able to include during the rebuttal process, all in favour of their method.", "rebuttal": "Thank you for your constructive feedback. We are glad that you find the problem we solve `\"very important\"`, our paper `\"can have a significant impact\"`, our ideas `\"novel\"`, our main claims `\"supported with good quality experiments\"`, our proposed approach `\"grounded theoretically\"` `\"despite simplicity\"`, and our paper `\"well written and easy to follow\"`. Below we address your comments one by one in detail.\n\n**W1. The paper emphasises that it offers a training-free method, which makes it unique amongst existing methods. However, I am finding myself confused why it is claimed to be training-free. It still requires to find the variance. I can get that on a higher level it is not training, but from the practical point of view, isn’t it just the same as training?**\n\nWe are sorry for the confusion. By \"training-free\", we meant to claim that the variance maximization framework proposed in TFB **does not require any gradient estimation with backpropagation**. This makes TFB more *stable*, more *plug-and-play*, and *easier to use*, especially for researchers and practitioners without a Bayesian background. \n\nFrom another perspective, most training-free frameworks for LLM inference introduce a few hyperparameters. By convention, the community does not consider the process of tuning these hyperparameters as part of \"training.\" In TFB’s case, only a single (hyper)parameter, $\\sigma$, is introduced, accompanied by a principled guideline for selecting its value. Hence, we describe our algorithm as \"training-free\". \n\nWe will include this clarification in the revision as suggested.\n\n**W2. The paper is generally well put into the existing literature but after a quick search I have found this work, for example, Onal, Emre, et al. \"Gaussian stochastic weight averaging for Bayesian low-rank adaptation of large language models.\" arXiv preprint arXiv:2405.03425 (2024) - that seems to be related.**\n\nThanks for bringing this work to our attention. \n\nAfter careful comparison, we find the performance of SWAG [1] close to or worse than our two included baselines LAP and BLoB. This is why we chose LAP and BLoB as they are more representative Bayesian low-rank adaptation works. \n\nWe will be sure to cite and discuss SWAG in our revision.\n\n**Q1. Line 241. Missing reference for Meta-Llama-3.1-8B**\n\nThanks for mentioning this. We will add the reference to Llama-3.1-8B in the revision.\n\n**Q2. Line 257. The acronym PEFT is not introduced (only introduced later in the Appendix)**\n\nThank you. We will explain the acronym of PEFT at its first appearance in the paper. \n\n**Q3. Figure 1 is unclear. It has a lot of elements that needs to be described. For example, what fire symbols mean, why $W_0$ is faded for TFB but still contributes to $z$, what different colours mean, etc.**\n\nThank you for the helpful suggestion. \nIn Figure 1, we use the fire symbol in orange to denote the trainable parameters during the Bayesianization process, the snowflake symbol in blue to denote frozen parameters, and the sample symbol in green to denote Bayesian components (i.e., the approximate posterior). The faded $W_0$ is simply for visual clarity, as it overlaps with the arrow. We will add a legend to Figure 1 to make these distinctions clearer, as suggested.\n\n**Q4. Lemma D.1 is placed in a weird place, namely before the theorem, while the proof of the theorem says “We have the following lemma” as if it should be after this text. Moreover, as the reader reads the text and encounters the lemma first, it stands there very confusingly without a proof or reference.**\n\nThank you for the great advice. We will move Lemma D.1 into the proof of the Theorem 4.1 to make it more coherent and accessible to readers. \n\n**Q5. Figure 2. In the first two plots apparently red and blue lines are inseparable. Maybe making one of the lines dashed would be better for visibility. At least text explanation is needed.**\n\nThis is a good idea. We will change the MLE curve (red) to a dashed line and add explanation in the caption to make it visually clear in the revision.\n\n**Q6. Lines 344-345. “Beyesianization” -> ‘Bayesianization”**\n\nThank you. We will fix the typo. \n\n**References**\n\n[1] Onal, Emre, et al. \"Gaussian stochastic weight averaging for Bayesian low-rank adaptation of large language models.\" arXiv preprint arXiv:2405.03425 (2024)"}, {"review_id": "0VKVF686PV", "reviewer": "Reviewer_3U1X", "summary": "This paper proposes Training-Free Bayesianization (TFB), which makes pre-trained LoRA adapters into Bayesian adapters. It provides a practical way of doing uncertainty quantification in LLMs, and the paper shows that TFB's variance maximization is equivalent to generalized variational inference with some mild conditions. The paper also shows superiority in empirical results.", "questions": "- line 21: \"its internal internal\" is it a typo?\n- line 86: When you denote matrix as $\\boldsymbol{X}=\\left[\\boldsymbol{x}_1, \\cdots, \\boldsymbol{x}_n\\right] \\in \\mathbb{R}^{m \\times n}$ with comma separted, I don't think your definition for notation of the vector, $\\operatorname{vec}(\\boldsymbol{X})=\\left[\\boldsymbol{x}_1^{\\top}, \\boldsymbol{x}_2^{\\top}, \\cdots, \\boldsymbol{x}_n^{\\top}\\right]^{\\top} \\in \\mathbb{R}^{(m n) \\times 1}$, could be confusing\n- line 93: What is \"the\" LoRA layer you are referring to here? I think it would be better to mention what that layer is.\nIs there any difference between \"LoRA\", \"LoRA adapter\", and \"low-rank adapter\"? The authors seem to use different expressions that refer to the same thing.\n-  line 100: What is the \"prior\" here? Are $B, \\theta$ observations? It is hard to understand what \"Bayesianized low-rank adapter's posterior\" is. Why do you call it Bayesian at all?\nAlso, I guess since the authors used $q$ to denote the approximate posterior(?) of $\\operatorname{vec}(\\boldsymbol{W})$, should I also understand $q\\left(A_{i j}\\right)$ to be the approximate posterior of $A_{i j}$?\n- line 126: Is it correct that the mean of the posterior distribution of a random variable is the random variable itself? ( $\\boldsymbol{M}=\\boldsymbol{A}^{\\prime}$ )\n- line 186: What is $P_{\\boldsymbol{\\theta}}(y \\mid \\boldsymbol{x})$ and what is $P\\left(\\boldsymbol{y} \\mid \\boldsymbol{x}, \\boldsymbol{W}_n\\right)$? Is the latter a true probability density function?\n- line 187: Does $\\theta$ include $B$ as well? or is it just $\\{M, \\Omega\\}$ as it was first defined in line 101\n- line 188: It is not valid to justify the selection of such a small number of Monte Carlo samples ( $N=10$ ) by referring to previous work. Do the authors acknowledge possible errors due to the gap between $P_{\\boldsymbol{\\theta}}(y \\mid \\boldsymbol{x})$ and the empirical average?\n- line 210: Missing \\prime for A on the right-hand side? $q\\left(\\boldsymbol{A}^{\\prime} \\mid \\boldsymbol{\\theta}\\right)=\\prod_{i j} \\mathcal{N}\\left(A_{i j} \\mid M_{i j}, \\Omega_{i j}^2\\right)$\n- In table 1, is the standard deviation calculated with $N=10$ samples?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n- This paper brings a novel way to change pretrained LoRA adapters into a Bayesian setting\n- It also provides theoretical grounds for showing the equivalence between TFB's variance maximization and generalized variational inference with mild conditions\n- It covers a decent amount of dataset/benchmark models to compare in the empirical analysis\n\nWeakness\n- Some of the arguments are not clear, and the presentation could be improved. Also, it is a little bit confusing why they call this Bayesianization at the first place.\n- Please refer to the questions for more details", "was_revised": true, "final_justification": "After the rebuttal discussion, I would like to keep my score as it is since there are still things to be revised in the manuscript for an acceptance.", "rebuttal": "We thank you for your constructive feedback. \nWe are glad you find our approach `\"novel\"`, with `\"theoretical grounds\"` under `\"mild conditions\"`, and that our experiments cover `\"a decent amount of dataset/benchmark models\"` and show `\"superiority\"`. Below we address your comments one by one in detail.\n\n**W1.1 Some of the arguments are not clear, and the presentation could be improved.**\n\nWe are sorry for the confusion and will include all clarifications (W1.2-Q10 below) in the revision. \n\n**W1.2 Also, it is a little bit confusing why they call this Bayesianization at the first place.**\n\nWe apologize for the confusion. We follow prior work to use \"Bayesianization\" as the noun form of \"Bayesianize\" [1, 2]. This term aptly describes TFB’s process of transforming a pretrained, deterministic, non-Bayesian LoRA into a Bayesian model.\n\n\n**Q1 \\& Q9. line 21: \"its internal internal\" is it a typo? \\& line 210: Missing \\prime for A on the right-hand side?**\n\nWe are sorry for the typos and will fix them in the revision.\n\n**Q2. line 93: When you denote matrix as ... with comma separted, I don't think your definition for notation of the vector ... could be confusing**\n\nHere in matrix $X=[x_1, \\cdots, x_n]\\in \\mathbb{R}^{m\\times n}$, the column vectors $x\\_i\\in \\mathbb{R}^{m\\times 1}$ are separated by commas, making $x\\_i^\\top \\in \\mathbb{R}^{1\\times m}$ a *row* vector. \nTherefore $[x_1^\\top, \\cdots, x_n^\\top]\\in \\mathbb{R}^{1\\times (mn)}$ is a long *row* vector, and transposing this long row vector leads to a long *column* vector $[x_1^\\top, \\cdots, x_n^\\top]^\\top \\in \\mathbb{R}^{(mn)\\times 1}$.\n\n**Q3. line 93: What is \"the\" LoRA layer you are referring to here? I think it would be better to mention what that layer is. Is there any difference between \"LoRA\", \"LoRA adapter\", and \"low-rank adapter\"? The authors seem to use different expressions that refer to the same thing.**\n\nWe apologize for the confusion.\nIdeally, `\"LoRA adapter\"` and `\"LoRA\"` refer specifically to the two low-rank matrices $\\\\{B, A\\\\}$, while `\"LoRA layer\"` refers to the LoRA adapter together with the pre-trained weights $W_0$. In Line 93, the LoRA layer refers to the operation $BAh$ in Eqn. (1), where z is the output.\n\nIn this paper, we have occasionally `\"LoRA\"`, `\"LoRA layer\"`, and `\"LoRA adapter\"` interchangeablly. \nWe will clarify these distinctions and use the terms more consistently in the revision, as you suggested.\n\n\n\n**Q4. line 100: What is the \"prior\" here? Are $B, \\theta$ observations? It is hard to understand what \"Bayesianized low-rank adapter's posterior\" is. Why do you call it Bayesian at all? Also, I guess since the authors used $q$ to denote the approximate posterior(?) of $\\text{vec}(W)$, should I also understand $q(A_{ij})$ to be the approximate posterior of $A_{ij}$?**\n\n**Prior.** The prior of TFB is a Gaussian distribution with mean equal to the pre-trained weights $W_0$ and covariance $\\sigma_p I$, where $\\sigma_p$ is larger than the local convexity range $\\epsilon_0$. See `Theorem 4.2` and `Eq. 30` for details.\n\n**Approximate Posterior $q(A_{ij})$.** Here in Eq. 2, we introduce LoRA Layer with Asymmetric Bayesianization [2], where $B$ and $\\theta=\\\\{M, \\Omega\\\\}$ are the parameterization, but not observations. As you mentioned, it is appropriate to understand $q(A)$ as the approximate posterior of the LoRA component $A$, and consequently, Eq. 2 is the **equivalent approximate posterior** of the full random weight matrix $W$. \n\n**Why Call It Bayesian.** The reason why the preceding work of BLoB [2] calls it Bayesian is that they define a prior distribution $P(W)$ on the full-weight matrix. They then optimize the ELBO from scratch in a similar way as described in Eq. 12. This ELBO contains the data-likelihood term $\\mathbb{E}_{q(W|\\theta)}[\\log P(D|W)]$ and the KL-divergence term between the approximate posterior and the prior $KL[q(W|\\theta)\\|P(W)]$.\n\n\n**Q5. line 126: Is it correct that the mean of the posterior distribution of a random variable is the random variable itself? ($M=A^\\prime$)**\n\nWe apologize for the overloaded definition of $A'$. \nBased on your advice, we have revised it to: \n\n\"We convert the deterministic matrix $A'$ into a variational distribution with the mean matrix $M=A'$ and the standard deviation matrix $\\Omega$.\"\n\n\n**Q6. line 186: What is $P_\\theta(y|x)$ and what is $P(y|x,W_n)$? Is the latter a true probability density function?**\n\n\nTypically, a Bayesian neural network's output is the expected output $E_{q(W|\\theta)}[P(y|x,W)]$, where $x$ is the input, $y$ is the output, and $q(W|\\theta)$ is the approximate posterior of the LLM parameters $W$. \n\nIn practice, we approximate this expectation through Monte Carlo sampling: $E_{q(W|\\theta)}[P(y|x,W)]\\approx \\frac{1}{N}\\sum_{n=1}^{N} P(y|x, W_n)$, where $W_n\\sim q(W|\\theta)$ is the $n$-th sample drawn from the approximate posterior distribution, $N$ is the total number of samples, and $P(y|x,W_n)$ is the probability distribution on the output variable, depending on the sampled LLM parameters $W_n$.\n\n\n**Q7. line 187: Does $\\theta$ include $B$ as well? or is it just $M,\\Omega$ as it was first defined in line 101**\n\nYou are right. $\\theta=\\\\{M,\\Omega\\\\}$, as the set of parameters modeling the approximate posterior of the weight matrices, does **not** include $B$. This is because the matrix $B$ is kept deterministic and the sampling is performed only over the random variable $A$ (whose mean and standard deviation are $M$ and $\\Omega$, respectively).\n\n\n**Q8. line 188: It is not valid to justify the selection of such a small number of Monte Carlo samples ($N=10$) by referring to previous work. Do the authors acknowledge possible errors due to the gap between $P_\\theta(y|x)$ and the empirical average?**\n\nThis is a good point. Increasing the number of samples generally yields more accurate estimates of the expected output, thereby improving model performance in terms of ECE and NLL. This observation is supported by previous work [2] and partially by our analysis in `Table 5, Section 5.6`. \n\nInspired by your comments, we ran additional experiments and report TFB's performance for sample sizes ranging from $N=0$ to $N=160$ on the WG-S dataset. Each experiment was repeated three times with different random seeds on Llama3.1-8B, and the results are reported as `mean (std)` in the table below:\n\n**Table A. Influence of the number of samples at test time for TFB.**\n\n| | **N=0** | N=1 | N=2| N=3| N=4| N=5| N=10  | N=20| N=40 | N=80 | N=160 |\n|-|-|-|---|---|---|---|---|---|---|---|---|\n| **ACC ($\\uparrow$)**   | 77.72 (0.12) | 74.11 (0.65) | 75.84 (0.62) | 75.81 (0.73) | 77.28 (0.52) | 76.83 (0.53) | 76.96 (0.36) | 77.55 (0.25) | 77.89 (0.38) | 77.55 (0.08) | 78.11 (0.20) |\n| **ECE ($\\downarrow$)** | 15.43 (0.15) | 17.99 (1.06) | 12.68 (0.58) | 10.03 (0.67) | 8.17 (0.79)  | 8.34 (0.22)  | 7.59 (0.34)  | 6.68 (0.27)  | 6.15 (0.54)  | 6.24 (0.66)  | 6.00 (0.23)  |\n| **NLL ($\\downarrow$)** | 0.74 (0.02) | 0.96 (0.02)  | 0.72 (0.01)  | 0.64 (0.01)  | 0.60 (0.01)  | 0.58 (0.01)  | 0.53 (0.01)  | 0.52 (0.01)  | 0.51 (0.01)  | 0.50 (0.01)  | 0.50 (0.01)  |\n\nAs shown in the table, increasing the number of samples at test time generally improves the alignment between the Monte Carlo estimates and expectation in theory, leading to better uncertainty estimation. \n\nOur choice of $N=10$ samples in the paper balances performance and test-time efficiency: it achieves significant improvement in ECE and NLL while introducing acceptable computational overhead. The performance gap between this setting and extremely large $N$ is also mild.\n\n\n**Q10. In table 1, is the standard deviation calculated with $N=10$ samples?**\n\nFor all baselines and our TFB, we run all experiments with $3$ different random seeds and report the standard deviation. $N=10$ is the number of samples we use for Monte Carlo estimation of the Bayesian Model Averaging.\n\n\n**References**\n\n[1] Bacharach, Michael, and Susan Hurley. \"Issues and advances in the foundations of decision theory.\" Foundations of decision theory (1991): 1-38.\n\n[2] Wang, Yibin, et al. \"Blob: Bayesian low-rank adaptation by backpropagation for large language models.\" Advances in Neural Information Processing Systems 37 (2024): 67758-67794."}]

Author rebuttal:

Meta-review: This paper proposes a method to derive Bayesian-style uncertainty quantification for the low-rank adaptation of LLMs, without additional training. Under a Gaussian model of the updates, the method consists of searching for a maximal variance solution for the posterior. The paper theoretically characterizes this approach, using the framework of variational inference and performs an extensive empirical study. The reviewers appreciated the timeliness and practicality of the approach.

Some key points of discussion were about exactly how Bayesian the approach is, considering the strong assumptions that, e.g., limit the prior; the paper should be more transparent about these limitations and in what way its Bayesian formulation is an approximation. More convincing baselines were also requested by the reviewers, and some were provided during the discussion phase; these should be included. Some clarification on the fact that "training-free" does not mean no data use, but rather no gradient-based optimization, should also be part of the paper. The paper is notationally heavy, impeding understanding by those not as versed in the field; a glossary and guidance early in the paper could alleviate this.
Decision: Accept (poster)