C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models
Authors: Amir Hossein Rahmati
Sanket Jantre
Weifeng Zhang
Yucheng Wang
Byung-Jun Yoon
Nathan Urban
Xiaoning Qian
Venue: NeurIPS 2025 poster ()
Keywords: Low-rank Adaptation
Uncertainty Quantification
Probabilistic Modeling
Primary area: probabilistic_methods
TL;DR: We propose Contextual Low-Rank Adaptation (C-LoRA), an uncertainty-aware, parameter-efficient fine-tuning approach by developing lightweight, data-dependent LoRA modules that dynamically adapt uncertainties for robust and calibrated predictions.

Abstract: Low-Rank Adaptation (LoRA) offers a cost-effective solution for fine-tuning large language models (LLMs), but it often produces overconfident predictions in data-scarce few-shot settings. To address this issue, several classical statistical learning approaches have been repurposed for scalable uncertainty-aware LoRA fine-tuning. However, these approaches neglect how input characteristics affect the predictive uncertainty estimates. To address this limitation, we propose Contextual Low-Rank Adaptation (**C-LoRA**) as a novel uncertainty-aware and parameter efficient fine-tuning approach, by developing new lightweight LoRA modules contextualized to each input data sample to dynamically adapt uncertainty estimates. Incorporating data-driven contexts into the parameter posteriors, C-LoRA mitigates overfitting, achieves well-calibrated uncertainties, and yields robust predictions. Extensive experiments on LLaMA2-7B models demonstrate that C-LoRA consistently outperforms the state-of-the-art uncertainty-aware LoRA methods in both uncertainty quantification and model generalization. Ablation studies further confirm the critical role of our contextual modules in capturing sample-specific uncertainties.  C-LoRA sets a new standard for robust, uncertainty-aware LLM fine-tuning in few-shot regimes. Although our experiments are limited to 7B models, our method is architecture-agnostic and, in principle, applies beyond this scale; studying its scaling to larger models remains an open problem. Our code is available at https://github.com/ahra99/c_lora.

C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models
Amir Hossein Rahmati
Sanket Jantre?
Weifeng Zhang' Yucheng Wang'
Byung-Jun Yoon Nathan M. Urban2 Xiaoning Qian
1 Texas A&M University, College Station, TX
Brookhaven National Laboratory, Upton, NY
{amir wwr__hoseein__aaaee weifengzhang, wangyucheng, bjyoon, xqian)@tamu. edu {sjantre, nurban, byoon, xqiani]0bnl. gov
Abstract
Low-Rank Adaptation (LoRA) offers a cost-effective solution for fine-tuning large language models (LLMs), but it often produces overconfident predictions in data- scarce few-shot settings. To address this issue, several classical statistical learning approaches have been repurposed for scalable uncertainty aware LoRA fine-tuning. However, these approaches neglect how input characteristics affect the predictive uncertainty estimates. To address this limitation, propose Contextual Low-Rank Adaptation (C-LoRA) as novel uncertainty-aware and parameter efficient fine- tuning approach, by developing new lightweight LoRA modules contextualized each input data sample to dynamically adapt uncertainty estimates. Incorporating data-driven contexts into the parameter posteriors, C-LoRA mitigates overfitting, achieves well-calibrated uncertainties, and yields robust predictions. Extensive experiments on LLaMAA-7B models demonstrate that C-LoRA consistently out- performs the sate-offthe-aae uncertainty-awant LoRA methods in both uncertainty quantification and model generalization. Ablation studies further confirm the critical role of our contextual modules in capturing sample-specific uncertain- ties. C-LoRA sets a new standard for robust, uncertainty-aware LLM fine-tuning in few-shot regimes. Although our experiments are limited to 7B models, our method cechitecture-gnnoss and, in principle, applies beyond this scale; study- ing its scaling to larger models remains open problem. Our code is available at com aataa////999/arraa
Introduction
Large Language Models (LLMs) 1-7 have shown their promising potential in diverse areas 18 Due to their general-purpost language understanding and generation capabilities with human-level performance 22, fine-tuning LLMs to various downstream tasks has drawn significant attention (19)23). However, when fine-tuned for downstream tasks with limited data, LLMs may hallucinate produce overconfident results, which becomes a serious concern 24
To address this, reliable
estimation of uncertainty has become essential 19 20 28
To enable predictive uncertainty quantification (UQ). probabilistic inference with Bayesian neural network (BNN) has been studied in deep learning 41] where neural network weights are treated random variables with variational inference (VT) approximating the true posterior to provide reliable Adopting these approaches for L.LMs limited by their prohibitive computational and
UQ 42 43
memory costs compared conventional methods 199 all the parameters of an LLM challenging.
20 22 making full Bayesian fine-tuning of
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Parameter Efficient Fine-Tuning Tuning (PEFT) methods, such as Low-rank Adaptation (LoRA), substantially reduce the number of learnable parameters and thereby mitigate the significant computational ex- penses and excessive memory utilization 21 44] This efficiency facilitates employing Bayesian methods for UQ LLMss [19 20 49
Most of the recent studies considered more straightfor-
ward solution, like in 51 where authors considered ensemble methods. [99] proposed a post-hoc Bayesian inference method for the LoRA adapters using Laplace approximation [34]. (20] repurposed mean-field VI based Bayesian framework for LoRA-based fine-tuning jointly estimate the LoRA parameters' variational means and variances [20]. However, the uncertainty stemming from data, aleatoric uncertainty. has not been considered in the existing methods, leading poor performance especially when fine-tuning with limited data. This motivates us to focus on incorporating aleatoric uncertainty for a novel parameter efficient fine-tuning -en-ttning approach in this work.
We propose Contextual Low-Rank Adaptation, C-LoRA, which enables an explicit consideration of aleatoric uncertainty (data uncertainty). This formulation not only leads to faster training. but also provides sample-dependent uncertainty estimation for each layer, which leads beneficial and more favorable uncertainty-aware LLM fine-tuning under small-data scenarios. In particular, we introduce a contextual module for modeling the stochasticity low-dimensional space dependent on the data, which enables a low-cost contextualized estimation of prediction uncertainty in terms of computation and memory usage. Our contributions can be outlined as follows:
We propose a new end-to-end Bayesian framework for scalable uncertainty-awar LLM fine-tuning via contextualized LoRA on data in the lower-dimensional space by a flexible contextual module;
Our framework allows for efficient modeling of aleatoric (data) uncertainty;
We showcase the superiority of C-LoRA regarding both UQ capabilities and generalizability via extensive experiments across different tasks;
Through ablation experiments, we demonstrate the significance of our new contextual module on achieving better UQ while offering competitive accuracy with only minor drops on some tasks.
2
Preliminaries
In this paper, vectors and matrices are denoted by bold lowercase and uppercase letters, respectively. Most of our mathematical notations follow the ones adopted in [2I). 19 and [39
2.1
Low-Rank Adaptation (LoRA)
LoRA, parameter-efficient fine-tuning approach, adapts a pre-trained language model to down- stream tasks [21] It rests on the assumption that the required weight updates have a low intrinsic dimensionality, so LoRA freezes the original weights and instead learns low-rank update matrices. To this end, the modified forward pass becomes:
h (Wo + AW)x= (Wo + BA)x,
where x E R and E Rd are input and output vectors, respectively, and Wa E RRdk represents the frozen pre-trained weights. LoRA inserts two low-rank update factors B E Rdxr and A E Rexk with r < min (d, k). Hence, the number of trainable parameters reduces to r x (d + k) which is considerably lower than d' & in the full matrix. This dramatically cuts storage and computational costs while matching full-matrix fine-tuning performance. Here on, we set d, so that Wa E Bdd
2.2 Bayesian Uncertainty Estimation
Let D [x; y/ I === be dataset of N independent and identically distributed (i.i.d)) observations, where each x; is an input sample and yi is the corresponding output. the Bayesian paradigm, rather than selecting a single best-fit model parameterization, we maintain a distribution over all plausible model parameters. Specifically, given model parameters 0, Bayesian inference aims to characterize the posterior distribution p(0 (D) using Bayes' rule: p(0(D) x p p(D)0) p(O), where p(D(0) is the likelihood of the observed data under parameters 0. and p(0) is the prior distribution reflecting beliefs about 0 before seeing any data.
To generate predictions for a new input x*, Bayesian model averaging, which integrates over the posterior is applied and the intractable integral is approximated using Monte Carlo sampling:
ff ppl''>x [)xx))))))(
M
pv""x" D) =
())((())d) do ~
Σ m==1
ply" xx", 0m); Om ~ p(OD)
(2)

---

Output: Paly" D
Lightweight LoRA
Standard
Wa
Contextual Module
Wg
LLM
A ( ni)
MLP
W/K
Wg
(Mt ni)
Attention
fA
2
Prompt: Answer (Truc/False)
Inpot:
X
C-LoRA
BLoB
Figure A visual representation of our proposed method, Conextual LoRA (C-LoRA) and the Bayesian LoRA by backpropagation (BLoB).
2.3 Bayesian Low-Rank Adaptation
Despite the existence of scalable posterior inference methods, full Bayesian treatment of LLMs remains computationally prohibitive. By restricting Bayesian updates the LoRA parameters, tractable uncertainty quantification scheme can be achieved; however, even Markov chain Monte Carlo (MCMC) over millions of LoRA weights is too costly. As practical compromise, Bayesian LoRA by backpropagation (BLoB) [20) embeds uncertainty estimation directly into fine-tuning via mean-field variational inference LoRA adapters. More specifically, they keep B deterministic in ( and Bayesianize A with a variational distribution g( (A) = =N( MNAllAA
l S2 ). where A and ?
the variational mean and variance estimates, respectively. To this end, they learn the variational distribution parameters by maximizing the Evidence Lower BOund (ELBO):
C' By [log B)] KL({(A) [[
Here, the first term measures the expected negative log-likelihood of the data under the variational posterior, while the second term is the Kullback-Leeblls (KL. divergence between the variational posterior and the prior, acting as a regularization. Using the reparameterization trick, BLoB jointly updates means and variances providing scalable, predictive uncertainty estimates. A visual represen- tation of BLoB framework presented in Figure T
3 Contextual Low-Rank Adaptation (C-LoRA)
We introduce the formulation of Contextual Low-Rank Adaptation (C-LoRA), which enables scal- able, efficient, data-dependent uncertainty quantification the sample level. First, we introduce lightweight LoRA factorization to reduce the computational burden of variational inference in Bayesian LoRA. Based on this factorization, we treat the LoRA weights' stochasticity be data- dependent, and learn the weight distribution parameters via variational Bayesian objective. A step-by-step description C-LoRA is presented in Algorithm T (Appendix
3.1 Lightweight LoRA Factorization
The standard LoRA update (T] introduces two low-rank matrices whose size scales with the frozen weight dimension d, that any Bayesian treatment weeaheent-whether both adapters stochastic only is (as in BLoB)-incure incurs computational cost that grows linearly in d. To break this dependency, insert a low-dimensional matrix E R between B and A. This modification reduces the stochastic parameter complexity to a constant in d, yielding a modified LoRA factorization as follows
h (Wo + AW)x (Wo + BEA)x.
(4)
Compared to the BLoB mean-field VI framework, this new factorization facilitates scalable, lightweight Bayesian LoRA by inferring a distribution over the elements of E while learning and A deterministically, making data dependenttd Bayesianization of LoRA fine-tuning scalable.

---

3.2 Stochastic LoRA Parameterization with Data Dependence
In conventional Bayesian neural networks, the focus on epistemic uncertainties introduced by treating model parameters as random variables drawn from a fixed distribution one that does not depend on the input data. Consequently, while the sampled parameters may vary across data instances, their underlying distribution remains invariant across all samples in the training set. This assumption limits the expressiveness uncertainty estimates, particularly in few-shot LoRA fine-tuning scenarios. To address this, we propose a data-dependent, or contextual, Bayesian fine-tuning paradigm, wherein the distribution of the parameters of low-rank adapters depends on the inputs x, for each data sample (x,, ys). Although one could perform Bayesian inference over B and A LoRA adapters from (T). this scales linearly with the frozen weight dimension d. Instead, by leveraging the lightweight LoRA factorization from we learn the low-dimensional E matrix contextually, yielding the weight updates as
A AW- BE&A
where Ex denotes the data-dependent version of E. To be specific, we take an input-dependent Gaussian variational posterior over Ex, 9 Exx x,) = =N(Pp (x,)) We learn these distri- bution parameters via lightweight per-layer auxiliary contextual modules comprising of small neural networks whose parameters across modules are collectively denoted as y. Such an input-dependent variational posterior resembles amortized inference in Bayesian modeling where 96 serves as an inference network that approximates p( Ex (/,, X;) x p(yi[x,, Ex, (Ex,).
Particularly, consider pre-trained LLM with L layers, each augmented by its own LoRA adapters B', , and A'. Let x denote the output of layer 1 1. We then model E autoregressively by conditioning on {E&, Jitt, leading to 9o (Ex, x,) = I' 96 (E xxx"") Especially, in layer 4, given z' = A 1 E IR" as input, the corresponding contextual module denoted by hi produces the parameters (m'g. S2Ie) of the E'x distribution output. We then draw E&2 conditioned on #e and S2/e. Finally, multiplying B', E, and z' and adding it to W/ x yields the output of the layer 1, x'.
This formulation enables to focus exclusively on heteroscedastic uncertainty-modelinng the variability in given x, x,-with the epistemic uncertainty modeling available via imposing prior on &p or Bayesianizing A or B when needed, thereby isolating and highlighting the advantages of data- data-dependent (heterscedetic) UQ
Contextual module parameterization. We parameterize each layer's 's contextual module with two fully- dully-connected layer neural network whose parameters at layer are denoted as s² In particular, each of these neural networks have inputs, C' hidden units, and 2 x 72 outputs with the nonlinear ReLU activation function connecting the two fully-connected layers.
3.3 Amortized Inference for Contextual LoRA
In our formulation, the variational distribution only conditioned on x,, while yi contributes through the training objective. We then learn the model parameters, &. by maximizing the ELBO of Σ, log p(y)(())))) = =, log p(yxx,, Ex, (Ex,, )dEx, given q (Ex, (x)::
E gos (x.) log Po x,, Ex.) KL xx)) p Exx))
(6)
Here, I denotes the parameters of all B and A adapters across layers. Hence, & (0, (p) denotes all model parameters. Unlike standard variational inference as in (3) our formulation makes the distribution of Ex, input-dependent and replaces the single KL term with sum of N per-sample KL divergences, whose combined impact scales with N. With the objective defined in (6). we adopt a simple and fixed Gaussian prior (Ex) for learning Egg in each layer. The complete learning objective can be equivalently expressed as summation over all samples: L = Exx,)eD L(x, y). Then, we learn the deterministic parameters B and A together represented by j using the expected negative likeeiiodhood (NLL) term while excluding the KL term. This ensures that learning of 0 remains supervised, yielding less noisy gradients per sample computed as
VeL(x, y) = E oo (e> xx V log Po (v|x, Ex).
We approximate this expectation with single Monte Carlo draw Ex q:b (Ex (x) for each sample. Next, to update the auxiliary network parameters yp, which appear in both the NLL and KT. terms of (6) we apply the reparameterization trick (52) to handle random sampling of Ex. Concretely, in

---

each layer, E'2 ~ NN S reparameterized as E u S'g El, where El e Rrxr is sampled from (0, I). Thus, sampling E's from 4s (Ex(x) is equivalent to evaluating deterministic mapping (E, x), enabling gradient computation via
l (0((0(E,x))x) )
VgL((x v) =BE-N(O.)) V
log po (o log
(8)
p(go(E,x))
3.4 Posterior Predictive Inference and Model Complexity
Posterior predictive inference and uncertainty estimation. Once C-LoRA training has converged, we obtain point estimate of the fine-tuned model by replacing Ex with its variational mean and computing p(y|x, M4 (x))).. As shown in Section [ we denote these point estimate models by setting M in our model names. Next, to demonstrate the quality of uncertainty estimation, we sample Ex from its inferred distribution M times and, similar to () approximate the posterior predictive
distribution using Monte Carlo sampling, yielding p(trest Xtest, D) 1/M EMMEme Em=1 P(V x. EX"), where Ex ~ I6 g (Ex|x), For example, for these calibrated models with M = 10 drawn samples in Section we label them with "M=10" in our model names. When "M=0", the posterior mean is used directly for evaluation.
Reducing contextual module complexity through feature reuse. In layer 1, the contextual module is designed to model gs (Exx -1) where x' is derived from natural language; hence, learning features from scratch becomes both non-trivial and computationally expensive. To mitigate this issue, follow 33] and take advantage of the main model by feeding z' into the contextual module instead. Additional computational cost of O(r*), stems from the auxiliary network hig whose fully connected layers have size C = ((22) < d. This overhead minimal compared the main model's per-layer operation cost of (d.). As a result, we can efficiently estimate uncertainty at the sample level in low-dimensional space, sidestepping the costly feature-learning burden within the contextual modules during fine-tuning.
4 Experiments
In this section, we conduct comprehensive experiments to demonstrate the effectiveness of our pro- posed C-LoRA approach for uncertainty quantification in LLMs on reasoning datasets from various domains. We first specify the experimental settings in Section 4.1 including baselines, fine-tuning setting, and evaluation metrics. We then benchmark C-LoRA against existing uncertainty quantifi- cation methods across six reasoning datasets to evaluate overall task performance and uncertainty quality in Section 4.2 Additionally, we test the robustness each method under distribution shift by fine-tuning on OBQÅ and evaluating related OOD datasets in Section 4.3 Lastly, perform an ablation to demonstrate the role of our auxiliary contextual module in Section 4.4
4.1 Experimental Settings
Baselines. We provide performance comparisons of C-LoRA with well-established cutting-edge baselines that include Deep Ensemble [49 51 53 Monte-Carlo dropout (MCD) 229] Bayesian LoRA with Backpropagation (BLoB)
20 a recent mean-field VI approach applied to LoRA
parameters; Laplace-LoRA (LA) [19) which uses Laplace approximation over adapter weights;
and Maximum Posteriori (MAP), representing deterministic point-estimanl baseline.
LLM fine-tuning settings. Following prior work, we evaluate the performance of C-LoRA by fine- tuning 6LaMA2-7B using the PEFT library [54] across six commonsense reasoning benchmarks, including both binary and multiple-choice classification tasks. For each task, the appropriate next- token logits are selected based on the format of the target output, which we include in the Appendix A and the model is trained to maximize the log- likeliood of the correct answer. Following previous works (20 21 28), we apply LoRA to the query, value, and output projection layers using default hyperparameters. All models are fine-tuned with batch size of 4. Baseline methods are trained for 5000 iterations, while C-LoRA is trained for 1500 iterations on all datasets, except for Winogrande- medium (WG-M), which is trained for 2000 iterations. To ensure consistent evaluation, curate a training-validation validation split by reserving 80% the original training set for fine-tuning and holding
Using contextual modules comprising of two fully connected layers with 64 hidden units (C=64) each.
5

---

out the remaining 20% as a validation set. This validation split is used for early stopping and checkpointing based on a metric combining validation accuracy and calibration quality (see Appendix F).
Evaluation metrics. We report results using three key evaluation metrics: accuracy (ACC), expected calibration error (ECE), and negative log-likelihood (NLL). While ACC reflects raw predictive performance. ECE and NLL are widely adopted metries for assessing the quality of uncertainty estimates. ECE measures the alignment between predicted confidence and actual correctness, while NLL evaluates the sharpness and correctness of the model's predicted probability distribution. All reported results are averaged over three independent runs with different random seeds, and we report the mean + standard deviation for each metric.
4.2 Evaluation Results under In-distribution Fine-tuning
In this section, we conduct our assessment of fine-tuning performances under the in-distribution scenario for six common-sense reasoning tasks. These tasks encompass: Winogrande-small (WG-S), Winogrande- medium (WG-M) (55], ARC-Challenge
(ARC-C), ARC-Easy (ARC-E) [56]. Open-
BookQA (57) and Boolq (58). For all the experiments, the same pre-trained LLaMA2-7B was used as the LLM backbone. For BLoB, we kept the same setting in (2)).
Accuracy and Uncertainty Quantification Table Tresennss the performance of various uncertainty quantification methods applied to LoRA-tuned LLaMA2-7B across six common-sense reasoning datasets. As complement, Figure 2 in Appendix K visualizes the results with bar plots. In this set of experiments, our proposed method. C-LoRA, as well as the baseline BLoB, are evaluated under both deterministic (M=0) and stochastic (M=10) posterior sampling settings. While C-LoRA, does not achieve the highest accuracy, it maintains competitive performance across all datasets, with accuracies that are typically within 1-2% of the best methods (c.g., ARC-C: 67.79%, OBQA: 78.26%). Notably, C-LoRA (M=0)- which uses the posterior mean without sampling performs similarly or slightly better than the sampled version on some tasks (e.g.. ARC-C: 69.02% vs. 67.79%), suggesting that the contextual posterior mean alone offers a robust deterministic approximation.
The strength of C-LoRA becomes more apparent when examining uncertainty calibration, as measured by expected calibration error (ECE). Here, C-LoRA (M=10) consistently achieves the lowest or second-lowest ECE across all datasets, including WG-S (6.86%), ARC-C (8.83%), ARC-E (4.27%), WG-M (3.71%). and BoolQ (1.62%). These values are significantly better than those obtained by MAP. MCD, and even Deep Ensemble, indicating that C-LoRA produces confidence estimates that are closely aligned with actual correctness. Even the deterministic variant, C-LoRA (M-0), outperforms BLoB (M=10) in calibration on datasets such as WG-S (7.16% vs. 11.23%) and performs comparably on BoolQ (1.46% vs. 1.46%). A similar pattern holds for negative log-likelihood (NLL), which reflects the quality of probabilisticc predictions. C-LoRA (M=10) achieves among the lowest NLL values in nearly all cases, such as 0.63 on WG-S, and 0.88 on ARC-C, often outperforming both BLoB (M=10) and Deep Ensemble. Additionally, C-LoRA (M=0) matches or slightly outperforms BLoB (M=10) in NLL on several tasks (e.g., ARC-C: 0.89 vs. 0.88), while avoiding the computational overhead of posterior sampling. In summary, while C-LoRA trades off small amount of accuracy compared to non-Ban Bayesian methods, it delivers substantial improvements in calibration and likelihood, which are essential for uncertainty-awee applications.
Post-hoc Calibration with Temperature Scaling To further evaluate calibration, we apply post- hoc temperature scaling (59 to both BLoB and C-LoRA and report the resulting ECE on six common-sense reasoning tasks in Table Bar plots of these results are presented in Figure 3 in Appendix K Even before temperature scaling. C-LoRA (M=10) achieves stronger calibration than BLoB on nearly every dataset. After applying temperature scaling, C-LoRA (M=0, tmp) achieves the best or second-best ECE in 4 of the datasets, including WG-S (4.00%), ), ARC-E (3.86%), ARC-C (6.30%), and VG-M(6 (2.90%). Notably. it outperforms BLoB (M=10, tmp) in 5 out of the 6 datasets, despite using no posterior sampling. Interestingly, C-LoRA (M=10, tmp) does not always show improvement over its uncalibrated variant. On datasets like OBQA and BoolQ, temperature scaling can actually worsen ECE (e.g., from 4.00% to 6.27% on OBQA). This may be due to the fact that posterior sampling already flattens the predictive distribution, and further scaling can overcorrect, leading to underconfident predictions. Also, temperature sealing minimizes NI.., a proper scoring rule, but it does not directly optimize ECE, which is binned, non-differentiable, and improper

---

metric. Consequently, if a model is already reasonably well-calibrate or if its logit distribution exhibits certain local structures, temperature scaling may shift predictions ways that degrade ECE in specific bins. Similar observations have been reported in prior work [60], where temperature scaling was found to degrade classwise ECE. These results suggest that C-LoRA (M=0) provides an effective balance between calibration quality and inference efficiency.
Table 1: Performance comparison of different methods applied LoRA on LLAMA2-7B weights accross six common sense reasoning tasks with validation set split from the training dataset. The best and the second best performances are specified in boldface and via underline respectively.
Datasets
Metric
Method
WG-S
ARC-C
ARC-E
WG-M
OBQA
BeolQ
MAP
60.37+1.04
07.67++..u
85.20-fo.6a
81.60+0o.40 7..57+0.78
87.68-+a.02
MCD
69.06221.4
66 36.66da.30
75.892a 85.49:2..4
81. 1620oa
87.67 47.67
Deep Ensemble
68.98+aa97
68.5 08.574tmm
86.2 86.24-h..m m0
77. 77.30+tm0 o8
82, 20
88.07-to.17
LA
18.18:
17:ta.7
86.30=tu.r
74. 1..
77.53tt
86, 86.45-tu.35
ACC
BLoB (M=0)
69.95
60.25.Lu.s
75.52 85.79.1aas
81.85
86.63.1n.a
BLoB (M=10)
16.55
66.66.1 2
84.56.1 2.20_
73.38 L
81.44
86.63.1..0
C-LORA (M-Ú)
67. 1625.27
69
02.02...03
88..7
772.7H7-2..
81.732:1000
85.81150.77
C-LoRA (M=10)
06.21
27.794
813822077
70. 70.48
78.26:21.1
84.611ux
MAP
20.76+1.0s
30.60++1.2
13.40-to.a
23.01+0.44
15.30++o.11
5.93+
MCD
28.4911.0
29.601ia.7
12.6922..0
20.7310 as
14.311..1
5.13
Deep Emsemble
28.7211.40
27.75.1..6
11.87-1o.10
18.67 29
13.9811.12
5.2411g
LA
11-11+ou
30. 30.51:tu.o
45.83=22m
10. 10.80 aw
35.65t1.14
18.22tuu44
ECE J.
BLoB (M=0)
21. o7
22.5 22.57::..2
10.13:tu.
12. 12.35 an
9.35=1.0
2 2.90
BL.B (M=10)
11.23+1144
10.77
4.20+-.05
4.52 to.91
3.82-1n56
1.46 to
CloRA (M=0)
7.16+2..2
12.28+1.o1
5.75+1.0
6.07-
6.10-11.0
1.46 +..0
C-LoRA (M=10)
6.86 8.862g.m
8.83
4.272:.9
3.71
1.0
1 16622
MAP
2.80
3.07
1.13to.
1.20-tu.12
1.44t..2
0.34+to.00
MCD
2,501...v
2.811
1.13.1o
1.16.a...
1.01
4.3224..00
Deep Ensemble
2.
2,20.1..00
0.91
1.04...00
0.87
0. 4..2
LA
0.62:2..0
1.17fpo
0.97
0.56cuua
0.98
0. 1.-5dzooo
NLL
BLoB (M=0)
0,90
0.632oooz
0.61
0.59
0. 0.31
BL.BB (M=10)
0.00-to.01
0.88
-o88
0.44+
0.54+a.00
0.51+t.ol
0.31 +oo1
C-LoRA (M-0)
0.64
0.80+foo
0.46
0.58
0.63
0.34
C-LoRA (M=10)
0.6314..m
0.88 0.88-g.00 0.00
0.48.4.02
0.57.4..s
0.59
0.35.1g..0
Table 2: Performance comparison of uncertainty quantification based the ECE metric with temperature scaling using validation sets over six common-sense reasoning tasks.
Datasets
Metric
Method
WG.5
ARC-C C
ARC-B
WG-M
OBQA
BoolQ
BLoB (M-0)
21.22
22.57-11.4
10.13+
12.35
9.35--1.08
2.00
BL (M-0, mm))
13.54+L.15
15.47+o.ma
6. 6.10+a..4
5.76+to.n
p.56+
1.80
BLoB (M=10)
11.23d
10.77
4.29
4.522n
3.82
1.46:2.0
BLoB (M=10, imp)
5.10
6.874
5.381
3.18-am
5.71
3.40=2
BCE
C-ooRA (M-))
7.10
12.28
3.76
6.07=4.35
6..0
1.46=u.s
C-LoRA (M=0, tmp)
1.00=1.7
6.58tuue
3.86:t..7
2.90=taa
1.90
1.78=u..
C.LoRA (M=10)
8.83
4.27
3.71
4.00
1.62 1.62-0.44 0.44
C-LoRA (M=10, tmp)
5.18+11.4
6.30
4.35+
3.87+1.77
6.27+1.1
2.67-0.20
4.3 Robustness Under Distribution Shift
Table presents model performance under both in-distribution (OBQA) and out-of-distribttion (OOD) settings with smaller (ARC-E, ARC-C) and larger (Chem, Phy) distribution shifts. Figure Appendix Klillustrates bar plots of the results. While C-LoRA does not achieve the highest accuracy under shift, it demonstrates strong robustness in uncertainty estimation, especially in calibration (ECE) and likelihood (NLL). terms of accuracy, Deep Ensemble and BLoB generally maintain higher predictive performance across OOD datasets. However, C-LoRA (M=10) remains competitive, achieving 65. 09% accuracy on ARC-C and 41. 11.00% on Chem-nearly matching the highest-performing methods. While C-LoRA (M=10) sees slight drop in accuracy under severe shift (e.g., Phy: 31.00%).
7

---

Table Performance comparison on out-of-distribun datasets. The following results are evaluated using LLaMA2-7B fine-tuned on the OBQA dataset in Table
Datasets
Metric
Method
Di-D
Smaller Diat Shift
Larger Dist. Shift
OBQA
ARC-C
ARC-E
Chem
Phy
MAP
17.
73.88
41 10:00
33 23
MCD
68.69 1..4
25.00.12.18
39.00.1..00
31.00
Deep Ensemble
82. +o.1
68.01 +1,25
+4.05+
42.33-+a.0
27.06+ 2..51
ACC1
BLoB (M=0)
81. s3
69. 09.4822..7
76.932
15.33d
26.662..0
BLoB (M=10)
81. 53
67. 17.224
75.1141.a a.
44.001:1..
33.66 33166622.00
C-LoRA (M-0)
81 13
06 4o..6
76.90+-1.s
10.00:+1.11
28 28.00++1.77
C-LoRA (M=10)
78. 20
65.09+t4.05
74. 74.29
41 11.00:2.4
31.00=
MAP
15. 15.30+o
25.51+1.35
20.09
29. 73:t33
36. 2.
MCD
11
23.41=b0744
18.30
28.67
36.
Deep Ensemble
13.98+1.22
20.90+F1.05
16.80+tu.0
16. 16.10-+2
20.74++3.33
BCE
BLoB (M-0)
9.35++1.0
15.76+1..42
11.70++u.02
14. 12++..0
27.35++4.01
BLoB (M=10)
3.824a98
9.774t..1
5.74
12. 12.6014..11
17.56.22.1
C-LoRA (M=0)
3.1
10.0811..00
6.42 1.12-2.0
15.42.1g.m 09
24.42....o
C-LoRA (M=10)
4.00+t.a4
8.83#11.44
6.68
12. 12.49+-..1
18. 18.10tt..a2
MAP
1.04
1.66
1.37 0.04
1.81
1.86.1...0
MCD
1.01 +o.07
1,58 F..8
1.24 0.00
1.81+t0.0
1.88-Fo,10
Deep Ensemble
0.87
1.15dtg.0
0.94=ace
1. 1.45:tmm
1.58
NLL
BLoB (M-0)
0.591..2
0,952to.0
0.72=amm
1.41=am
1.57.to.0
BLoB (M=10)
0.51
0.83+tuu
0.63 0.6
1.35ttuo
1.46+t..z
C-LORA (M-0)
0.33+amm
0.04=maaa
T43:fuo
1.5.f...m
C.LARA (M=10)
0.59.Lo.oz
0.80
0.67
1.30
1.42
it significantly outperforms all baselines in ECE and NLL, indicating more reliable and calibrated uncertainty estimates.
For calibration (ECE), C-LoRA (M=10) achieves the lowest error on Chem (12.49%) and ARC-C (8.83%). This trend also holds for Phy. where C-LoRA (M=10) yields 18.16% which is only slightly worse than BLoB and significantly better than all other baselines. This indicates that C-LoRA remains well-calibrated even the input distribution diverges from training. C-LoRA also excels in negative log- ligelikel (NLL),, achieving the lowest or second-lowest values across nearly all OOD datasets. On Chem and Phy, for example, C-LoRA (M=10) obtains NLL of 1.31 and 1 1.42 respectively, both lower than Deep Ensemble (1.45 and 1.58) and BLoB (1.35 and 1.46) and significantly better than all other baselines. Morcover, it is noteworthy that C-LoRA (M=0), which uses only the posterior mean, also maintains competitive performance. For example, ARC-C and ARC-E, C-LoRA (M=0) achieves 10.08% and 6.42% ECE respectively, compared to BLoB (M=10)'s 9.77% and 5.74%. This suggests that C-LoRA's contextual posterior mean already captures meaningful uncertainty, making it attractive for deployment in compute-constrainge scenarios.
4.4 Ablation Study: Impact of Contextual Module
We also conduct the ablation study investigating the importance of the contextual module for overall performance. Since in C-LoRA focus contextualizing the distribution of matrix E Rr to examine the effect of the contextual module, we compare with the results of fine-tuning the model on different tasks using lightweight factorization introduced in Section 3.1 with mean-field variational inference without the contextual module; we refer to this setting as FE in Tables 4 and
Table 4 underscores the importance of the contextual module by comparing FE and C-LoRA across the six common-sense reasoning tasks under in-distribution scenario. C-LoRA offers significant improvements with respect to ECE across almost all tasks for inference-time sample with both M=0 and M=10, except for BoolQ where it achieves the second best performance. Also, C-LoRA gives the lowest NLL across all tasks.
Furthermore, we assess the effect of the contextual module on generalization. Table 5 provides the performance comparison of the corresponding models fine-tuned on OBQA in Table under similar distribution shifts in Table C-LoRA outperforms FE across all datasets both smaller and larger distribution shifts, with respect to ECE. In addition, C-LoRA offers the lowest NLL on all tasks in both
8

---

distribution shifts. With respect to accuracy, in all tasks C-LoRA achieves a comparable performance, except for Chem where has subpar performance; however, this shortcoming is result of the tradeoff between generalized model with proper uncertainty quantification, and an overconfident model with a poor uncertainty estimation. In summary, these results indicate the significance of our auxiliary contextual module, particularly on enhancing the uncertainty quantification and generalization while maintaining competitive predictive performance.
Table 4: Impact of the contextual module on performance. The performance comparison is conducted across six common-sense reasoning tasks with a validation set split from the training dataset.
Datasets
Metric
Method
WG-S
ARC-C
ARC-E
WG-M
OBQA
BoolQ
FE (M=0)
65-45+-1.0
08. 18.36
86. 86.32=tu..
73
80, 40.46-to.11
84. 24.70tt..45
FE (M=10)
65.06ttas
68, 20:00.2.
85.08=tu.0
72.43+1u
80 30.93+t..0
84.05=tu.az
ACC
C.LoRA (M-0)
67.16
1..0
84.74....20
22.00
81.13.1
85.84.1..67
C-LoRA (M=10)
66.21
67.79++1.27
84.38
70.48+
78.26++2.61
84.64+0o.s1
FE (M-0)
23.76+1.64
27.26-Fo.00
12.23-Ho.25
14.90+1 188
11.23++o.a5
2.57+
FE (M-10)
18.12
19.60-t2.42 42
9.54-rou4z
12.21+1
8.42-t.50
1.34
BCE 1
C-LoRA (M=0)
7.3644..0
12.284..on
5.784: on
6.07
5.10
C-LoRA (M=10)
6.88.1g.0 o9
8,83 20
4.27.11.94
3.71 4.71.11.0 1.00
4.00 4.00-1n.54
1.62
FE (M-0)
1.85
0.85
0.69
0.68 1..8
0.34 Looon
FE (M=10)
0.78
1. 1.35
0.65
0.63
0.60.-
0.34
NLL
C-LoRA (M=0)
0.60
0.89#00.0
0. 0.16:b0.1
0.58
0. 1..30o
0 0.34
C-LoRA (M=10)
0.88=oo0
0.4822
0.57
0 =o.0
0.35
Table 5: Impact of the contextual module generalization. The performance comparison is
conducted on out-of- outto--distribution datasets using LLaMA2-7B fine-tuned on OBQA dataset in Table
Datasets
Metric
Method
In-Dist.
Smaller Diat Shift
Larger Dist. Shift
OBQA
ARC-C C
ARC-E E
Chem
Phy
FE (M=0)
80. 16
68. 21
71.72
15.00
31 31.66tt..11
FE (M=10)
80.93
60.65-F1.00
74.76-to.70
4.70
32.33+11.2
ACC1
C.T.RA (M-0)
81.13+1.0 1.00
60.66-Fo.75
76.90+1..3
1..0
28.00+1..7
C-LoRA (M=10)
26
74.2912.
41. 11.00dba.s
31.022..0
(M=0)
a3
18.44+1.a1
14.05+t..4
20 20.02:t2.26
31. 31775.22
(M=10)
8 8.42+u.a
15.78
11.00
17. 17.61:1122
26. 08.02+t..a
BCE
C.LRRA (M=0)
5.10
10.08
6.42211.44
15.42
24.42 Lg.0n
C-LoRA (M=10)
4.00sb..44
8.83:21.44
6.08=a.71
12. 12.199:1.1
18. 16:11.52
FE (M=0)
0.68+F0.04
1.03-fo.0
0.87=
1.53
1.77
FE (M=10)
0.80++o.0
0.95 0.95+to.u
0.78 0.00
1.46+t00
1.07+
N
C-LORA (M=0)
0.33.amm
0.85
0.642
15.4
C-LoRA (M=10)
0.59.1..5
0.80
0.67-a.co
1.31iam
1.42:tmm
5
Related Works
LLMs have demonstrated remarkable successes across wide range of applications, including code generation, scientific reasoning, and open-domain question answering; however, they are also notori- ous for hallucination-a phenomenon in which the model generates fluent but factually incorrect or unsupported content that is not grounded in the input or external knowledge. To mitigate halluci- nation, a variety of strategies have been proposed, including Reinforcement Learning from Human Feedback (RLHF) to align model outputs with human preferences 6 and Netrieval-Augmenne Generation (RAG) to ground responses in external knowledge 663 64 Additionally, probabilistic
methods such as BNNs 19 20) and conformal prediction 665 66 have been demonstrated to be able to quantify uncertainty and flag unreliable outputs. Recent work also shows the effectiveness of prompt-level interventions 67 68 by carefully crafting instructions and uncertainty-
prompts
to reduce hallucinated content without altering the model architecture.
Our work connects to previous efforts BNNs by introducing a Bayesian formulation over the LoRA layers; however, we extend this line of work by using data-dependent lightweight low-rank

---

factorization, where the intermediate matrix E is modeled as random variables, enabling flexible and context-aware uncertainty estimation to mitigate overconfidence and hallucination in LLMs.
Conclusion and Discussion
In this work, we introduced C-LoRR,, a novel uncertainty-aware parameter-efficiet approach for fine-tuning LLMs in an end-to-end Bayesian framework. C-LoRA facilitates, modeling the aleatoric uncertainty (data uncertainty) via an efficient and scalable contextual module, without compromis- ing the potential for estimating the model uncertainty. Through extensive empirical studies, we demonstrate the superior performance of our method in achieving outstanding accuracy and strong uncertainty quantification capabilities, with consistent generalization across diverse data distributions. Our method underscores the significance of modeling the aleatoric uncertainty in low-data regimes that can lead to substantial gains in generalization and trustworthiness of LLMs.
While C-LoRA establishes a promising foundation for uncertainty-awan parameter-effiient fine- tuning, several avenues remain open for future exploration. One interesting direction is extending the framework to multi-mode modal or multi-task settings, where capturing cross-modal or rask-dependent uncertainty could further enhance robustness and generalization. Another promising avenue inv integrating C-LoRA with active learning or adaptive data acquisition strategies, leveraging its un- certainty estimates to guide sample-efficient model updates. Moreover, exploring hierarchical or structured priors within the Bayesian formulation may offer richer representations both epistemic and alcatoric uncertainty.
7 Limitations
In this study, we restricted our experiments to fine-tuning LLaMA2-7B models, which limits our understanding how C-LoRA scales to larger architectures. Investigating the scaling behavior of C-LoRA on larger models remains open challenge. Furthermore, evaluating uncertainty in language models remains challenging due to the lack of standardized benchmarks with ground-truth uncertainty labels. While our evaluation-basee on held-out test sets and established metrics like ECE and NLL-provides meaningful insight into model calibration and predictive confidence, these metrics offer only an indirect view of uncertainty quality. Developing task-specific benchmarks or numan-in-the-loww protocols for evaluating uncertainty in NLP would be a valuable direction. Moreover, while C-LoRA is motivated by tattaddependen/lllattooie uncertainty, this work does not provide a theoretical guarantee that can disentangle aleatoric and epistemic uncertainty, which we leave to future investigation.
We also note that, as shown throughout our experiments, C-LoRA overall provides a more calibrated predictions and better uncertainty estimation at the expense of marginally reduced predictive perfor- mance which can be preferable in many high-stakes applications. However, in other domains such as spam detection where accuracy is more critical, this trade-off may not be desirable. Hence acknowledge that this trade-off is task dependent and should be considered before deploying C-LoRA (and in general any uncertainty-awarc method).
Finally, our approach employs amortized variational inference to approximate the posterior distri- bution over the LoRA adapter parameters, providing scalable and principled Bayesian treatment suitable for large language models. This variational approximation, based on a Gaussian family and reverse KI. objective, is inherently mode-seeking and may underestimate posterior uncertainty or fail to capture multi-modality in the true posterior. Investigating richer variational families and more comprehensive posterior predictive analyses represents an important direction for future work.
8 Acknowledgment
A.H.R., W.Z., Y.W., and X.N.Q. acknowledge the support from United States National Science Foundation (NSF) grants DMREF-2119101 SHF-2215573, and IIS-2212419. S.J., B.J.Y., N.M.U., and X.N.Q. acknowledge the support from the United States Department of Energy's Office of Science Biological and Environmental Research (BER) program under project B&R# KP1601017 and 1WP#CC140. Many of the numerical experiments were conducted using advanced computing resources provided by Texas A&M High Performance Research Computing.
10

---

References
[1] Ola Shorinwa, Zhiting Mci, Justin Lidard, Allen Z. Ren, and Anirudha Majumdar. A survey uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions, 2024.
[2] Shervin Minace, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2024.
[3] Alec Radford, Jeftrey Wu, Rewon Child, David Luan, Dario Amodci, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAI, 2019. Accessed: 2024-11-15.
[4] Tom B. Brown et al. Language models are few-shot learners. In H. I Larochllle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advancee Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[5] OpenAl et al. GPT-4 technical report, 2024.
[6] Hugo Touvron et al. LLaMA 2: Open foundation and fine-tuned chat models, 2023.
[7] Aaron Grattafiori et al. The LLaMA herd of models, 2024.
[8] Haotian Cui, Chloc Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. scGPT: toward building a foundation model for single-cell multi-omics using generative AL Nature Methods, 21(8):1470-144 1100000-1,,1111111 Aug 2024.
[9] Renqian Luo, Liai Sun, Yingee Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tic- Yan Liu. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6), September 2022.
[10] Karan Singhal, Shekoofeh Azizi, Tao Tu, et al. Publisher correction: Large language models encode clinical knowledge. Nature, 620(7973):E19, 2023.
[11] Xi Yang, Aokun Chen, Nima PourNeejtian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Authony B Costa, Mona G Flores, et al. A large language model for electronic health records. NPJ Digit. Med., 5(1):194 December 2022.
[12] Sanket Jantre, Tianle Wang, Gilchan Park, Kriti Chopra, Nicholas Jeon, Xiaoning Qian, Nathan M Urban, and Byung-Jun Yoon. Uncertainty aware adaptation of large language models for protein-protein interaction analysis. arXiv preprint a7Xiv:2502.06/73, 2025.
[13] Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D) White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nat. Mach. Intell., 6(5):525-535, May 2024.
[14] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: large language model for science, 2022.
[15] Hongyang Yang, Xiao-Yani Yang Liu, and Christina Dan Wang. FinGPT: Open-source financial large language models, 2023.
[16] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski. Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. BloombergGPT: A large language model for finance, 2023.
[17] David Thulke, Yingbo Gao, Petrus Pelser, al. ClimateGPT: Towards Al synthesizing interdisciplinary research climate change, 2024.
[18] Mark Chen, Jerry Tworek, Heewoo Jun, et al. Evaluating large language models trained on code, 2021.
[19] Adam X. Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. In International Conference on Learning Representations (ICLR), 2024.
[20] Yibin Wang, Haizhou Shi, Ligong Han, Dimitris N. Metaxas, and Hao Wang. BLoB: Bayesian low-rank adaptation by backpropagation for large language models. In Neural Information Processing Systems (NeurIPS), 2024.
[21] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[22] Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao, and Anna Rumshisky. Scaling down to scale up: A guide to effiiietr-efliiennt time-tunigg 2024.

---

[23] Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, and Jianfeng Gao. LiST:
Lite prompted self-training makes parameter-effiiient few-shot learners. In Marine Carpuat, Marie- Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Findings of the Association for Computa- tional Linguistics: NAACL 2022, pages 2262-22222 2281. Association for Computational Linguisties, 2022.
[24] Moxin Li, Wenjie Wang, Fuli Feng, Fenghin Zhu, Qifan Wang, and Tat-Seng Chua. Think twice before trusting: Self-detection for large language models through comprehensive answer reflection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11858-1188 11875, Miami, Florida, USA, November 2024. Association for Computational Linguisties.
[25] Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI I.I, Jie Fu, Junxian Hc, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMss In The Twelfth International Conference on Learning Representations, 2024.
[26] Jixuan Leng. Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overconfidence in LLMs: Reward calibration in RL.F. In The Thirteenth International Conference on Learning Representations, 2025.
[27] Guande He, Jianfei Chen, and Jun Zhu. Preserving pre-trained features helps calibrate fine-tuned language models. In The Eleventh International Conference Learning Representations, 2023.
[28] Cristian Mco, Ksenia Sycheva, Anirudh Goyal, and Justin Dauwels. Bayosian-loRA: LoRA based parameter efficient fine-tuning using optimal quantization levels and rank values trough differentiable Bayesian gales. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ ICML 2024), 2024.
[29] Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning, 2016.
[30] Qingping Zhou, Tengchao Yu, Xiaoqun Zhang, and Jinglai Li. Bayesian inference and uncertainty quantification for medical image reconstruction with Poisson data, Journal on Imaging Sciences. 13(1):29-52 52, 2020.
[31] Mohammad Hossein Shaker and Eyke Hüllermeier. Easemble-based uncertainty quantification: Bayesian versus Credal inference. In InnPrre Proceedings 31. workshop computational intelligence. KIT Scientific Publishing, 2021.
[32] Ziyu Wang and Christopher C. Holmes. On uncertainty quantification for near- Bear-Bayes optimal algorithms. In Sixth Symposium on Advances in Approximate Bayesian Inference -on Archival Track, 2024.
[33] Ethan Goan and Clinton Fookes. Bayesian Neural Networks: An Introduction and Survey, page 45-87.
Springer International Publishing. 2020.
[34] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux - effortless Bayesian deep learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 20089-20103. Curran Associates, Inc., 2021.
[35] Shujian Zhang, Xinjie Fan, Bo Chen, and Mingyuan Zhou. Bayesian attention belief networks. In Marina Mcila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 Proceedings of Machine Learning Research, pages 12413-12426. PMIR, 18-24 Jul 2021.
[36] Ruqi Zhang, Chunyuan Li. Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient MCMC for Bayesian deep learning. In International Conference on Learning Representations. 2020.
[37] Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being Bayesian, even just a bit, fixes overconfi- dence in ReLU networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5436-5446. PMLR, 13-18 Jul 2020.
[38] Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[39] Xinjie Fan, Shujian Zhang, Korawat Tanwisuth, Xiaoning Qian, and Mingyuan Zhou. Contextual dropout: An efficient sample-dependen dropout module. In International Conference on Learning Representations. 2021.
12

---

[40] Sanket Jantre, Shrijita Bhattacharya, Nathan M Urban, Byung-Jun Yoon, Tapabrata Maiti, Prasanna Bal- aprakash, and Sandeep Madireddy. Sequential Bayesian neural subnetwork ensembles. wxXiv:2266000944 2022.
[41] Sanket Jantre, Nathan M. Urhan, Xiaoning Qian, and Byung-Jun Yoon. Learning active subspaces for effective and scalable uncertainty quantificationn in deep neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2024.
[42] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe and Variational inference: review for statisticians. Journal the American Statistical Association, 1 12(518):859-87,, 2017.
[43] Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 1(40))1303-134 2013.
[44] Ning Ding, Yujia Qin, Guang Yang, et al. Paramcter-efficintt fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, (3):2220235, Mar 2023.
[45] Haokun Liu, Derek Tam, Mugeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficien fine-tuning is better and cheaper than in-context learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
[46] Zhengxiang Shi and Aldo Lipani. DePT: Decomposed prompt tuning for parameter-efficient fine-tuning. In The Twelfth International Conference on Learning Representations, 2024.
[47] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. crrameter-efficiene tne-tuning for large models: A comprehensive survey. Transactions on Machine Learning Research, 2024.
[48] Ali Edalati,, Marzieh Tahaei. Ivan Kobyzev, Vahid Partovi Nia, James Clark, and Mehdi Rezagholizadeh. KronA: Parameter--effcicnt tuning with Kronecker adapter. In Peyman Passban, Andy Way, and Mehdi Rezagholizadch, editors, Enhancing LLM Performance: Efficacy, Fine-Tuning, and Inference Techniques, pages 49-65 65. Springer Nature Switzerland, 2025.
[49] Oleksandr Balabanov and Hampus Linander. Uncertainty quantification in fine-tuned LLMs using loRA ensembles. In ICLR Workshop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable AI, 2025.
[50] Emre Onal, Klemens Flöge, Emma Caldwell, Arsen Sheverdin, and Vincent Fortuin. Gaussian stochastic weight averaging for Bayesian low-rank adaptation of large language models. In Sixth Symposium on Advances in Approximate Bayesian Inference Non Archival Track, 2024.
[51] Xi Wang, Laurence Aitchison, and Maja Rudolph. LoRA ensembles for large language model fine-tune tuning. 2023.
[52] Durk Kingma. Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. Advances in neural information processing systems, 28, 2015.
[53] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sysiems, volume 30. Curran Associates, Inc., 2017.
[54] Sourab Mangrulkar, Sylvain Gugger. Lysandre Debut, Younes Belkada. Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-ard parameter-offiiien fine-tuning methods. |nttps://github. com http//hgginngg.... peft 2022.
[55] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: an adversarial Winograd schema challenge at scale. Commun. ACM, 64(9):99-106 August 2021.
[56] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning challenge, 2018
[57] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2391-2391, Brussels, Belgium, October November Linguistics.
OoteberNNoemmbr 2018. 8. Association for Computational
13

---

[58] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ:
Exploring the surprising difficulty of natural yes/no questions. In Pruceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 2924-2936, Association for Computational Linguistics, 2019.
[59] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR, 06-11 Aug 2017.
[60] Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song. and Peter Flach. Beyond temperature scaling: Ohtaining well-calibrated multi-class probabilities with dirichlet calibration. In Wallach, H. Larochelle, A. Beygelzimer, d'Alché Buc, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[61] Yuxin Liang, Zhuoyang Song, Hao Wang, and Jiaxing Zhang. Learning to trust your feelings: Leveraging self-awareness in LLMs for hallucination mitigation. In Wenhao Yu, Weijia Shi, Michihiro Yasunaga, Meng Jiang, Chenguang Zhu, Hannaneh Hajishirzi, Luke Zettlemoyer, and Zhihan Zhang, editors. Proceedings of the 3rd Workshop Knowledge Augmented Methods for NLP. pages 44 58, Bangkok, Thailand, August 2024. Association for Computational Linguisties.
[62] Xueru Wen, Jie Lou, Xinyu Lu, Yuqiu Ji, Xinyan Guan, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, Debing Zhang. and Le Sun. On-policy self-alignment with fine-grained knowledge feedback for ballucination mitigation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 5215 -5331, Vienna, Austria, July 2025, Association for Computational Linguistics.
[63] Patrick Lewis, Ethan Perez, Alcksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kütrler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmente generation for knowledge- intensive NLP tasks. Advances neural information processing systems, 33:9459 9474, 2020.
[64] Yunfan Gao, Yun Xiong. Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmeentr generation for large language models: survey. arXiv preprint or 2iv: 10997, 2:1, 2023.
[65] Bhawesh Kumar, Charlic Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam, Conformal prediction with large language models for multi-choice question answering. arXiv preprint 2r0: 18404, 2023.
[66] Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jac Ho Sohn, Tommi Jaakkola, and Regina Barzilay, Conformal language modeling. In B. Kim, Y. Yuc, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Representation Learning, volume 2024, pages 11654-11681. 2024.
[67] Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei I. Liao, Silviu Pitis, Roger Baker Grosse, and Jimmy Ba. Calibrating language models via augmented prompt ensembles. In Workshop Challenges in Deployable Generative Al at International Conference on Machine Learning (ICML), 2023.
[68] Yasin Abbasi-Yadkori, Ilja Kuzborskij. András György, and Csaba Szepesvari. To believe or not to believe your LLM: Iterative prompting for estimating epistemic uncertainty. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[69] Kevin Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023.
[70] Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient pscudo-independent weight perturbations on mini-hatches. In International Conference on Learning Representations, 2018.
[71] Pavel Izmailov, Sharad Vikram, Matthew D Hoffman,, and Andrew Gordon Gordon Wilson. What Bayesian neural network posteriors really like? In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4629-4640. PMLR, 18-24 Jul 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As indicated in the abstract and introduction, we introduce a novel data- dependent, uncertainty-awar approach for fine-tuning LLMs.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the work and the proposed method at the end of the paper in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
Our work does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In Section 4 explain the settings of our experiments. Furthermore, in the Appendix, we clearly describe the hyperparameters that we utilized in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
16

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We provide the experiment details in Section 4 and technical details in Section Moreover, additional information is provided in the Appendix. We will release the code for the proposed approach upon acceptance of our paper.
3
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
aubllic/guudess/CdeeSStiiiss www.:s/wbbbisooooooooooooo. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//abps.cc/pullic/uuiide
ssssssssiiyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experiment settings and the hyperparameters specified in Section 4 and Appendix respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In all our tables, we report the mean and standard deviations for all metrics over three independent runs, across all tasks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the information regarding the computational resources that we used for conducting the experiments in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp/////eerppps.ccpppp///////a iet.
Answer: [Yes]
Justification:
We have reviewed and adhered to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[Yes)
Justification: The proposed method enables the access to trustworthy fine-tuned LLMs for downstream tasks with reliable UQ which is necessary in many safety critical applications. Further discussion the societal impacts of our method is provided in the Appendix.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We do not provide safeguards for our proposed methods as we do not see the potential of high risk of misuse of uncertainty-awer fine-tuning approach for reliable UQ in LLMs.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited papers which provided the pretrained model, as well as the packages and libraries we used. We have also cited the papers for datasets used in this work.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets.paperssit paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We will release the GitHub repository implementing our C-LoRA method with proper instructions upon the acceptance of our paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our study does not involve research with human subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: In this work, LLM was used only for editing and formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM for what should or should not be described.
hatr:::/eearrpp..co//nfeeeeeee
21

---

A Dataset Details
brief summary of the prompt templates used for fine-tuning LLaMA2 on common-seme reasoning tasks is provided in Table 6 More details on the size of training datasets after applying an 80% train-validation split, and the number of labels are gathered in Table 7
Table 6: Prompt templates for fine-tuning on common-sense reasoning tasks
task
prompt
Select one of the choices that answers the following question: [question] Choices: A. {option1 J. B. [option2). Answer:
Winogrande (WG-S/WG-M)
ARC (ARC-C/ARC-E), Openbook QA (OBQA), MMLU
Select of the choices that answers the following question: {question] Choices: A. (choicel). ). B. (choice2), C. [choice3). ).
[choice4]. Answer:
BoolQ
Answer the question with only True or False: [question] Context: [passage]
Table 7: Size of the training dataset after train-validation split and number labels in each dataset
WG-S
ARC-C
ARC-E
WG-M
OBQA
BoolQ
Training datasct size
512
892
1.8K
2.05k
3.97k
1.99k
Number of lahels
2
5
5
2
4
B Temperature Scaling
Temperature scaling is commonly adopted method to convert the predicted probabilities more calibrated values [59 69). To do so, it employs positive constant scaler, T, called the temperature parameter, to soften the softmax values making the distribution less peaky. That is, given the logit vector the new confidence can be expressed as,
q - (z/TT,
where a is the softmax operator: It has heen empirically shown in 59 that such temperature scaling leads to lower Expected Calibration Error (ECE) classification tasks. Here, T can be estimated via optimizing the Negative Log-Likelihood (NLL) on the validation set. Since does not change the maximum of the softmax function, the prediction is unaffected; therefore, does not change the model's 's accuracy while enhancing model's 's calibration.
C Uncertainty Estimation Metrics
Uncertainty quantification is commonly assessed by NLL and ECE in the literature. The summation of the negative expected log probability of predicting the correct label is calculated for NLL. That is, for the model Po. and a dataset size N, NLL is computed as,
N
NLL Σ
log Po(y))
(10)
where y; denotes the correct label. This metric promotes the model to assign a higher probability the correct predictions. For an overconfident model in an incorrect prediction, the probability of correct answer decreases, which leads to an increase in NLL. ECE on the other hand, estimates how well model is calibrated by assessing how close the model's confidence is its accuracy. Specifically, by binning the predictions according to the confidence levels, this metric is calculated by the weighted average of the absolute difference in each bin, that is,
M Bml [acc( N
ECE
(1))
m=1
22

---

where m,) and conf( Bm) indicate the average accuracy and confidence in each bin, Bm
acc(Bm) =
Σ
1(h: = yc), com) = [m Bm
Σ
P(i)),
(12)
Bm
iE
in which [mm] denotes the number of examples in bin m.
D Hyperparameters
Table 8 summarizes the hyperparameters for LoRA fine-tuning of LLaMA2-7B. selected based on prior work [19H2]] and default values provided in PEFT library [54]] Following [20], we optimize the KL term with SGD and linear learning-rate scheduler.
Table 8: Hyperparameters of LoRA fine-tuning of LlaMA2-7B
Hyper-parameter
Value
Optimizer
AdamW
LR Scheduler
Linear
Learning Rate
le-4
Batch Size
Max Sequence Length
300
LoRA
16
LoRA
8
E On the Importance of Flexibility
To study the impact complexity we examine the performance of FE with the case where the matrix E is diagonal matrix with random variables. We refer to this as DE.
Table summarizes the performance of DE and FE with M=0 and M=10 under in-distribution scenario for the six common-senss reasoning tasks. These models are ordered from top to bottom for each metrie, such that each successive model more flexible than the previous one. From Table 9 is clear that the flexibility increases, the performance generally improves. Specifically, in almost all tasks, FE provides a better uncertainty quantification with respect DE, with a lower ECE and NLL. Although FE does not always deliver the highest accuracy, it shows comparable performance across all tasks. Particularly, it provides the best accuracy ARC-C under both deterministic (M=0) and stochastic (M=10) settings. It can be concluded from Table thht that more flexible model yields more reliable uncertainty quantification, with no significant loss predictive accuracy.
To assess the effect of model complexity on generalization, we further compared DE and FE under out-of-distribution scenario, using the models fine-tuned OBQA from Table As Table Table[[Ojiddictte. FE consistently delivers superior uncertainty quantification over all tasks under both smaller and larger distribution shifts. It also delivers the best accuracy in almost all tasks.
These findings from Tables and 10 confirm that richer stochastic structures enhance both general- ization and uncertainty estimation, motivating focus contextualizing the full matrix E.
F Checkpoint Metric
Given our goal achieving a well-calibrated model while maintaining competitive accuracy relative to the states of the art (SOTAs), devise metric that incorporates both accuracy (ACC) and expected calibration error (ECE). Specifically, we define:
C= (1 ACC.a) ECE
with ACCaal and ECE representing the validation accuracy and expected calibration error, respec- tively. During training, the model evaluated using this criterion every 100 steps. We summarize the training algorithm of C-LoRA in Algorithm
23

---

Table 9: Impact of different levels flexibility on performance. Performance comparison is conducted across six common-secr sense reasoning tasks with a validation set split from the training dataset. The best and the second best performances are specified in boldface and via underline respectively.
Datasets
Metric
Method
WG-S
ARC-C
ARC-E
WG-M
OBQA
BoolQ
DE (M-0)
65.98+2.44
26.77+11.8
85.61-ho.36
75.19-t..0
81 11.80
87.47+4..0
(M=10)
66.09:1..0
66.88
85.71
75.19-
22.00
87.444..0
ACC
PE (M=0)
18.352
$6.324
73.17=2.50
60.461..11
a4.
(M=10)
65.06:11.aa
68.20:422.0
85.08
72.43-1r.pe
80.93-tr..0
84.63-ta..a
(M=0)
30.99:11.3
29 29.63:41.06
13.17
21.91
14.19-0.37
4.67
DE (M=10)
27 27.00:bg.0z
26.2644117
11.79:4a.
20.21:1g.00
13.24:20.4
4.41
BCB
(M=0)
23.76411.64
27.26+
12.23-bo.25
14.90-1198
11.233+o.a5
2.57
FE (M=10)
18. 18,1221.52
19.60=2.4
9.54
12.21:1.87
8.42
1.34=p.2
DE: (M=O)
2.14
2.83+
1.17+a.0s
1.18
0.91++o.02
0.32
(M=10)
1.58=0005
2.21:000
1.522.05
0.85
0.32=0.00
NIL.
TE (M-0)
0.80==.077
1864to
0.85
0.00
0.6840.01
0.31
(M=10) 0.78:to.0
1.35
0.65
0.633bo.2
0.60
0.34
Table 10: Impact of different levels of complexity on generalization. Performance comparison is The following results are evaluated using LLaMA2-7B
conducted on outoo--distribttion datasets. fine-tuned on the OBQA dataset. The best and the second best performances are specified in boldface and via underline respectively.
Datasets
Metric
Method
In-Dist.
Smaller Dist. Shift
Larger Dist. Shift
OBQA
ARC-C
ARC-E
Chem
Phy
DE (M=0)
81.80=1106
68, 46
74.75:to.
44 44.33:2..0 as
31.00=
DE (M=10)
82. 62.0622.41
67.90 67
74.80
43.33 57
30.662..51
ACC
FE (M-0)
80.4620.11
68.91 18.91=1.22
74.72:t0.6
45.00 15.00::1.78
31.06=a.61
FE (M=10)
80.93+o..3
66.65-1.00 1.09
74.76+
32.33+1662 46.77
DE (M=0)
14.19
23.51 t12.0
17.74tho.4
23.37-tr.97
34.33+t2a4
DE (M=10)
13.24:f..44
22.31
16.81:1177
21.90: 30
35.14th..8
BCE
FE (M=0)
11. 11.23:2o.s
18 1..51
14.65cto.4
20.02:22.26
31.75=ba.00
FE (M=10)
8. 8.42:ba.50
15.78:
11.691...
17.61
26. 26.02:11.5
(M=0)
0.914ba.22
1.35:bo.0s
1.08tto.aa
1.64
1.86doo.05
DE (M=10)
0.85dba
1.28
1.04
1.62:toee
1.82:b.05
NLL
FE (M=))
0.68
1.03
0.87+
L.8+
1.77
FE (M=10)
0.60
0.95 to.0z
0.78 t.78
1.46 ho.00
1.67
G
Flipout
To speed up the sampling, we apply the Flipout technique-origine originally introduced in 770 and also adopted by [5I C-LoRA to the low-rank matrix E. In particular, having two randomly sampled flipping vectors = {-1, +1)" and t = +1,,+ and considering b; to be the i-th input in mini-batch, the output after flipout is:
0 Wb; Wob, + BEAb; Wob, B(gE + (E .C ((rT)))Ab E~ E~N((011)
H Discussion on Results of Laplace Approximation
For experiments involving fine-tuning via Laplace approximation (LA), we used the publicly released code provided by the authors of [19. However, the results that we attempted to reproduce-reportee repporedd in in our main paper-are are substantially worse than those reported in the original paper [19]. This discrepancy is especially pronounced for the OBQA and Boolq datasets. Our findings
24

---

Algorithm Contextual Low-rank Adaptation (C-LoRA)
Require: Dataset split: Dunin Dvalidation- Diest
Require: contextual module hig. determínistic parameters 0, number of iterations T, evaluation frequency feval, learning rate checkpoint metric
f- 20
2: for + 0 to do
3:
x. y ~ Dirain
4:
x f- x
5:
for 1 + to do
6:
Ug. SR 1 (x!-1)
7:
8 IO,
8:
Ex Flipout(E")
9:
end for
10:
0
0 V/Lxx()))
Eq. Eq.
11:
o - 4 - V (xx.8)
if l mod feral 0 then
13:
Compute validation accuracy and ECE
14:
b- vCC ECEval
15:
b < b then
b Save 0, 10 Record performance on Diest
17:
D Checkpoint best model
19:
end if
20: end
21:
end for
consistent with those reported by the authors of BLoB 200, which suggests that the degradation in performance may stem from sub-optimal MAP solutions that negatively impact the quality of the Laplace approximation-based LoRA fine-tuning.
Additionally, in our experiments, hardware with higher memory capacity than BLoB and our C-LoRA implementations. This might make the practical applicability of LA more challenging considering the scalability compared to other competing methods.
observed that LA is notably memory-intensive, requiring
I Discussion on the choice of prior
In this work, we adopt fixed Gaussian prior which, when paired with variational Gaussian posterior, enables closed-form KL divergence and further facilitates performance comparison across baseline methods. However, due to Monte Carlo approximation of the ELBO, alternative richer distributions could also considered as priors. While richer priors may offer empirical benefits in specific settings, they can introduce additional computational and design challenges. Moreover, due to the highly expressive contextual modules, despite the simplicity, fixed Gaussian prior mainly act as light regularizer rather than as strict inductive bottleneck. Prior work [71] has also shown that Bayesian Model Averaging (BMA) is robust to the choice of prior, with posterior predictive behavior remaining similar across different prior families.
Empirical Study of Contextualized Variance
To further illustrate that our model captures input-depender uncertainty, we analyze the predictive variance behavior of the LM head for three OBQA questions, each sharing the same correct label (option "C"). For each question, we extract the predicted variance matrix from the final layer, compute the mean variance across tokens, and report the & norm of the resulting vector to summarize overall uncertainty.
25

---

Table 11: Example questions and corresponding variance magnitudes. Despite sharing the same label, the model assigns distinct variances to each question, reflecting input-dependent uncertainty.
Question
Variance &z norm
QI: What would light bounce off of when it hits it?
3.7661
Q2: A mother births what?
3.6792
Q3: What happens when animals in hot environments are active?
3,9443
Despite sharing the same label, the model assigns distinct variances to each question, reflecting differences in ambiguity and specificity of the input. This confirms that C-LoRA modulates its predictive distribution based on input semantics, not merely on class labels.
We further examined whether the learned variance meaningfully affects the model's representations. Specifically, we computed the l norm of the difference between the last-token embedding at M = 0 and the mean embedding across M = 10 stochastic samples. For the three representative examples (QI-Q3), the differences ranged from 6.18 to 7.25, while the embedding norms themselves were approximately 120-122, corresponding to relative change of about 5-6%. In embedding space, this level of perturbation is non-trivial and indicates meaningful input-dependent variability introduced by the contextualized variance modules.
We extended this analysis a broader set of examples, comparing 8 correct and 8 incorrect predictions. For each, we computed the l norm of the difference between the deterministic embedding (M = 0) and the mean of embeddings from M 10 samples, then averaged across each group. Correct predictions had an average difference of 6.7 + 1.09, while incorrect predictions showed higher variability at 8. 2. This systematic increase embedding perturbation for incorrect predictions suggests that C-LoRA's learned variance corresponds greater uncertainty for ambiguous difficult inputs, further supporting its interpretation as modeling input-specific uncertainty.
K
Visualization of Results
For a visual performance comparison, we present the results under in-distribution scenario, after temperature scaling, and the results under outoo--disttibution scenario, reported in Tables 2 and using bar plots.
prrusentsssnnts bar plots summarizing the results reported in Table 1 As discussed in Section all methods exhibit similar predictive performance overall, while C-LoRA and BLoB (M 10) achieve superior uncertainty quantification, as measured by ECE and NLL. Figure also
also illustrates the effect As noted Section 4.2 temperature visualizes the out-of-distribution results from Table 3 via bar plots. Consistent with in in-distribution results, performance degrades as distribution shifts; however, predictive performance remains largely similar across methods, with C-LoRA and BLoB demonstrating more robust uncertainty quantification according to ECE and NLL. As mentioned earlier, it worth noting that, although C-LoRA offers similar predictive performance overall, it can underperform on certain datasets in exchange for better uncertainty quantification.
of temperature scaling on calibration, corresponding to Table 2 scaling substantially improves calibration. Finally. Figure 4
MAP
10
MCD
Accuracy 1
Deep Ensemble
LA
(M=0)
BLoB [M]10)
C-LoRA (M=C)
C-LoRA (M-10)
WG355
ARC-C
A
WGG-M
OBQA
Bcolo
HGG5
ARC-C
ARC-E
WG-M
OBGA
Boolo
WG-5
ARC-C
ARC-E
WG-M
OBQA
Buolo
Figure 2: Visualization of the results in Table
26

---

BLoB 01=0)
BLaB (M-10)
C LoRA (m=0)
LuRA imm0 0.0mp.
E02
LoRA (m-10)
C-LOA tm.
WG-5
ARCLL
ARC-E
WU-M
OBUA
Boolt
Figure 3: Visualization of the results in Table
2.00
MCD
Deep Ensemble
Accuracy 1
oB (M-D)
BLoB (M=10)
(M-0)
C-LoRA (M=10)
050
622
CBCA
ARC C
ARC-E
Chem
Phy
CBR4
ARC-C
ARC E
Chem
Phy
OBQA
ARC-C
ARC-E
Chem
Plyy
Figure 4: Visualization of the results Table
L Broader Impacts
Our uncertainty- aware fine-tuning framework integrating uncertainty quantification in LLMs could enhance the safety, reliability, and trustworthiness of AI systems deployed in high-stakes settings such as medical diagnosis, atmospheric modeling, and autonomous navigation, where unrecognized model approach explicitly models aleatoric data
emros can have major consequences. Moreover, uncertainty, it is particularly well-suited for low-resource tasks and rare-event prediction, providing access robust LLM-based AI tools in fields ranging from global health to societal governance and policy-making.
We anticipate minimal additional computational overhead compared to standard fine-tuning, and because our framework is compatible with existing Bayesian extensions for epistemic uncertainty, offers clear path toward unified uncertainty quantification without introducing undue complexity. We do not foresee any negative socictal impacts beyond those already inherent to large-scale model deployment, and we believe that equipping models with calibrated confidence measures is essential step toward more ethical, accountable, and human-centered AI.
M Additional Information
In our contextual module, we C, the number of hidden units, 64 in order to ensure sufficient expressiveness for learning meaningful features. Also, to have a robust performance, the variance output, 2gg, uses a sigmoid activation function; however, we do not use any activation function for the mean Pg. All the experiments for almost all methods were conducted using 1 NVIDIA A100 GPU with 40 GB memory except for BLoB, for which we used 2 NVIDIA A100 with 40 GB memory. Also for LA we used NVIDIA L40S GPU with 48 GB memory due to its memory demands.
27


Reviews:
[{"review_id": "PiiPsNAjFs", "reviewer": "Reviewer_Hu4h", "summary": "This paper introduces C-LORA (Contextual Low-Rank Adaptation), a novel parameter-efficient fine-tuning approach designed for uncertainty estimation in Large Language Models (LLMs). The core problem addressed is the overconfident predictions often produced by LoRA in data-scarce few-shot settings, and the neglect of input characteristics in existing scalable uncertainty-aware LoRA methods. C-LORA tackles this by developing lightweight LoRA modules that are contextualized to each input data sample, allowing for dynamic adaptation of uncertainty estimates. The method incorporates data-driven contexts into parameter posteriors, which aims to mitigate overfitting, achieve well-calibrated uncertainties, and yield robust predictions. The paper highlights C-LORA's ability to explicitly model aleatoric (data) uncertainty, which is a key differentiator from prior works. Extensive experiments across various common-sense reasoning datasets demonstrate C-LORA's superior performance in both uncertainty quantification (UQ) and model generalization compared to state-of-the-art uncertainty-aware LoRA methods. Ablation studies further confirm the critical role of the contextual modules in capturing sample-specific uncertainties.", "questions": "1.\tImpact of Contextual Module Complexity: The paper mentions the auxiliary contextual module has C hidden units, with C=O(r2 ). Could the authors provide more insights or a small ablation on how varying C (or the number of layers in the contextual module) might affect performance (both UQ and accuracy) and computational overhead? This would help understand the module's sensitivity to its own architecture.\n\n2.\tTrade-off between Accuracy and UQ: While the paper acknowledges the trade-off, could the authors elaborate on scenarios or tasks where this accuracy difference (e.g., 1-2% drop) might be a critical limiting factor for C-LORA's deployment, despite its superior UQ? For instance, in applications where very high accuracy is paramount and minor calibration improvements are acceptable.\n\n3.\tPrior Distribution Choice for Ex : The paper states adopting a \"simple and fixed Gaussian prior p(Ex)\" for learning Ex. Could the authors discuss the sensitivity of C-LORA's performance to the choice of this prior? Are there alternative prior choices that might be more theoretically grounded or empirically beneficial, especially for different data distributions or task types?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tOriginality & Significance (High): The paper introduces a novel and important concept of \"contextualized\" LoRA, specifically for dynamic uncertainty estimation in LLMs. The explicit modeling of aleatoric uncertainty, which is often overlooked in existing Bayesian PEFT methods, is a significant contribution, especially in low-data regimes. This innovation sets a new standard for robust, uncertainty-aware LLM fine-tuning.\n\n2.\tQuality & Effectiveness in UQ (High): C-LORA consistently outperforms state-of-the-art uncertainty-aware LoRA methods in uncertainty quantification, as measured by ECE and NLL. This is a strong indicator of the method's ability to produce well-calibrated and reliable uncertainty estimates. The results in Table 1 and Table 2 clearly demonstrate this superiority.\n\n3.\tRobustness under Distribution Shift (High): The experiments on out-of-distribution (OOD) datasets show C-LORA's strong robustness in uncertainty estimation (ECE and NLL), even when the input distribution diverges from training. This is crucial for real-world applications where models might encounter unseen data.\n\n4.\tComprehensive Experimental Validation: The paper provides extensive experiments across six common-sense reasoning tasks, including both in-distribution and out-of-distribution scenarios. The comparison with a good set of baselines (Deep Ensemble, MCD, BLoB, LA, MAP) is thorough.\n\nWeaknesses:\n\n1.\tCode and Data Accessibility (Moderate): The paper states that code will be released upon acceptance. While this is a common practice, the lack of immediate access to the code for verification during the review process is a minor drawback for reproducibility.\n\n2.\tMarginally Lower Accuracy in Some Cases (Minor): While C-LORA excels in UQ, it is noted that its accuracy is sometimes competitive but not always the highest, typically within 1-2% of the best methods. Although the paper frames this as a trade-off for better UQ, it's worth acknowledging. For example, in Table 3 (Chem dataset with M=0) accuracy is noticeably lower than some baselines.\n\n3.\tTemperature Scaling Inconsistency (Minor): The observation that temperature scaling can sometimes worsen ECE for C-LORA (M=10) on certain datasets (e.g., OBQA and BoolQ) is interesting. While the explanation suggests posterior sampling already flattens the distribution, a deeper analysis or alternative post-hoc calibration strategies could further strengthen the calibration aspect.", "was_revised": true, "final_justification": "The authors have answered all my questions. I recommend this paper for acceptance.", "rebuttal": "Thank you for your thoughtful and constructive feedback. We are glad you found our approach novel, effective in uncertainty quantification, and robust across distribution shifts, and our experimental validations comprehensive. We also appreciate your questions and suggestions regarding model complexity, performance trade-offs, and prior sensitivity, which we address below.\n\n\n\n**W1. Code and data availability.**\n\nWe understand the reviewer’s concern, but this year’s NeurIPS rebuttal guidelines do not permit providing any links to external pages including an anonymized code repository. We will definitely release our code upon acceptance. Additionally, we have included all relevant hyper-parameters, hardware specifications, and implementation details in the appendix (sections D and J).\n\n**W2. Marginally Lower Accuracy in Some Cases.**\n\nWe have acknowledged the trade-off between predictive performance and uncertainty estimation in the experiments section where we provided discussions based on our results. We will further highlight this point in the introduction.\n\n\n**W3. Temperature scaling inconsistency.**\n\nHere, we conducted a deeper analysis using predictive entropy and mutual information to investigate different types of uncertainty as done in recent uncertainty disentanglement efforts. Particularly, we adopt the decomposition of predictive entropy (PE) from as in [1] (Section 2.2) and [2] (Section 3.4), where the expected entropy (EE) captures data uncertainty (aleatoric), and mutual information (MI) reflects epistemic uncertainty. Note that this decomposition follows the relation PE=MI+EE. \n\nThe reported results in the table show EE and MI computed at $M=10$ across both in-distribution (OBQA) and out-of-distribution (ARC-C, ARC-E, Chem, Phy) test sets, with the model trained on OBQA for all experiments. The proportion of aleatoric uncertainty out of total uncertainty captured by our model is consistently higher than the one captured by BLoB (on average, around 0.99 and 0.82 for C-LoRA and BLoB, respectively), demonstrating a stronger focus on modeling input-dependent uncertainty.\n\nSince our model primarily captures aleatoric uncertainty, it is often well-calibrated in noisy or ambiguous regions. Temperature scaling, being a global post-hoc adjustment that does not account for input-dependent variability, may therefore have a limited effect or even slightly worse calibration in such settings. Hence, temperature scaling might need to be applied with some caution when input-dependent uncertainty modeling is involved. \n\nMoreover, since temperature scaling optimizes NLL but not ECE—which is non-differentiable and bin-based—it may inadvertently harm calibration in certain regions. Prior work [3] has reported similar findings, where temperature scaling can degrade classwise-ECE (Appendix, section C.1). \n\nFinally, we want to note that temperature scaling was used only as a standard post-hoc tool to potentially improve uncertainty estimation capabilities and is not part of our model.\n\n\n**EE and MI Scores (C-LoRA vs BLoB, M=10)**\n\n|Metric | Method | OBQA | ARC-C | ARC-E | Chem  | Phy   |\n|--------|--------|------|-------|-------|-------|-------|\n| EE     | C-LoRA    | 0.538 | 0.705 | 0.583 | 1.155 | 1.185 |\n|        | BLoB   | 0.349 | 0.480 | 0.388 | 0.999 | 1.018 |\n| MI     | C-LoRA    | 0.010 | 0.011 | 0.008 | 0.007 | 0.008 |\n|        | BLoB   | 0.111 | 0.132 | 0.113 | 0.114 | 0.116 |\n\n\n\n\n**Q1. Could the authors provide more insights or a small ablation on how varying C (or the number of layers in the contextual module) might affect performance (both UQ and accuracy) and computational overhead?** \n\n|    Metric   | Simple | Original | Complex | Complex + |\n|-------|---------------|----------|---------|-----------|\n| ACC (↑)   | 66.43         | 69.02    | 65.98   | 66.1      |\n| ECE (↓)   | 13.06         | 12.28    | 9.61    | 8.87      |\n| NLL (↓)   | 0.92          | 0.89     | 0.90     | 0.87      |\n\n\nIn the original submission, we used a fixed architecture: a single hidden layer with 64 neurons (To clarify- In footnote on page 5, “two fully connected layers” denotes one hidden layer and an output layer in each contextual module). To explore the impact of model complexity, we conducted additional ablation experiments on the ARC-C dataset with $M = 0$, comparing the original model to its variants of varying complexity: 1) **`Simple`**: 1 hidden layer with 4 neurons, 2) **`Complex`**: 1 hidden layer with 100 neurons, and 3) **`Complex+`**: 2 hidden layers with 64 neurons each. Results (averaged over 3 random runs, on 1 A100 40 GB GPU) show that increasing the contextual module’s complexity leads to improvements in ECE and NLL, indicating better uncertainty quantification, at the expense of prediction accuracy. However, the original architecture offers strong predictive performance and reasonably good calibration with favorable NLL, making it a solid choice for the contextual module. That said, for tasks where better calibration and lower NLL are prioritized, a more complex architecture may be more suitable.\n\n In terms of computational overhead, all models had similar training time (90–100 minutes), suggesting that the contextual module’s architecture has minimal impact on runtime, compared to the overall LoRA fine-tuning procedure in our setup.\n\n\n**Q2. Could the authors elaborate on scenarios or tasks where this accuracy difference might be a critical limiting factor for C-LORAs deployment, despite its superior UQ?**\n\nIn many high-stakes applications—such as medical diagnosis, autonomous driving, financial risk modeling, and weather forecasting—well-calibrated uncertainty estimates are essential, and slight reductions in accuracy may be acceptable when they improve reliability and safety. However, in domains such as spam detection, malware classification, or other security-sensitive tasks, accuracy is often the top priority, and drop in performance may outweigh the benefits of better calibration. \n\nWe agree this trade-off is task-dependent and should be carefully considered when evaluating C-LoRA (and in general any uncertainty aware method) for deployment. We will include this in the conclusion and limitations section.\n\n\n**Q3. Could the authors discuss the sensitivity of C-LORA's performance to the choice of this prior? Are there alternative prior choices that might be more theoretically grounded or empirically beneficial?**\n\nWe adopt a fixed Gaussian prior for $E_x$ which when paired with a Gaussian approximate posterior $q(E_x)$ enables a closed-form KL divergence, which reduces loss gradient variance and stabilizes training. Because our ELBO objective is estimated by Monte Carlo sampling (as NLL term does not have closed-form expression), we could also support alternative priors (e.g. Gaussian mixtures, Laplace, VampPrior, or normalizing-flow priors) as long as their KL is tractable or can be unbiasedly estimated. While richer priors might offer empirical benefits in some settings, they also introduce additional computational and design challenges. In practice, our contextual posterior $q(E_x)$ is highly expressive (parameterized by a neural network), so the Gaussian prior mainly acts as a light regularizer rather than a strict inductive bottleneck.\n\n\nMoreover, prior work [4] has shown that Bayesian Model Averaging (BMA) is robust to the choice of prior scale and that posterior predictive behavior remains similar across prior families such as Gaussians, mixture of Gaussians, and logistic priors. \n\nFinally, we want to point out that our aim is to compare inference and performance across methods under a consistent modeling setup. Since prior works- BLoB and LAP both use a simple Gaussian prior, this choice facilitates fair comparison.\n\nWe will include a brief discussion about the aforementioned points in the final version of the paper.\n\nWe hope that we have addressed your concerns and look forward to the discussions if there are additional questions.\n\n[1] Mucsányi, Bálint, et al. “Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks” NeurIPS 2024\n\n[2] Jishnu Mukhoti and Yarin Gal. “Evaluating Bayesian Deep Learning Methods for Semantic Segmentation”. arXiv preprint, 2018.\n\n[3] Kull, Meelis, et al. \"Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration\", NeurIPS 2019\n\n[4] Izmailov, Pavel, et al. “What Are Bayesian Neural Network Posteriors Really Like?” ICML PMLR 2021"}, {"review_id": "gEsOkmtLZO", "reviewer": "Reviewer_wZ8F", "summary": "This paper addresses the problem of uncertainty quantification in large language models, introducing a new method that proposes to take into account the data at hand to represent the aleatoric uncertainty. This sets it apart from many previous UC methods for LLMs which tended to only train a weight posterior, is less suited to representing uncertainty relating to the specific input sequence at test time. Contextual Low-Rank Adaptation (C-LoRA) modifies the usual $\\mathbf{W} _{0} + \\mathbf{B}\\mathbf{A}$ LoRA adaptors by inserting a small $\\mathbf{E} \\in \\mathbb{R}^{r \\times r}$ matrix, which is independent of the underlying model weight size $d$ giving $\\mathbf{W} _{0} + \\mathbf{BEA}$. This allows $\\mathbf{E}$ to be treated as a latent random variable, with overhead $O(r^2)$ per layer independent of $d$, whose mean and variance are predicted using a small context module from the current hidden state, or the input sequence at layer $l = 0$. This context module is trained to output distribution parameters $\\phi$ for $q _{\\phi}(\\mathbf{E} _{\\mathbf{x} _i} \\vert \\mathbf{x} _{i})$ such that when sampled the LLM accurately represents the aleatoric uncertainty: with $\\mathbf{E}$ being sharply peaked for certain inputs $\\mathbf{x} _{i}$, and broad on ambiguous inputs. Experiments on standard datasets and under distribution shift show that C-LoRA substantially improve calibration over many baseline methods, even when using few MC samples at inference time. Ablations show that removing this contextual conditioning also removes these gains. C-LoRA represents a useful contribution to the field", "questions": "- The $r \\times r$ (i.e. $16 \\times 16$) matrix at each layer results in a very small set of parameters to infer. Is this sufficient scale to meaningfully learn to represent data uncertainty? When does $r$ become too small? Is this why you train C-LoRA for only 1500 iterations while the baselines are trained for 5000?\n- Could you please characterise the computational cost of C-LoRA in terms of wall clock time for training and inference on the same computational resources. How does the fine-tuning time compare between vanilla LoRA and C-LoRA, and at inference time how much longer does it take to make a prediction?\n- You focus exclusively on aleatoric uncertainty and not epistemic uncertainty. Could you quantify whether there is anything to be gained from a Bayesian treatment of the LoRA parameters or even the contextual network's parameters?", "limitations": "The work largely improves upon existing baseline methods for uncertainty quantification in LLMs, and while it makes valuable progress towards improving the calibration of this class of methods, it suffers from the same limitation of being constrained to single-token predictions and does not make progress on how to represent uncertainty in more modern LLM use-cases. Language models have grown in popularity mainly due to their use as generative models and not discriminative models. While the single-token constraint still encapsulates some non-trivial subset of LLM use-cases (multiple-choice question answering, LLM-as-judge, critic models in RL, point predictions for judgemental forecasting, etc), the current work only considers multiple-choice question answering tasks. Moreover, this paper does not address ways in which we might extend the uncertainty quantification to multi-token generative applications, such as reasoning traces, uncertainty-aware decoding mechanisms, or simply articulating the uncertainty in natural language, which are possibly far more pertinent to modern LLM deployments and use-cases. The work could be improved by formulating new datasets or tasks that move the field of UQ for LLMs away from MCQA and towards more modern and general use-cases.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Within the small sub-field of probabilistic treatments of LLMs for uncertainty quantification, the paper makes a good contribution. Earlier attempts straightforwardly applied methods from Bayesian deep learning, which were themselves developed in simpler settings: where the model inputs were straightforward sensor measurements or observations, and where data-dependent output uncertainty could be adequately expressed by learning a weight posterior distribution to have good model evidence (i.e. yielding appropriate output uncertainty estimates after being pushed through the model), based on the data and patterns observed during training. However, in LLMs and sequence models, the input may be a complicated question requiring extended reasoning or nuanced understanding. Expressing the output uncertainty using the weight posterior alone is a rather blunt tool, and this paper proposes a sensible approach to modulate the output uncertainty based on the data point at hand.\n\nFollowing prior work attempting to perform Bayesian inference over LLM parameters, the authors freeze the model weights and operate over the adapter to reduce the otherwise prohibitive computational cost. However, they go further to remove the direct dependence on the base model's dimension, inserting a fixed-dimension $r \\times r$ matrix, where $r \\ll d$, capping the number of parameters to infer as $O(r^2)$. To amortize the inference of the variational parameters $\\phi$, the authors propose a simple 2-layer ReLU network with parameters $\\varphi$ which they call a 'contextual module', that initially accepts the input $\\mathbf{x} _{i}$ at layer $l = 0$, and the output of layer $l - 1$ for subsequent layers. The clarity of this section may be improved for readers unfamiliar with variational inference by making more explicit how $\\phi$ and $\\varphi$ relate. The authors leave the use of diagonal Gaussian distributions throughout relatively unexamined: while this is a sensible choice on account of its cheap computational properties, at higher ranks ignoring correlations inside $\\mathbf{E}$ could matter..\n\nTraining proceeds using straightforward VI objectives, with the authors choosing to directly train the LoRA parameters $\\mathbf{\\theta} = \\{\\mathbf{A}, \\mathbf{B}\\}$ rather than use a Bayesian treatment. It is unclear whether the authors train $\\mathbf{\\theta}$ in a discrete training phase before $\\varphi$, or in an interleaved manner. It would be good to clarify this.\n\nThe authors re-use a standard set of relatively diverse benchmarks, with the same - if now somewhat outdated - Llama 2 7B model as baselines used. The experimental results are good, with accuracy in line with expectations, and good calibration results. The distribution-shift experiments also demonstrate good robustness, with uncertainty appropriately reported under these conditions. Finally their ablation study shows that data-conditioning via the auxiliary contextual module is important to achieve the strong uncertainty quantification.\n\n**Clarity improvements:**\n- in Equation 6 for the objective, make it clear which parameters are being optimized; for instance $\\mathcal{L}(\\theta, \\phi)$ instead of just $\\mathcal{L}$\n- emphasise how $\\phi$ and $\\varphi$ relate more clearly\n- The sentence at line 111 reads awkwardly: \"*This leads to model parameters that can vary from data to data, while their distribution remains invariant for all the samples within the training data.*\". Perhaps it could be revised along the lines of \"In prior approaches, the model parameters sampled for each example may differ, but the underlying weight distribution is fixed and identical across all training examples\"\n\n**Nits:**\n- line 106: elements of E elements -> elements of E\n- line 125: Evidence lower-bound (ELBO) of the evidence -> lower bound of the model evidence", "was_revised": true, "final_justification": "I find that this is a reasonably strong paper which makes a well-motivated contribution for representing uncertainty in LLM predictions. My limited concerns regarding parameterisation, notational clarity and the number of trainable parameters were adequately addressed in the rebuttal. The criticism raised by reviewer ddgU disputing the Bayesian nature of the method, while fair, perhaps downplays the variance term's contribution, and might be more of a semantic disagreement: if one focuses on this paper's framing as a method for deriving aleatoric uncertainty from the input context, I do not believe that this is grounds for rejection. Therefore, I recommend acceptance as a novel and well-motivated approach for representing input-dependant aleatoric uncertainty, in a well-written paper.", "rebuttal": "Thank you for your detailed and insightful feedback. We are pleased that you recognize the motivation behind our approach and see value in the proposed method’s ability to represent aleatoric uncertainty, its robustness under distribution shift, and the empirical gains from contextual conditioning. We address your comments and suggestions point by point below.\n\n\n**W1. Emphasise how $\\phi$ and $\\varphi$ relate more clearly.**\n\nWe want to clarify that $\\phi$ represents all the parameters collectively, that is $\\phi=\\lbrace \\theta, \\varphi \\rbrace$, where $\\theta$ represents parameters of all the $B$ and $A$ adapters collectively and $\\varphi$ represents parameters of all the contextual modules collectively. $q_\\phi$ represents the inference network (encoder) in the amortized VI perspective. We will explicitly mention this for improved clarity.\n\n\n\n**W2.  Whether the authors train $\\theta$ in a discrete training phase before $\\varphi$?**\n\nTo clarify, the parameters $\\theta$ and $\\varphi$ are learned simultaneously in an interleaved manner, rather than in separate phases.\n\n\n**W3. The authors leave the use of diagonal Gaussian distributions throughout relatively unexamined.**\n\nWhile we did not explore diagonal Gaussian contextualization directly, we did conduct a complexity study (included in the appendix, section E) comparing diagonal versus full $E$ matrix parameterizations (detailed explanation of FE and DE is provided in Section 4.4 and Appendix section E). In this study, we found that using a full $E$ matrix led to significantly better uncertainty estimates (as measured by ECE and NLL) in both in-/out-of-distribution scenarios, while achieving similar accuracy compared to the diagonal variant. Based on these empirical results, we chose to focus on contextualizing the full $E$ matrix in the main experiments, as it offered more reliable performance across metrics of interest. \n\n\n**W4. Several nits.**\n\nWe will address clarity improvements and nits underscored by you in the final version.\n\n**Q1. Is this sufficient scale to meaningfully learn to represent data uncertainty? When does $r$ become too small? Is this why you train C-LoRA for only 1500 iterations while the baselines for 5000?**\n\nOur choice of $r = 8$ adheres to the default settings in the PEFT library and LoRA paper, and is also adopted by BLoB. This ensures reproducibility and fair comparison with prior works. Moreover, low-rank adapters with small $r$ have been shown to offer a good balance between model capacity and training stability. The rank $r$ controls the number of stochastic parameters injected into the model. A larger $r$ increases expressiveness and the ability to model complex uncertainty, but can also lead to higher computational cost and potentially less stable training due to the increased stochasticity. Conversely, a very small $r$ can restrict the model’s capacity to represent uncertainty, especially in tasks with more complex latent structures. However, we acknowledge that the optimal value of $r$ can be task- and model-specific. Finally, we observed that C-LoRA converged reliably within 1500 iterations, and additional training did not improve performance. We attribute this to the lower-dimensional and structured nature of C-LoRA’s parameter space.\n\n\n\n\n**Q2. How does the fine-tuning time compare between vanilla LoRA and C-LoRA, and at inference time?**\n\n**Training Time (in minutes)**\n\n|   Method    | WG-S  | ARC-C | BoolQ |\n|-------|-------|-------|-------|\n| C-LoRA   | ≈84   | ≈93   | ≈224  |\n| MLE   | ≈114  | ≈110  | ≈136  |\n\n**Inference Time (in minutes)**\n\n|  Method           | WG-S  | ARC-C | BoolQ |\n|-------------|-------|-------|-------|\n| C-LoRA, M=0    | ≈1.75 | ≈0.35 | ≈6.80  |\n| C-LoRA, M=10   | ≈21.60 | ≈4.28 | ≈77.38|\n| MLE         | ≈1.68 | ≈0.44 | ≈4.55 |\n\nWe report both training and inference wall-clock times in the table where all experiments were conducted on the same hardware (NVIDIA A100 40GB). As shown in the table, the inference time of C-LoRA with $M=0$ is comparable to MLE (vanilla LoRA) across all datasets—e.g., 1.75 vs. 1.68 minutes on WG-S, and 6.8 vs. 4.55 minutes on BoolQ. This suggests that contextual modeling incurs minimal overhead at inference when no sampling is involved, while still producing well-calibrated predictions. For applications requiring more accurate uncertainty estimates, C-LoRA with sampling (e.g., $M=10$) leads to improved calibration with additional computational cost. Training time for both methods is comparable except for BoolQ. This longer runtime is likely due to its larger validation set, that increases evaluation overhead during training. This training cost can be reduced by adjusting the evaluation frequency. Finally, we note that while the total training time is comparable, the per-iteration runtime is higher for C-LoRA than for MLE (approximately 0.05 minutes vs. 0.02 minutes), due to the difference in the number of training iterations. \n\n\n\n\n\n**Q3. Whether it can be quantified that there is anything to be gained from a Bayesian treatment of parameters?**\n\nOur work focuses on aleatoric uncertainty, modeled via data-dependent variational distributions over the LoRA weights. This design is intentional: we aim to capture input-dependent uncertainty arising from ambiguous or multimodal outputs.\n\nWhile placing distributions over parameters is often associated with epistemic uncertainty, the data-dependent nature of our variational posterior means the uncertainty varies with the input. This aligns more closely with aleatoric uncertainty, which reflects inherent variability in the data rather than model uncertainty. That said, placing a distribution over LoRA weights is complementary to the current approach and could capture residual epistemic uncertainty in these adapter parameters, as the backbone is frozen and most learnable capacity resides in the LoRA adapters.\n\nWe also compared against a data-independent variational posterior (BLoB in Sections 4.2, 4.3, and FE in Section 4.4) and found that the data-dependent version (C-LoRA) yields significantly better uncertainty estimates—suggesting aleatoric effects dominate in these experiments. Extending Bayesian modeling to the full network (e.g., the backbone) would be computationally nontrivial extension of our work and is beyond the scope of current work, but we agree it is a valuable direction for future work.\n\nWe hope that we have addressed your concerns and look forward to the discussions if there are additional questions."}, {"review_id": "jqcevtiUSI", "reviewer": "Reviewer_TRNk", "summary": "This work focuses on incorporating aleatoric uncertainty for a novel, parameter-efficient fine-tuning approach. This work introduces a contextual module for modeling the stochasticity in low-dimensional space dependent on the data in the LoRA design. This low-dimensional space enables a low-cost contextualized estimation of prediction uncertainty based on few-shot samples. Comprehensive experiments were conducted to verify the proposed model's performance.", "questions": "* Why do the conventional Bayesian neural networks severely limit the uncertainty estimation in few-shot Lora fine-tuning settings in L113?\n* For the formula (5), is it reasonable to only have $E$ dependent on $x$? Why do the $A$ and $B$ not have the dependency on $x$? \n* In L135, why do we not need to consider the $A$ and $B$ in the autoregressive formula, but only consider $E$? \n* How many parameters are manually set in the proposed work?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n\n* The introduction is well motivated.\n* The model part is novel, though the motivation for the model design needs more clarification.\n* The experiments are comprehensive across in-distribution and distribution shift domains.  \n\nWeakness\n\n* The motivation for the model design of $E$ is not very clear. For the formula (5), is it reasonable to only have $E$ dependent on $x$? Why do the $A$ and $B$ not have the dependency on $x$? Also, in L135, why do we not need to consider the $A$ and $B$ in the autoregressive formula, but only consider $E$? \n* The model writing needs more revision. For example, Sec. 3.3 implied the parameterization. But it is unclear whether the parameterization is the same as the reparameterization in L154 of Sec. 3.4.\n* Based on Table 1, it looks like the model improves the uncertainty estimation performance (ECE) but decreases the task performance (ACC and NLL).", "was_revised": true, "final_justification": "Thank you for the detailed response. It has addressed most of my concerns. Please merge the explanation about Section 3.3 in the revision. I am improving the score to 4.", "rebuttal": "Thank you for your constructive feedback. We appreciate that you found our introduction well-motivated, the model design novel, and the experimental evaluation comprehensive. We also value your detailed comments regarding the model formulation and writing clarity, as well as your observations on performance trade-offs. We address each of these points below.\n\n**W1.1 Motivation for the model design of $E$**\n\nThe core motivation behind our design is to effectively capture data-dependent uncertainty in a scalable and stable manner. In principle, any of the components $A$, $B$, or $E$ can be made input-dependent by introducing a contextual module. However, conditioning either $A$ or $B$ on the input significantly increases the computational burden—both in terms of memory and compute—due to their dimensionality being linear complexity with respect to frozen weight matrix dimension which tends to be large. In our C-LoRA setting, for $B$ and $A$ LoRA adapters where $d\\gg r$ (e.g., $d=4096$, $r=8$), this results in needing to predict a large number of mean and variance parameters per input ($O(d\\cdot r)$), which hinders scalability and affects training robustness and stability. In contrast, the matrix $E$ operates in the lower $r\\times  r$ space (constant complexity w.r.t. $d$) and offers a computationally efficient and more stable avenue for introducing input-dependent behavior. By contextualizing $E$, we can model structured, input-specific variations in the adaptation layer without incurring the drawbacks of full contextualization.\nTherefore, our decision to restrict contextualization to $E$ is a deliberate tradeoff: it retains the ability to model data uncertainty (aleatoric uncertainty) effectively, while remaining computationally tractable and robust in practice. This strikes a meaningful balance between expressivity and efficiency.\n\n\n**W1.2 and Q2. Why $A$ and $B$ are not dependent on $x$?** \n\nAligned with the previous answer, we specifically contextualize only $E$, and keep $A$ and $B$ deterministic and shared across inputs. This choice is driven by practical considerations: contextualizing $A$ and $B$, which are high-dimensional ($d \\times r$), would introduce significant computational and memory overhead, and in our experiments, could reduce training stability. By modeling input-dependence through the lower-dimensional $E_x$, we strike a balance between efficiency and expressiveness—capturing aleatoric uncertainty while keeping the overall model lightweight and robust.\n\n\n\n**W1.3 and Q3. Why only $E$ is modeled in an autoregressive manner?** \n\nHere, by modeling $E_x^l$ in an autoregressive manner, we introduce an autoregressive distribution on $E_x^l$. As, $B$ and $A$ are learned deterministically, we do not consider them in an autoregressive fashion.\n\n**W2. Revision of the model writing**\n\nIn section 3.3, the parameterization refers to the layer-wise auxiliary contextual module’s parameters. As mentioned on line 141, we parameterize the per-layer contextual module with a neural network with two fully-connected layers, collectively denoting their parameters by $\\varphi$.\n\nIn line 154, for effective sampling from the distribution of $E_x^l$ we employ a widely known reparameterization trick from Bayesian Machine Learning literature.\n\nWe will edit these parts for improved clarity.\n\n**W3. Performance based on table 1 (improvement on ECE but potential task performance reduction )**\n\nWe have discussed the trade-off between predictive performance and uncertainty estimation in the experiments section, supported by our empirical findings. That said, we would like to point out that C-LoRA consistently achieves strong performance across all three metrics—ACC, ECE, and NLL. Notably, in the low-sample setting ($M=0$), which is most relevant for practical efficiency, C-LoRA ($M=0$) outperforms BLoB ($M=0$) on both ECE and NLL across nearly all datasets, while maintaining competitive accuracy. Similarly, in the $M=10$ setting, C-LoRA performs on par or better than BLoB ($M=10$) in terms of NLL, and almost always outperforms all baselines on ECE, with comparable accuracy.\n\nMoreover, in out-of-distribution (OOD) settings, C-LoRA achieves either the best or second-best scores in both ECE and NLL, demonstrating not only improved calibration but also robust generalization.\n\nThus, we acknowledge that C-LoRA may not always achieve the highest accuracy across all datasets; however, it consistently delivers strong uncertainty quantification (UQ) performance, as evidenced by the lowest or second-lowest ECE and NLL in most settings. We believe this reflects a favorable trade-off: while minor accuracy fluctuations (typically within 1–2%) can occur, these are well-compensated by substantial improvements in calibration and robustness, especially under distribution shift and limited inference budgets.\n\n\n**Q1. Why BNN limits UQ in few shot fine-tuning?**\n\nConventional Bayesian neural networks (BNNs) typically learn a single, input-agnostic posterior distribution over model parameters. Accurately approximating both the mean and variance of that posterior typically requires large datasets, so in a few-shot LoRA scenario the posterior remains broad and prior-dominated. Because it doesn’t depend on the input, it produces uniform (“homoscedastic”) epistemic uncertainty across all $x$, failing to spike for novel or out-of-distribution inputs. In contrast, our C-LoRA injects stochasticity via input-conditioned latent variables, producing heteroscedastic, data-dependent uncertainty and delivering better calibration and predictive performance (as shown by lower NLL and ECE) even with scarce data.\n\n\n**Q4. How many parameters are manually set?**\n\nIn our proposed work, we manually set only a limited number of hyper-parameters. These include the reduced dimension $r$, the architectural details of the contextual module such as the number of layers and hidden neurons, learning rates, and other similar training-related parameters. A list of these hyper-parameters along with their values is provided in the appendix (section D). Importantly, all other parameters of the model ($A$, $B$, and contextual modules’ weights) are learned automatically during training, without manual intervention.\n\n\nWe hope that we have addressed your concerns and look forward to the discussions if there are additional questions? Moreover, we kindly request that you reevaluate our work in light of our rebuttal."}, {"review_id": "ue9kt1EmcM", "reviewer": "Reviewer_ddgU", "summary": "This paper introduces Contextual Low-Rank Adaptation (C-LoRA), a novel uncertainty-aware PEFT method for LLMs. The key innovation of C-LoRA is to make uncertainty estimates dependent on the input $x$ rather than using fixed parameter distributions like existing Bayesian LoRA approaches such as BLoB. The method introduces a lightweight LoRA factorization that inserts a low-dimensional matrix $E$ between the standard LoRA matrices $B$ and $A$, then uses a small auxiliary neural network to learn input-dependent distributions for $E$ at each layer. This allows C-LoRA to capture aleatoric (data-dependent) uncertainty while maintaining computational efficiency. Through extensive experiments on six commonsense reasoning tasks using Llama2-7B, the authors demonstrate that C-LoRA achieves superior uncertainty calibration (measured by ECE and NLL) compared to existing methods like BLoB, Deep Ensemble, and Laplace-LoRA, while maintaining competitive accuracy and showing better robustness under distribution shift scenarios.", "questions": "1. Does each layer have an independently trained MLP for predicting the matrix $E_x$, or do they share one MLP? \n2. I find the analysis on why the post-hoc temperature scaling even worsens the calibration not convincing. Do you suggest fitting the temperature on the training data can lead to its overfitting on the test? Does this process (of calibration) involve Bayesian Model Averaging or not? Do you have an independently held-out validation data for calibration? If you do, then this case could rarely happen. \n3. How does C-LoRA scale with the test-time compute, i.e., the number of samples? Does it show the same tendency observed in BLoB, where a larger number of samples lead to better ECE and NLL?\n4. For the remaining questions, please refer to the section of Weaknesses. \n5. The citation format is broken, i.e., with no venues included. Please fix it.\n\n**I would happily raise my score if the authors are able to address my weaknesses & questions above.**", "limitations": "yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. **Addresses a Critical Problem:** This paper tackles the important issue of overconfident predictions in fine-tuned LLMs, which is essential for deploying models in safety-critical applications where uncertainty quantification is crucial.\n2. **Computationally Efficient Framework:** It introduces an (even-more) lightweight LoRA factorization that reduces complexity of learning and sampling during the test time, making Bayesian Uncertainty Quantification scalable for LLMs while maintaining effectiveness.\n3. **Strong Empirical Performance:** It consistently achieves superior confidence calibration (lowest ECE scores) across six reasoning datasets and demonstrates robust performance under distribution shift, outperforming established baselines like BLoB, Deep Ensemble, and Laplace-LoRA.\n4. **Comprehensive Ablation Studies:** It provides thorough experimental validation showing the necessity of the contextual module, with systematic comparisons across multiple datasets and distribution shift scenarios that clearly demonstrate the method's effectiveness.\n5. **Clear and Well-Structured Writing:** The paper is well-organized with clear mathematical formulations, intuitive explanations of the contextual approach, and comprehensive experimental details that facilitate understanding and reproducibility.\n\n**Weaknesses**\n1. **Limited Discussion on Epistemic/Aleatoric Uncertainty:** While the paper claims to focus on aleatoric uncertainty, the distinction between aleatoric and epistemic uncertainty isn't rigorously established, and the method may conflate both types. \n2. **Limited Scale of the Backbone LLMs:** Experiments are only conducted on Llama2-7B. It's unclear how the method scales to larger models (13B+), performs on other model families (Qwen, Mistral, etc.), or more recent models (Llama3.1), limiting generalizability claims.\n3. **Lack of Intuitive, Qualitative Demonstrations:** The paper claims that C-LoRA is able to fix the problem of previous methods' inability of  quantifying input-specific data uncertainty, but we do not see a dedicated section/result illustrating this point. Maybe the authors can adopt the same visualization framework in BLoB's Appendix C.6 to better show their point? Or a distribution of the variance of different input data?", "was_revised": true, "final_justification": "All the existing experimental results are showing C-LoRA is a good MAP estimate + very small noise added to the weights, nullifying its core claim of being Bayesian.", "rebuttal": "Thank you for your detailed and thoughtful feedback. We’re glad you found our method novel, well-motivated, and effective in both calibration and robustness. We appreciate your suggestions regarding uncertainty types, scaling, and visualization, as well as your insightful questions, which we address below.\n\n\n**W1. Limited discussion on epistemic/aleatoric uncertainty.**\n\nOur work primarily aims to enhance the model’s awareness of aleatoric (input-dependent) uncertainty, rather than to explicitly disentangle aleatoric and epistemic uncertainty.\n\nWe acknowledge that we do not provide a theoretical guarantee that C-LoRA can fully separate the two types of uncertainty. However, our architectural design, especially the contextual module, is intended to make the model more sensitive to variations in the input, thereby emphasizing aleatoric uncertainty.\n\nThat said, our method is still capable of capturing both aleatoric and epistemic components. As shown in our additional experiments (in answer to weakness 3), the ratio of aleatoric uncertainty in our model is consistently higher than in the baseline  (on average, around 0.99 and 0.82 for contextual and BLoB, respectively), suggesting a stronger focus on modeling input-dependent uncertainty.\n\nWe appreciate the suggestion and will incorporate a more detailed discussion of this distinction and its limitations in the final version.\n\n\n\n**W2. Limited Scale of the Backbone LLMs.**\n\nIt is true that our current experiments are conducted on the LLaMA2-7B model. We conduct our experiments using  LLaMA2-7B due to its wide adoption and reasonable computational cost for extensive experimentation. Furthermore, most of the recent Bayesian PEFT methods employ LLaMA2-7B model in their experiments and we stick to the same model for reproducibility and fair comparison.\n\nThat said, our proposed C-LoRA is a model-agnostic framework that does not rely on any architecture-specific assumptions or scaling-related heuristics. Therefore, we believe the efficiency and effectiveness of our method can generalize to larger-scale models (e.g., 13B+), other model families (e.g., Qwen, Mistral), and more recent versions (e.g., LLaMA3.x).\n\nWe agree that extending experiments to additional backbones would strengthen our claims on generalizability. As it would require a significant amount of computational resources, we leave this for future work, and will clarify this limitation in the final version.\n\n\n\n**W3. Lack of intuitive, qualitative demonstrations.**\n\n**EE and MI Scores (C-LoRA vs BLoB, M=10)**\n\n\n| Metric | Method | OBQA | ARC-C | ARC-E | Chem  | Phy   |\n|--------|--------|------|-------|-------|-------|-------|\n| EE     | C-LoRA    | 0.538 | 0.705 | 0.583 | 1.155 | 1.185 |\n|        | BLoB   | 0.349 | 0.480 | 0.388 | 0.999 | 1.018 |\n| MI     | C-LoRA    | 0.010 | 0.011 | 0.008 | 0.007 | 0.008 |\n|        | BLoB   | 0.111 | 0.132 | 0.113 | 0.114 | 0.116 |\n\n\n\nDue to the regulations of NeurIPS, we are not able to provide any visualization. Hence, to assess input-dependent uncertainty, we adopt the decomposition of predictive entropy (PE) as in  [1] (Section 2.2) and [2] (Section 3.4), where the expected entropy (EE) captures data uncertainty (aleatoric), and mutual information (MI) reflects model uncertainty (epistemic). Note that this decomposition follows the relation PE=MI+EE. \n\nIn our analysis (as shown in the table), we compute EE and MI at $M=10$  across both in-distribution (OBQA) and out-of-distribution (ARC-C, ARC-E, Chem, and Phy) test sets. The model is trained on OBQA for all experiments. This setting allows us to better probe input-specific uncertainty, as OOD examples are more likely to elicit higher uncertainty levels that reveal how sensitively the model responds to unfamiliar or ambiguous inputs. We observe that the C-LoRA model consistently exhibits higher EE than BLoB across all datasets, while maintaining substantially lower MI. This suggests that the C-LoRA model is more attuned to data-dependent uncertainty, without attributing uncertainty to model parameters—indicating a more calibrated and stable epistemic profile. Particularly, the low MI suggests that most of the uncertainty in C-LoRA’s predictions comes from input variability, not model instability.\n\nIn general, the results in the table support our claim that C-LoRA is more effective at capturing input-specific uncertainty. \n\n\n\n\n\n\n**Q1.  If each layer has an independent MLP?**\n\nWhile the architecture of the contextual module is consistent across layers, each layer has its own independently parameterized module. That is, the MLP used to predict the matrix is not shared across layers, and its parameters are learned separately for each layer during training.\n\n\n**Q2. Why does temperature scaling worsen the performance?**\n\nWe indeed observe that, for OBQA and a few similar datasets, temperature scaling (TS) did not yield improvements in ECE, and in fact slightly worsened it, despite no meaningful change in average NLL as indicated in the following. This outcome, while unintuitive at first glance, has been observed in prior work and reflects the limitations of TS in some settings. \n\nWe tuned the temperature on a held-out validation set (not training data), so overfitting is unlikely. Since TS directly optimizes NLL, we verified that NLL before and after TS remained effectively unchanged (e.g., 0.59 for OBQA), suggesting TS reached a flat region in the NLL landscape but slightly altered the confidence distribution.\n\nAlso, temperature scaling minimizes NLL, which is a proper scoring rule, but does not directly minimize ECE, which is a binned, non-differentiable, and improper metric. Hence, if a model is already moderately calibrated, or if the logit distribution has specific local structure, TS may shift predictions in a way that harms ECE in certain bins. \n\nAdditionally, since our model is more sensitive towards data-dependent (aleatoric) uncertainty, it may already exhibit well-calibrated behavior in regions of high variability. This may further limit the effectiveness of a global post-hoc method like temperature scaling, which cannot adapt to such input-specific uncertainty patterns.\n\nTo find the temperature parameter, we used a single sample ($M = 1$) without Bayesian model averaging (BMA). While averaging over multiple samples ($M>1$) would constitute BMA, we did not adopt that here. \n\nSimilar effects are reported by Kull et al. [3], who show that TS can worsen classwise-ECE—for example, increasing cw-ECE from 0.296 to 0.327 on the car dataset and degrading mid-range calibration on balance-scale. These findings are consistent with our observations.\n\nWe emphasize that temperature scaling was used purely as a standard post-hoc tool to potentially improve the uncertainty quantification capabilities, and is not a component of the model itself. \n\n\n\n\n**Q3. How does C-LoRA scale with the test-time compute?**\n\n| Metric    | M=0     | M=1     | M=2     | M=4     | M=10    | M=15    | M=20    | M=50    |\n|------|-------|-------|-------|-------|-------|-------|-------|-------|\n| ACC (↑) | 81.13 | 77.6  | 77.93 | 78.33 | 78.26 | 78.33 | 78.6  | 78.6  |\n| ECE (↓) | 5.10  | 5.02  | 4.21  | 3.55  | 4.00     | 4.72  | 4.37  | 3.67  |\n| NLL (↓) | 0.53  | 0.60   | 0.60   | 0.59  | 0.59  | 0.59  | 0.59  | 0.59  |\n\nIn the table, we report the mean performance among 3 random seeds when the model is fine-tuned on the OBQA task. We observe that increasing the number of samples $M$ at test time generally improves the calibration performance, as reflected in the decreasing ECE values. Specifically, ECE drops from 5.10 at $M=0$ to 3.55 at $M=4$, and continues to decline slightly with additional samples, reaching 3.67 at $M=50$. This suggests that aggregating predictions over multiple samples helps reduce miscalibration.\n\nIn terms of NLL, the deterministic prediction ($M=0$) yields the lowest value (0.53), indicating confident predictions with strong likelihood. Among the stochastic variants ($M\\geq 1$), NLL remains relatively stable around 0.59–0.60, showing a slight decreasing trend with more samples and indicating some benefit from sampling.\n\nAccuracy is highest at $M=0$ and slightly lower across other values of $M$, but remains fairly stable overall. This suggests that increasing $M$ primarily affects uncertainty estimation rather than core predictive accuracy.\n\nOverall, as the number of test-time samples increases, calibration improves consistently, and likelihood shows marginal gains, while accuracy remains largely unchanged. This pattern is consistent with the fact that our method primarily models data (aleatoric) uncertainty, for which sampling has a more limited impact compared to approaches that also capture epistemic uncertainty.\n\n\n\n**Q4. Broken citation format.**\n\nWe will fix the citation format to include the conference names (i.e venues).   \n\n\nWe hope that we have addressed your concerns and look forward to the discussions if there are additional questions?\n\n\n\n[1] Mucsányi, Bálint, et al. “Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks” NeurIPS 2024\n\n[2] Jishnu Mukhoti and Yarin Gal. “Evaluating Bayesian Deep Learning Methods for Semantic Segmentation”. arXiv preprint, 2018.\n\n[3] Kull, Meelis, et al. \"Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration\", NeurIPS 2019"}]

Author rebuttal:

Meta-review: The paper introduces C-LORA (Contextual Low-Rank Adaptation), a parameter-efficient fine-tuning method for large language models (LLMs) that enhances uncertainty quantification by incorporating data-dependent contextual modules. Unlike traditional LoRA-based Bayesian approaches, which rely on static parameter posteriors, C-LORA dynamically adapts its uncertainty estimates based on each input. It achieves this through a small auxiliary network that predicts the distribution over an intermediate matrix inserted into the LoRA adapters, enabling explicit modeling of aleatoric uncertainty. The method is validated on six commonsense reasoning datasets, showing improved calibration (ECE, NLL) and robustness to distribution shift while maintaining competitive accuracy.

**Strengths:**
- Introduces a novel contextual mechanism that explicitly models aleatoric (input-specific) uncertainty, a significant step beyond prior Bayesian PEFT methods.
- Demonstrates consistent improvements in uncertainty calibration metrics (ECE, NLL) across multiple datasets and under distribution shift.
- Maintains computational efficiency by using a fixed-size latent representation and a compact context module, independent of model scale.
- Comprehensive experimental validation, including strong ablation studies confirming the effectiveness of the contextual module.
- Clear organization, detailed mathematical exposition, and plans to release code support reproducibility and clarity.

**Weaknesses:**
- Experiments are limited to Llama2-7B; no evaluation on larger models or other families (e.g., Mistral, Llama3), reducing generalizability.
- Slight decrease in task accuracy (1–2%) in certain datasets when compared to the best-performing baselines.
- The distinction between aleatoric and epistemic uncertainty is underdeveloped and may lead to conceptual ambiguity.
- Lacks qualitative illustrations of input-dependent uncertainty (e.g., visualizations or example-based variance analysis).
- Some notational clarity and architectural details (e.g., contextual module structure and training process) could be improved.

Most concerns have been addressed by the authors during the rebuttal period, and some reviewers have raised their scores accordingly, with 4 positive ratings at the end of the rebuttal. I am recommending acceptance. Note that

- One reviewer also mentioned that part of the proposed method uses MAP and is therefore not fully Bayesian, which I tend to agree. It is expected that the authors will make this clearer in the final version. 
- The authors are expected to denote the fact that their method has only undergone testing on 7B as a limitation (preferably also mention this in the abstract) and that studying scaling of the underlying methodology remains an open problem.
Decision: Accept (poster)