Title: C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Low-rank Adaptation

Uncertainty Quantification

Probabilistic Modeling
Abstract: Low-Rank Adaptation (LoRA) offers a cost-effective solution for fine-tuning large language models (LLMs), but it often produces overconfident predictions in data-scarce few-shot settings. To address this issue, several classical statistical learning approaches have been repurposed for scalable uncertainty-aware LoRA fine-tuning. However, these approaches neglect how input characteristics affect the predictive uncertainty estimates. To address this limitation, we propose Contextual Low-Rank Adaptation (**C-LoRA**) as a novel uncertainty-aware and parameter efficient fine-tuning approach, by developing new lightweight LoRA modules contextualized to each input data sample to dynamically adapt uncertainty estimates. Incorporating data-driven contexts into the parameter posteriors, C-LoRA mitigates overfitting, achieves well-calibrated uncertainties, and yields robust predictions. Extensive experiments on LLaMA2-7B models demonstrate that C-LoRA consistently outperforms the state-of-the-art uncertainty-aware LoRA methods in both uncertainty quantification and model generalization. Ablation studies further confirm the critical role of our contextual modules in capturing sample-specific uncertainties.  C-LoRA sets a new standard for robust, uncertainty-aware LLM fine-tuning in few-shot regimes. Although our experiments are limited to 7B models, our method is architecture-agnostic and, in principle, applies beyond this scale; studying its scaling to larger models remains an open problem. Our code is available at https://github.com/ahra99/c_lora.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Contextual Low-Rank Adaptation (C-LoRA), a novel uncertainty-aware PEFT method for LLMs. The key innovation of C-LoRA is to make uncertainty estimates dependent on the input $x$ rather than using fixed parameter distributions like existing Bayesian LoRA approaches such as BLoB. The method introduces a lightweight LoRA factorization that inserts a low-dimensional matrix $E$ between the standard LoRA matrices $B$ and $A$, then uses a small auxiliary neural network to learn input-dependent distributions for $E$ at each layer. This allows C-LoRA to capture aleatoric (data-dependent) uncertainty while maintaining computational efficiency. Through extensive experiments on six commonsense reasoning tasks using Llama2-7B, the authors demonstrate that C-LoRA achieves superior uncertainty calibration (measured by ECE and NLL) compared to existing methods like BLoB, Deep Ensemble, and Laplace-LoRA, while maintaining competitive accuracy and showing better robustness under distribution shift scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. **Addresses a Critical Problem:** This paper tackles the important issue of overconfident predictions in fine-tuned LLMs, which is essential for deploying models in safety-critical applications where uncertainty quantification is crucial.\n2. **Computationally Efficient Framework:** It introduces an (even-more) lightweight LoRA factorization that reduces complexity of learning and sampling during the test time, making Bayesian Uncertainty Quantification scalable for LLMs while maintaining effectiveness.\n3. **Strong Empirical Performance:** It consistently achieves superior confidence calibration (lowest ECE scores) across six reasoning datasets and demonstrates robust performance under distribution shift, outperforming established baselines like BLoB, Deep Ensemble, and Laplace-LoRA.\n4. **Comprehensive Ablation Studies:** It provides thorough experimental validation showing the necessity of the contextual module, with systematic comparisons across multiple datasets and distribution shift scenarios that clearly demonstrate the method's effectiveness.\n5. **Clear and Well-Structured Writing:** The paper is well-organized with clear mathematical formulations, intuitive explanations of the contextual approach, and comprehensive experimental details that facilitate understanding and reproducibility.\n\n**Weaknesses**\n1. **Limited Discussion on Epistemic/Aleatoric Uncertainty:** While the paper claims to focus on aleatoric uncertainty, the distinction between aleatoric and epistemic uncertainty isn't rigorously established, and the method may conflate both types. \n2. **Limited Scale of the Backbone LLMs:** Experiments are only conducted on Llama2-7B. It's unclear how the method scales to larger models (13B+), performs on other model families (Qwen, Mistral, etc.), or more recent models (Llama3.1), limiting generalizability claims.\n3. **Lack of Intuitive, Qualitative Demonstrations:** The paper claims that C-LoRA is able to fix the problem of previous methods' inability of  quantifying input-specific data uncertainty, but we do not see a dedicated section/result illustrating this point. Maybe the authors can adopt the same visualization framework in BLoB's Appendix C.6 to better show their point? Or a distribution of the variance of different input data?\n\nQUESTIONS:\n1. Does each layer have an independently trained MLP for predicting the matrix $E_x$, or do they share one MLP? \n2. I find the analysis on why the post-hoc temperature scaling even worsens the calibration not convincing. Do you suggest fitting the temperature on the training data can lead to its overfitting on the test? Does this process (of calibration) involve Bayesian Model Averaging or not? Do you have an independently held-out validation data for calibration? If you do, then this case could rarely happen. \n3. How does C-LoRA scale with the test-time compute, i.e., the number of samples? Does it show the same tendency observed in BLoB, where a larger number of samples lead to better ECE and NLL?\n4. For the remaining questions, please refer to the section of Weaknesses. \n5. The citation format is broken, i.e., with no venues included. Please fix it.\n\n**I would happily raise my score if the authors are able to address my weaknesses & questions above.**\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on incorporating aleatoric uncertainty for a novel, parameter-efficient fine-tuning approach. This work introduces a contextual module for modeling the stochasticity in low-dimensional space dependent on the data in the LoRA design. This low-dimensional space enables a low-cost contextualized estimation of prediction uncertainty based on few-shot samples. Comprehensive experiments were conducted to verify the proposed model's performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n* The introduction is well motivated.\n* The model part is novel, though the motivation for the model design needs more clarification.\n* The experiments are comprehensive across in-distribution and distribution shift domains.  \n\nWeakness\n\n* The motivation for the model design of $E$ is not very clear. For the formula (5), is it reasonable to only have $E$ dependent on $x$? Why do the $A$ and $B$ not have the dependency on $x$? Also, in L135, why do we not need to consider the $A$ and $B$ in the autoregressive formula, but only consider $E$? \n* The model writing needs more revision. For example, Sec. 3.3 implied the parameterization. But it is unclear whether the parameterization is the same as the reparameterization in L154 of Sec. 3.4.\n* Based on Table 1, it looks like the model improves the uncertainty estimation performance (ECE) but decreases the task performance (ACC and NLL).\n\nQUESTIONS:\n* Why do the conventional Bayesian neural networks severely limit the uncertainty estimation in few-shot Lora fine-tuning settings in L113?\n* For the formula (5), is it reasonable to only have $E$ dependent on $x$? Why do the $A$ and $B$ not have the dependency on $x$? \n* In L135, why do we not need to consider the $A$ and $B$ in the autoregressive formula, but only consider $E$? \n* How many parameters are manually set in the proposed work?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of uncertainty quantification in large language models, introducing a new method that proposes to take into account the data at hand to represent the aleatoric uncertainty. This sets it apart from many previous UC methods for LLMs which tended to only train a weight posterior, is less suited to representing uncertainty relating to the specific input sequence at test time. Contextual Low-Rank Adaptation (C-LoRA) modifies the usual $\\mathbf{W} _{0} + \\mathbf{B}\\mathbf{A}$ LoRA adaptors by inserting a small $\\mathbf{E} \\in \\mathbb{R}^{r \\times r}$ matrix, which is independent of the underlying model weight size $d$ giving $\\mathbf{W} _{0} + \\mathbf{BEA}$. This allows $\\mathbf{E}$ to be treated as a latent random variable, with overhead $O(r^2)$ per layer independent of $d$, whose mean and variance are predicted using a small context module from the current hidden state, or the input sequence at layer $l = 0$. This context module is trained to output distribution parameters $\\phi$ for $q _{\\phi}(\\mathbf{E} _{\\mathbf{x} _i} \\vert \\mathbf{x} _{i})$ such that when sampled the LLM accurately represents the aleatoric uncertainty: with $\\mathbf{E}$ being sharply peaked for certain inputs $\\mathbf{x} _{i}$, and broad on ambiguous inputs. Experiments on standard datasets and under distribution shift show that C-LoRA substantially improve calibration over many baseline methods, even when using few MC samples at inference time. Ablations show that removing this contextual conditioning also removes these gains. C-LoRA represents a useful contribution to the field\n\nSTRENGTHS AND WEAKNESSES:\nWithin the small sub-field of probabilistic treatments of LLMs for uncertainty quantification, the paper makes a good contribution. Earlier attempts straightforwardly applied methods from Bayesian deep learning, which were themselves developed in simpler settings: where the model inputs were straightforward sensor measurements or observations, and where data-dependent output uncertainty could be adequately expressed by learning a weight posterior distribution to have good model evidence (i.e. yielding appropriate output uncertainty estimates after being pushed through the model), based on the data and patterns observed during training. However, in LLMs and sequence models, the input may be a complicated question requiring extended reasoning or nuanced understanding. Expressing the output uncertainty using the weight posterior alone is a rather blunt tool, and this paper proposes a sensible approach to modulate the output uncertainty based on the data point at hand.\n\nFollowing prior work attempting to perform Bayesian inference over LLM parameters, the authors freeze the model weights and operate over the adapter to reduce the otherwise prohibitive computational cost. However, they go further to remove the direct dependence on the base model's dimension, inserting a fixed-dimension $r \\times r$ matrix, where $r \\ll d$, capping the number of parameters to infer as $O(r^2)$. To amortize the inference of the variational parameters $\\phi$, the authors propose a simple 2-layer ReLU network with parameters $\\varphi$ which they call a 'contextual module', that initially accepts the input $\\mathbf{x} _{i}$ at layer $l = 0$, and the output of layer $l - 1$ for subsequent layers. The clarity of this section may be improved for readers unfamiliar with variational inference by making more explicit how $\\phi$ and $\\varphi$ relate. The authors leave the use of diagonal Gaussian distributions throughout relatively unexamined: while this is a sensible choice on account of its cheap computational properties, at higher ranks ignoring correlations inside $\\mathbf{E}$ could matter..\n\nTraining proceeds using straightforward VI objectives, with the authors choosing to directly train the LoRA parameters $\\mathbf{\\theta} = \\{\\mathbf{A}, \\mathbf{B}\\}$ rather than use a Bayesian treatment. It is unclear whether the authors train $\\mathbf{\\theta}$ in a discrete training phase before $\\varphi$, or in an interleaved manner. It would be good to clarify this.\n\nThe authors re-use a standard set of relatively diverse benchmarks, with the same - if now somewhat outdated - Llama 2 7B model as baselines used. The experimental results are good, with accuracy in line with expectations, and good calibration results. The distribution-shift experiments also demonstrate good robustness, with uncertainty appropriately reported under these conditions. Finally their ablation study shows that data-conditioning via the auxiliary contextual module is important to achieve the strong uncertainty quantification.\n\n**Clarity improvements:**\n- in Equation 6 for the objective, make it clear which parameters are being optimized; for instance $\\mathcal{L}(\\theta, \\phi)$ instead of just $\\mathcal{L}$\n- emphasise how $\\phi$ and $\\varphi$ relate more clearly\n- The sentence at line 111 reads awkwardly: \"*This leads to model parameters that can vary from data to data, while their distribution remains invariant for all the samples within the training data.*\". Perhaps it could be revised along the lines of \"In prior approaches, the model parameters sampled for each example may differ, but the underlying weight distribution is fixed and identical across all training examples\"\n\n**Nits:**\n- line 106: elements of E elements -> elements of E\n- line 125: Evidence lower-bound (ELBO) of the evidence -> lower bound of the model evidence\n\nQUESTIONS:\n- The $r \\times r$ (i.e. $16 \\times 16$) matrix at each layer results in a very small set of parameters to infer. Is this sufficient scale to meaningfully learn to represent data uncertainty? When does $r$ become too small? Is this why you train C-LoRA for only 1500 iterations while the baselines are trained for 5000?\n- Could you please characterise the computational cost of C-LoRA in terms of wall clock time for training and inference on the same computational resources. How does the fine-tuning time compare between vanilla LoRA and C-LoRA, and at inference time how much longer does it take to make a prediction?\n- You focus exclusively on aleatoric uncertainty and not epistemic uncertainty. Could you quantify whether there is anything to be gained from a Bayesian treatment of the LoRA parameters or even the contextual network's parameters?\n\nLIMITATIONS:\nThe work largely improves upon existing baseline methods for uncertainty quantification in LLMs, and while it makes valuable progress towards improving the calibration of this class of methods, it suffers from the same limitation of being constrained to single-token predictions and does not make progress on how to represent uncertainty in more modern LLM use-cases. Language models have grown in popularity mainly due to their use as generative models and not discriminative models. While the single-token constraint still encapsulates some non-trivial subset of LLM use-cases (multiple-choice question answering, LLM-as-judge, critic models in RL, point predictions for judgemental forecasting, etc), the current work only considers multiple-choice question answering tasks. Moreover, this paper does not address ways in which we might extend the uncertainty quantification to multi-token generative applications, such as reasoning traces, uncertainty-aware decoding mechanisms, or simply articulating the uncertainty in natural language, which are possibly far more pertinent to modern LLM deployments and use-cases. The work could be improved by formulating new datasets or tasks that move the field of UQ for LLMs away from MCQA and towards more modern and general use-cases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces C-LORA (Contextual Low-Rank Adaptation), a novel parameter-efficient fine-tuning approach designed for uncertainty estimation in Large Language Models (LLMs). The core problem addressed is the overconfident predictions often produced by LoRA in data-scarce few-shot settings, and the neglect of input characteristics in existing scalable uncertainty-aware LoRA methods. C-LORA tackles this by developing lightweight LoRA modules that are contextualized to each input data sample, allowing for dynamic adaptation of uncertainty estimates. The method incorporates data-driven contexts into parameter posteriors, which aims to mitigate overfitting, achieve well-calibrated uncertainties, and yield robust predictions. The paper highlights C-LORA's ability to explicitly model aleatoric (data) uncertainty, which is a key differentiator from prior works. Extensive experiments across various common-sense reasoning datasets demonstrate C-LORA's superior performance in both uncertainty quantification (UQ) and model generalization compared to state-of-the-art uncertainty-aware LoRA methods. Ablation studies further confirm the critical role of the contextual modules in capturing sample-specific uncertainties.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tOriginality & Significance (High): The paper introduces a novel and important concept of \"contextualized\" LoRA, specifically for dynamic uncertainty estimation in LLMs. The explicit modeling of aleatoric uncertainty, which is often overlooked in existing Bayesian PEFT methods, is a significant contribution, especially in low-data regimes. This innovation sets a new standard for robust, uncertainty-aware LLM fine-tuning.\n\n2.\tQuality & Effectiveness in UQ (High): C-LORA consistently outperforms state-of-the-art uncertainty-aware LoRA methods in uncertainty quantification, as measured by ECE and NLL. This is a strong indicator of the method's ability to produce well-calibrated and reliable uncertainty estimates. The results in Table 1 and Table 2 clearly demonstrate this superiority.\n\n3.\tRobustness under Distribution Shift (High): The experiments on out-of-distribution (OOD) datasets show C-LORA's strong robustness in uncertainty estimation (ECE and NLL), even when the input distribution diverges from training. This is crucial for real-world applications where models might encounter unseen data.\n\n4.\tComprehensive Experimental Validation: The paper provides extensive experiments across six common-sense reasoning tasks, including both in-distribution and out-of-distribution scenarios. The comparison with a good set of baselines (Deep Ensemble, MCD, BLoB, LA, MAP) is thorough.\n\nWeaknesses:\n\n1.\tCode and Data Accessibility (Moderate): The paper states that code will be released upon acceptance. While this is a common practice, the lack of immediate access to the code for verification during the review process is a minor drawback for reproducibility.\n\n2.\tMarginally Lower Accuracy in Some Cases (Minor): While C-LORA excels in UQ, it is noted that its accuracy is sometimes competitive but not always the highest, typically within 1-2% of the best methods. Although the paper frames this as a trade-off for better UQ, it's worth acknowledging. For example, in Table 3 (Chem dataset with M=0) accuracy is noticeably lower than some baselines.\n\n3.\tTemperature Scaling Inconsistency (Minor): The observation that temperature scaling can sometimes worsen ECE for C-LORA (M=10) on certain datasets (e.g., OBQA and BoolQ) is interesting. While the explanation suggests posterior sampling already flattens the distribution, a deeper analysis or alternative post-hoc calibration strategies could further strengthen the calibration aspect.\n\nQUESTIONS:\n1.\tImpact of Contextual Module Complexity: The paper mentions the auxiliary contextual module has C hidden units, with C=O(r2 ). Could the authors provide more insights or a small ablation on how varying C (or the number of layers in the contextual module) might affect performance (both UQ and accuracy) and computational overhead? This would help understand the module's sensitivity to its own architecture.\n\n2.\tTrade-off between Accuracy and UQ: While the paper acknowledges the trade-off, could the authors elaborate on scenarios or tasks where this accuracy difference (e.g., 1-2% drop) might be a critical limiting factor for C-LORA's deployment, despite its superior UQ? For instance, in applications where very high accuracy is paramount and minor calibration improvements are acceptable.\n\n3.\tPrior Distribution Choice for Ex : The paper states adopting a \"simple and fixed Gaussian prior p(Ex)\" for learning Ex. Could the authors discuss the sensitivity of C-LORA's performance to the choice of this prior? Are there alternative prior choices that might be more theoretically grounded or empirically beneficial, especially for different data distributions or task types?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}