Be Confident in What You Know: Bayesian Parameter Efficient Fine-Tuning of Vision Foundation Models

Authors: shankar  deep
  pandey
spandan  pyakurel
qi  yu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 21:35:23

Abstract: Large transformer-based foundation models have been commonly used as pre-trained models that can be adapted to different challenging datasets and settings with state-of-the-art generalization performance. Parameter efficient fine-tuning ($\texttt{PEFT}$) provides promising generalization performance in adaptation while incurring minimum computational overhead. However, adaptation of these foundation models through $\texttt{PEFT}$ leads to accurate but severely underconfident models, especially in few-shot learning settings. Moreover, the adapted models lack accurate fine-grained uncertainty quantification capabilities limiting their broader applicability in critical domains. To fill out this critical gap, we develop a novel lightweight {Bayesian Parameter Efficient Fine-Tuning} (referred to as $\texttt{Bayesian-PEFT}$) framework for large transformer-based foundation models. The framework integrates state-of-the-art $\texttt{PEFT}$ techniques with two Bayesian components to address the under-confidence issue while ensuring reliable prediction under challenging few-shot settings. The first component performs base rate adjustment to strengthen the prior belief corresponding to the knowledge gained through pre-training, making the model more confident in its predictions; the second component builds an evidential ensemble that leverages belief regularization to ensure diversity among different ensemble components.
Our thorough theoretical analysis justifies that the  Bayesian components can ensure reliable and accurate few-shot adaptations with well-calibrated uncertainty quantification. Extensive experiments across diverse datasets, few-shot learning scenarios, and multiple $\texttt{PEFT}$ techniques demonstrate the outstanding prediction and calibration performance by $\texttt{Bayesian-PEFT}$.

Main paper: 
 introduction: 
 Transformer-based foundation models have been developed as general models with state-of-the-art generalization performance [32, 66, 52, 28] . These models leverage the rich meta-knowledge acquired during the pre-training stage to effectively adapt to complex downstream tasks [32] , where pretraining is usually performed on massive-scale annotated datasets (e.g., [35, 55] ) through supervised learning [32, 66, 28] or self-supervised learning [52] . To achieve effective adaption, various parameterefficient fine-tuning (PEFT) approaches have been developed [38, 27, 9, 54] that introduce a small number of tunable parameters either within or outside of the backbone architecture to ensure good generalization capability while incurring little computational overhead because most parts of (or the entire) backbone architecture is frozen during fine-tuning [25, 59, 67] . Bias-fine tuning [9] , a representative partial tuning-based PEFT, only fine-tunes the bias parameters to downstream tasks. Adapter [51] and side-tune [72] fine-tuning techniques are instances of extra module-based PEFT that introduce extra parameterized modules and fine-tune them based on the downstream tasks. Visual Prompt-tuning [32] (VPT) follows the popular prompt learning paradigm by introducing a learnable Despite the attractive generalization performance, most foundation models adapted to downstream few-shot tasks through PEFT exhibit a somewhat surprising and undesirable behavior that may prevent them from being applied to many critical domains. Figure 1 (b) shows the predictive accuracy versus the Expected Calibration Error (ECE) of a foundation model after performing few-shot adaptation on CIFAR-100 using a series of representative PEFT methods, including VPT [32] , Adapter [51] , Bias Fine Tuning [71] , and Side-tune [72] . It is clear that the adapted model is able to provide accurate predictions even after fine-tuned on limited training samples. For example, all PEFT methods help to boost the model's accuracy to over 75% using just 5-shot fine-tuning and the accuracy reaches 80% after 10-shot fine-tuning. However, the adapted model is very poorly calibrated as shown by the large ECE scores, which are consistently over 0.3 across all the fine-tuning methods. Increasing the fine-tuning size does not show clear improvement and sometimes even hurts the calibration performance. While one may expect the poor ECE to be caused by over-confidence as we fine-tune a large foundation model using very limited training samples, the detailed ECE plots as shown in Figure 2 (a)-(c) reveal that the model is in fact severely under-confident. For example, after adapting to the 1-shot training dataset, the VPT fine-tuned model can already achieve a test accuracy close to 50% but is under-confident in almost all its predictions leading to an ECE score over 0.45. The under-confidence issue is observed for all representative PEFT methods across different datasets and various few-shot learning settings as evidenced by our experiments.The under-confident few-shot adaptation behavior of foundation models closely mimics how human experts with rich domain knowledge in their own disciplines tend to make conservative decisions when facing new tasks that deviate from their own expertise. Analogous to their human counterparts, the rich prior knowledge gained through the pre-training stage of foundational models overshadows the relatively limited knowledge obtained through few-shot fine-tuning, which prevents them from making more confident predictions in the downstream tasks. Unreliable uncertainty (i.e., confidence) quantification makes the predictions provided by these models less trustworthy, which may limit applying the promising "pre-train-then-finetune" paradigm to many critical domains. As shown in Figure 2 (a)-(c), the fine-tuned model seldom makes any predictions with confidence over 80%, making it difficult to leverage these predictions in any high-stakes decision-making process.The need to balance between the rich prior knowledge gained through pre-training and the new knowledge obtained through few-shot adaptation inspires us to investigate the under-confidence issue from the Bayesian perspective. In particular, we propose to look into the predictive behavior of the few-shot adapted foundation model through the lenses of evidential learning [56] , which is built upon Bayesian theorem and subjective logic (SL) theory [33] . As part of the recent advances in modern Bayesian modeling, evidential learning provides a cost-effective way to perform Bayesian inference with the capability to quantify fine-grained second-order uncertainty [57] . By leveraging the important theoretical connection between fine-grained uncertainty and model accuracy [46] , we unveil the underlying reason that supports the good predictive performance of a few-shot adapted foundation model and the root cause for the under-confident behavior. Drawing from this important insight as outlined above, we propose to integrate the modern PEFT techniques into a novel lightweight Bayesian framework, referred to as Bayesian-PEFT (B-PEFT), aiming to achieve highly reliable and accurate few-shot adaptations with well-calibrated and trustworthy uncertainty quantification.The proposed Bayesian framework offers two important components to address the under-confidence issue while ensuring reliable prediction under challenging few-shot settings. The first component makes novel adjustments to the base rates introduced by the SL theory to strengthen the prior belief corresponding to the knowledge gained through pre-training. Meanwhile, the adjustment does not change the relative order of the belief assigned to different classes, which ensures that the model accuracy is maintained. Our theoretical analysis shows that the proposed base rate adjustment strategy leads to more confident predictions (by increasing the gaps between the belief assigned to the groundtruth class and the rest) without compromising the model's accuracy. Figure 2d shows that B-PEFT significantly improves the model calibration. To further enhance the reliability of both prediction accuracy and uncertainty quantification when performing few-shot adaptation, the second component performs Bayesian model averaging by building a diversity-inducing evidential ensemble. In addition to using different random initialization of the PEFT components, diversity is further enhanced through incorrect belief regularization that penalizes a model for assigning a high belief to a non-ground-truth class. By controlling the strength of belief regularization, different ensemble components are guided to learn from diverse features in the data space, where a light penalty allows an ensemble component to learn the common discriminative features while a heavy one will force an ensemble component to learn rare features to avoid errors on the difficult data samples. A deeper theoretical analysis of the proposed diversity-induced evidential ensemble is equivalent to Stein Variational Gradient Descent (SVGD) based ensembles [13] . Experiments on multiple challenging few-shot learning tasks justify the superior performance of B-PEFT over state-of-the-art PEFT baselines, in terms of both prediction accuracy and uncertainty calibration performance. Our contributions can be summarized as follows:• We identify the severe under-confidence issue of pre-trained foundation models after performing parameter-efficient fine-tuning over few-shot datasets. The fine-grained uncertainty analysis through evidential learning and SL theory reveals the root cause for their good predictive performance while being under-confident. • We develop a novel lightweight Bayesian framework that integrates state-of-the-art PEFT techniques with two Bayesian components to address the under-confidence issue while ensuring reliable prediction under challenging few-shot settings. The first component performs base rate adjustment to strengthen the prior belief corresponding to the knowledge gained through pre-training, making the model more confident in its predictions; the second component builds an evidential ensemble that leverages belief regularization to ensure diversity among different ensemble components. • We perform thorough theoretical analysis to justify why the proposed Bayesian components can ensure reliable and accurate few-shot adaptations with well-calibrated uncertainty quantification. • We carry out experiments with 4 benchmark datasets, 5 different few-shot settings, and 3 parameter efficient fine-tuning techniques that demonstrate the effectiveness of the developed model. 
related works: 
 Parameter Efficient Fine Tuning of Foundation Models. Transformer-based foundation models [63, 15] have been developed as an improvement over traditional convolution-based architectures [29, 31] for computer vision tasks. The transformer-based models achieve strong generalization performance [40] after training on large datasets. Moreover, the pre-trained transformers can be fine-tuned in limited data settings leading to state-of-the-art performance [32, 27] . As a computation, memory, and parameter-efficient alternative to full fine-tuning of such large pre-trained transformers, different Parameter Efficient Fine Tuning (PEFT) approaches have been developed. PEFT techniques freeze most of the large transformer backbone, fine-tune the remaining backbone parameters and/or introduce lightweight extra modules for adapting to the downstream task. Existing approaches can broadly be categorized as extra-module-based [72, 54] , partial-tuning-based [71, 9] , and visual prompt tuning-based [68, 68, 28] methods. Extra-module-based methods (e.g., Adapter [51] , side-tune [72] ) introduce small additional learnable modules and keep the pre-rained backbone frozen. Partial-tuningbased methods (e.g., Bias [9] ) keep a large portion of the backbone frozen, and fine-tune only part of the foundation model to downstream tasks. Visual prompt-based PEFT methods (VPT) introduce a learnable prompt variable along with a learnable classification head over the fixed pre-trained backbone to be adapted to the downstream task. VPT [32] has shown significant improvement over other PEFT techniques, and can even outperform full fine-tuning in multiple datasets/settings [28] .Calibration in Deep Learning Models Calibration methods have been increasingly explored to achieve trustworthy deep-learning models. Post-hoc calibration methods [26, 43, 73] aim to learn a calibration map for a standard trained deep learning model such that the map can transform the poorly calibrated probabilities to calibrated probabilities. Regularization-based calibration approaches introduce explicit regularization (such as with L 2 regularization [26] , entropy regularization [50] ), or implicit regularization (such as with focal loss [39] ) during training to ensure that the trained model is calibrated. Data augmentation methods such as Label smoothing [44] , and mixup training [60] [74] have also been explored for developing calibrated deep learning models. Recent survey [65] provides a discussion of the most relevant works towards developing calibrated deep learning models.Most existing calibration methods are designed to tackle the over-confidence issue, which is more commonly observed for large models trained from limited data due to overfitting. We observe that fine-tuned foundation models exhibit severe under-confidence in their predictions, where existing calibration techniques are less effective. To this end, we propose a lightweight Bayesian framework that fills this critical gap.Few-Shot Adaptation and Relationship with Meta-Learning. In this work, we consider few-shot adaption with a focus on N -way K-shot classification [64, 22] , where the model is presented with a few-shot training set with N -class, each having K examples. For instance, 1-shot Cifar100 training set consists of 1 sample from each of the 100 classes. The model is then evaluated on the test set, which is identical to the query set in the meta-testing tasks [58] . It is worth to note that meta-learning (e.g., matching networks [64] , MAML [16] , VERSA [23] , PLATIPUS [17] ) leverages an episodic learning paradigm to achieve few-shot adaptation, where both meta-training and meta-testing are done on the task level in an episodic fashion [69, 37] with a large number of N -way K-shot training tasks. In this work, we consider more challenging few-shot adaptation tasks (e.g., 100-way 1-shot in Cifar100 and 102-way 1-shot in Flowers102) compared to the commonly used 5-way 1-shot meta-learning tasks. We leverage the power of the pre-trained foundation models, which eliminates the need of task based episodic meta-training. From a meta-learning perspective, the pre-training phase for the foundation model could be viewed as performing meta-knowledge acquisition similar to meta-training. The pre-trained model can be seen as an expert equipped with the meta-knowledge, and parameter-efficient fine-tuning performs quick adaptation to the downstream tasks, analogous to the support-set based adaptation done in meta-testing.EQUATION3. Incorrect belief-sum based regularization as in Units-ML [46] L Units reg (x, y)EQUATIONAll three regularizations guide the model to minimize the incorrect evidence(ideally close to zero). In our experiments, we consider KL-based incorrect evidence regularization. 
bayesian parameter-efficient fine-tuning of foundation models: 
 We start by introducing some fundamental concepts from evidential learning, which will serve as key building blocks in the proposed Bayesian-PEFT framework. We then detail the two Bayesian components: base rate adjustment to address under-confidence and diversity-inducing evidential ensemble to improve the reliability on both prediction accuracy and uncertainty quantification.Evidential Deep Learning (EDL) [56] introduces a computationally efficient framework to transform deterministic deep learning (DL) models into uncertainty-aware models. The key idea is to introduce a higher-order conjugate prior distribution over the predicted likelihood distribution and train the DL model to output parameters of the higher-order distribution. Towards classification, EDL models [56, 11] introduce Dirichlet prior distribution for the multinomial likelihood distribution. Specifically, the output softmax layer of the DL model is replaced by a monotonic, non-negative transformation function (e.g., ReLU, SoftPlus, or exp) to obtain the evidence for different classes that are transformed into the Dirichlet parameters. Mathematically, for a DL model f θ (•), and an input sample x, we havee i = E f θ (x) i α i = e i + a i × W (1)where e i is the output evidence for the i th class from the model f θ (•) and input sample x, a i is the base rate for the i th class, W is the non-informative prior weight usually set to the number of classes, E is the non-negative transformation function, and α i parameterizes a Dirichlet distribution.Existing EDL works usually adopt a non-informative base rate ofa i = 1 N ∀i ∈ [1, N ].Furthermore, a multinomial distribution Mult(y|p) over labels is parameterized as E[p i ] = αi S , where the total Dirichlet Strength S = N i=1 α i . An evidential model can be trained via a Type-II Maximum Likelihood-based evidential loss L Log (x, y) [56] with KL regularization that penalizes evidence assigned to non-ground-truth classes:EQUATIONwhere α = y + (1 -y) ⊙ α. Once trained, the evidential model can predict an evidence vector e = (e 1 , e 2 , ...e N ) ⊤ for a given test sample x. From the predicted evidence, we obtain the model's belief (b) over different classes, the correct belief b cor , incorrect belief b inc , and vacuity u asEQUATIONwhere vacuity u is a second-order uncertainty [33] that captures the model's lack of knowledge in its prediction; b cor and b inc quantify model accuracy and error, respectively. However, neither b cor nor b inc can be evaluated without the ground-truth label, which is not available in the testing phase.Existing theoretical work has established an important connection between b inc and dissonance [46] , which is another second-order uncertainty [33] that can be quantified without the ground-truth label. More specifically, dissonance dis can be evaluated asEQUATIONwhere Bal(•, •) is the relative mass balance function between two belief masses. The dissonance essentially captures the conflicting belief assigned to different classes [57] .To gain a deeper understanding of the under-confident few-shot adaptation behavior of foundation models, we perform fine-grained uncertainty analysis using the predicted evidence from an evidential model. To this end, we replace the softmax layer in a VPT fine-tuned transformer model with an exponential-based evidential head that outputs non-negative evidence for different classes. We analyze the output evidence from the evidential model that reveals some interesting insights.Why is the model accurate? First, we observe that the relative order of the evidence assigned to different classes is accurate. This implies that the model outputs relatively greater evidence for the correct class compared to all other classes that ensure the model's strong predictive performance. To more precisely quantify the model's accuracy, we adapt the lower bound of incorrect belief theorem developed for meta-learning [46] to evaluate the model accuracy through its predicted dissonance: Theorem 1. Consider an evidential model that outputs incorrect belief of b inc and the dissonance in the beliefs is dis. Then, the incorrect belief of the model will be at least half of the dissonance for all predictions from the evidential model.EQUATIONFigure 3a shows the test accuracy vs. dissonance curve, which is aligned with the relationship between the incorrect belief and the dissonance given in the theorem above, where a low dissonance implies a low b inc (or a high accuracy). From all the testing samples, we observe relatively low dissonance and the highest is only slightly above 0.7 as shown in the figure. We further evaluate the Area Under the Curve of the Accuracy vs. (1dis) and obtain an AUC of 0.82 as shown in Figure 3b . This implies the model is able to clearly discriminate the ground-truth label from the rest without much confusion (i.e., low dissonance) which ensures its good prediction accuracy. Why is the model under-confident? Despite being able to assign relatively more evidence to the correct label over the rest, it is also interesting to observe that the model generally assigns very low evidence to all the labels, including the correct one. Figure 3c shows the evidence distribution of one representative test data sample from Cifar100. As can be seen, most classes are assigned very low evidence that is close to zero. The ground-truth class is assigned higher evidence, but it is far from sufficient to make the prediction confident. The resultant confidence is only 0.025 while the vacuity is extremely high at 0.875, implying that the model believes it has very limited knowledge of the data sample despite it correctly identifying the correct label. Figure 3d shows the predicted vacuity over all the test samples, most of which are assigned a very high vacuity. This confirms the overly conservative behavior of the model, where the low confidence is primarily due to the insufficient allocation of evidence in its predictions. On the other hand, since the model is fairly accurate, it is reasonable to believe that the model underestimates the contribution of the rich prior knowledge gained through pre-training.Base rate adjustment to strength the prior belief. The fine-grained uncertainty analysis through the lenses of evidential learning not only explains the good predictive performance of the few-shot adapted model through PEFT but also unveils the root cause for its under-confident behavior, which is under-estimation of the contribution from the prior knowledge to the downstream task. While the classical Bayes' theorem offers a principal idea to address the issue, which is to strengthen the prior belief, there is a lack of practical way to achieve this. To this end, we propose to leverage the base rate introduced by the subjective logic theory [33] as an effective vehicle to adjust the prior belief gained through pre-training. According to (1) , adjusting the base rate has the effect of changing the Dirichlet parameter α, which will change the confidence for the prediction given by max i E[p i ]. However, base rate adjustment needs to meet two key requirements: (1) the relative order of the Dirichlet parameters assigned to different classes should be preserved so that the predictive performance of the model remains unaffected, (2) the gap between the Dirichlet parameters for different classes is transformed such that the model becomes more confident in its predictions, making it well-calibrated. To meet these requirements, we we propose a transformation function A m to the model's output evidence such that the model is well calibrated without any compromise in the generalization performance:EQUATIONwhere χ = (χ 1 , χ 2 , ...χ N ) ⊤ is the adjusted base rate, and m ≥ 1 controls the base rate transformation. The adjusted base rate χ considers evidence of all classes as a reference via e min , and transforms the gap between different class evidences such that the model is well calibrated. Lemma 2. The base-rate adjusted model that uses learnable base rate χ = (χ 1 , χ 2 , ..., χ N ) ⊤ has the same generalization performance compared to the model using fixed base rate ofa i = 1 N ∀i ∈ [1, N ] Theorem 3.For any m ≥ 1, the transformation function A m transforms the base rate for the class with the highest evidence e max and class with the second highest evidence e 2nd such that the gap in Dirichlet parameters between the two classes is non-decreasing. Remark. Theorem 3 ensures that the expected probability E[p i ] for the predicted class i has an increased gap with the rest of the classes, which results in an increase of the model's confidence. Therefore, if the prediction is accurate, the model's calibration performance will be improved. Meanwhile, Lemma 2 ensures that the good prediction accuracy of the model is maintained by the proposed base rate adjustment strategy. The detailed proofs are given in Appendix D.The second Bayesian component of the proposed B-PEFT framework aims to further improve the reliability of both prediction accuracy and uncertainty quantification when performing few-shot adap- The effectiveness of the ensembles has been empirically demonstrated across multiple datasets/settings [30] with theoretical guarantees [2] . However, standard deep ensembling leads to limited diversity among the ensemble components as it only considers the random initialization of components. We propose a novel diversity-inducing ensembling scheme for the evidential models. Similar to the deep ensemble [36] , we also consider randomly initialized evidential models. We additionally train each ensemble component with different strengths for incorrect evidence regularization along with evidential loss objective. The overall objective for each ensemble component is identical to (2).However, each ensemble component is trained with different incorrect evidence (or belief) regularization strengths (i.e., different components place different priorities for the minimization of incorrect evidence over the maximization of correct evidence) which leads to diversity among the components.Since each component's priority for minimizing the incorrect evidence is different, the components focus on different attributes/features in the data that help the model avoid overfitting to an identical set of discriminative features. As a result, the proposed evidential ensembling scheme implicitly pushes the ensemble components away from each other, making it equivalent to the repulsive force in the Stein Variational Gradient Descent (SVGD) [12, 13] .Lemma 4. For given incorrect evidence regularization L inc reg , and E ensemble components with regularization strengths λ p , p ∈ [1, P ], the ensemble components in the evidence space are implicitly pushed away from each other by a force λ p ∇L inc reg that acts identical to the repulsive force in Stein Variational Gradient Descent (SVGD) based ensembles. Remark. The detailed proof is given in the Appendix. We present an intuitive visualization of the update of the evidential ensemble model for different strengths (λ 1 < λ 2 < ... < λ P ) of incorrect evidence regularization for different seeds in Figure 5 . Each ensemble component aims to maximize the likelihood (direction -→ A ) and minimize incorrect  
experiments and results: 
 Experiment setup, datasets, and baselines. We consider K-shot adaptation (i.e., the dataset has K examples per class in the training set) with Cifar10 [1] , Cifar100 [1] , Food101 [8] , and Flowers102 [45] datasets. For instance, the 2-shot Cifar100 dataset has 2 examples per class leading to a total of 200 labeled training samples. For all datasets and experiments, the training set is a few-shot dataset, and the evaluation is done on the standard test set available with benchmark datasets. Details of the few-shot training datasets along with additional experiment details are presented in the Appendix E. We consider large pre-trained vision transformer with ViT backbone [15] and consider Visual Prompt Tuning (VPT) [32] , along with bias fine-tuning [9] and adapter fine-tuning [72] as the PEFT techniques (We use VPT as the representative PEFT where not specified due to its superior performance). We consider accuracy-preserving post-hoc calibration techniques including Temperature Scaling (TS) [26] , Parameterized Temperature Scaling (PTS) [61] , and Isotonic Regression (IR-MC) [6] as the baseline calibration techniques.Prediction and calibration performance. We first consider standard cross-entropy (CE)-based PEFT of the supervised pre-trained ViT model on few-shot datasets. We present the accuracy and calibration results of VPT in Table 1 (a). We observe that the straightforward adaption of the models leads to accurate but under-confident models as indicated by a high ECE. The evidential models as shown in Table 1 (b) have comparable or better generalization performance across the datasets/settings. However, these models are also severely under-confident similar to CE-based models indicated by high ECE and accuracy-confidence trends (see Figure 11a , 11b in the Appendix). The overall performance of the calibrated evidential model using base-rate adjustment is presented in Table 1 (c). As can be seen, the accuracy remains the same as the adjusted base rate expands the gap between evidence of the class and preserves the relative order in the predicted class evidence. It effectively tackles the under-confidence issue, which leads to a significant improvement in the overall ECE performance across the datasets and settings (also see Figure 11c in the Appendix). Table 1 (d) shows the results of the proposed B-PEFT model that introduces a diversity-enforcing ensemble of calibrated evidential models trained with different strengths of incorrect evidence regularization. performance. Such a learning signal helps the model avoid overfitting (addressing the potential overconfidence issue) and leads to further improvement in generalization and the calibration.We further carry out experiments with additional PEFT techniques of bias and adapter fine-tuning and on the K-shot Cifar100 dataset in Table 2 . We observe that these PEFT techniques lead to a lower generalization performance (as indicated by a lower test accuracy) compared to the VPT-based technique. Nevertheless, the same underconfidence issue remains as shown in Comparison with existing calibration techniques. We compare our proposed evidential calibra tion technique with existing calibration techniques using a representative Cifar100 dataset on different few-shot classification settings. We consider Isotonic Regression (IR-MC) [6] , Temperature Scaling (TS) [26] , and Parameterized Temperature Scaling (PTS) [61] as baselines. Overall results are presented in Table 3 . The baseline model and its evidential extension are poorly calibrated. All calibration techniques transform the logits to address the under-confidence issue of the model. Our proposed Base-rate adjusted Evidential model (BR-Evid) addresses the under-confidence issue in the evidential model leading to significant improvement in the ECE. B-PEFT model further improves on the BR-Evid model as it effectively addresses both the under-confidence issue (via the base rate adjustment) and overconfidence issue (via the Bayesian ensembling) that leads to superior calibration results as indicated by the lowest ECE in all few-shot settings.Uncertainty quantification results. We investigate the uncertainty quantification performance of our proposed calibrated evidential ensemble model. The developed model can reflect the model uncertainty (i.e., the model's confidence in its predictions) through the ensemble agreement/disagreement and the distributional uncertainty through the sharpness of evidential prior distribution (i.e., the vacuity). For the analysis, we use Cifar10 as in-distribution (ID) dataset for different shots, and Cifar100 as OOD dataset. As shown in Figure 6 , we plot vacuity and variance distribution for 1 and 5 shots. A single model outputs the vacuity that can be used to detect OOD samples. We can see ID samples are in the lower vacuity region and OOD samples are in the higher vacuity region.As the number of shots increases, the region becomes more separate. However, for 1 shot, there are some OOD samples in the lower vacuity region as well. Since we can not trust vacuity alone for uncertainty, we utilize the variance of ensemble components to quantify model uncertainty. As we see in Figure 6 (c), the variance of OOD samples mostly lies in the high variance region. As we increase the number of shots, the variance shifts towards the lower region. High variance indicates that model-predicted vacuity can not be totally trusted. This behavior is qualitatively shown in Figure We observe that when only a single component outputs a low vacuity for the OOD samples in Figure 7 (a-b ), the variance of the ensemble is high, implying a high model uncertainty. When all the components output a high vacuity to the OOD sample in Figure 7 (c), the variance is also low, implying a low model uncertainty. Ablation study. We carry out ablation with 1-shot Cifar100 dataset to study the impact of the base rate transformation order m (Section 3.2) for different strengths of incorrect evidence regularization on the calibration performance. As we increase m, the probability gap between classes improves, leading to more confident predictions. However, the model starts to become overconfident for large m values (see Figure 8 ).Moreover, with an increase in incorrect evidence regularization strength, the optimal m value decreases to a smaller value (e.g., optimal m = 2.0 for λ = 0.1, and optimal m = 1.5 for λ = 10). Choosing a proper m leads to the best-calibrated evidential model. We present additional results studying the impact of m in Appendix F.1.Limited by space, we provide results comparing meta-learning methods on standard few-shot tasks in Appendix F.4, discuss impact of various components (number of classes, data size, and unfrozen parameters) in Appendix F.5, and include additional experiments and comparisons, including OOD settings, in Appendix F.6. Moreover, we carry out additional ablation experiments to study the impact of incorrect evidence regularization strength (Appendix F.2), and the impact of ensemble components (Appendix F.3). We further carry out experiments and discuss applying PEFT to foundation models pre-trained in a self-supervised fashion (Appendix G). We discuss the societal impact and limitations of the work in Appendices H and I, respectively. 
conclusion: 
 In this work, we focus on transformer-based large vision foundation models and investigate different parameter-efficient fine-tuning techniques for effective few-shot adaptation. We observe that the existing models are severely under-confident, especially in challenging datasets and settings. Moreover, existing models lack fine-grained uncertainty quantification capabilities. We extend the models to uncertainty-aware evidential models, and resort to the evidential framework to develop a novel Bayesian parameter efficient fine-tuning (B-PEFT) framework that integrates evidence-based base rate adjustment to addresses the under-confidence and a diversity inducing evidential ensemble technique to further improve the reliability in model prediction and uncertainty quantification. The B-PEFT framework possesses theoretically sound properties to ensure its superior generalization capability and robust calibration behavior. We carry out intensive experiments across different benchmark datasets and diverse few-shot settings that demonstrate the outstanding performance of B-PEFT. A Organization of the Appendix• In Section B, we present the table with summary of the key symbols used in this work.• In Section C, we present related works in uncertainty-aware deep learning, describe evidential deep learning for classification in details, and discuss some key advantages of using evidential models to address the under-confidence issue over standard Bayesian models. • In Section D, we present proof of all our theoretical claims.• In Section E, we present the details of hyperparameters and additional experiment details.• In Section F, we present additional experiment results including the impact of m for calibration performance, comparison results with meta-learning methods, the impact of the number of classes, data size, and unfrozen parameters, comparison in OOD settings, the impact of incorrect evidence regularization strength, the results of additional PEFT methods, and the impact of ensemble components. • In Section G, we discuss the calibration and accuracy behavior for PEFT of large foundation models pre-trained in a self-supervised fashion. • In Section H, we discuss societal impact of our work.• In Section I, we discuss the limitations of our work and present potential future direction.The source code for the experiments carried out in this work is attached in the supplementary materials and is available at the link: https://github.com/ritmininglab/B-PEFT Accurate quantification of predictive uncertainty is essential for the development of trustworthy Deep Learning (DL) models. To this end, DL models have been augmented to become uncertaintyaware using a variety of approaches such as ensemble-based approaches [36, 49] , bayesian neural networks based approaches [42, 19, 7] , and deterministic neural network based approaches [56, 11, 3] . Deep ensemble techniques [36, 49] construct an ensemble of neural networks, and the agreement/disagreement across the ensemble components is used to quantify different uncertainties. Alternatively, Bayesian neural networks [19] [7] [42] have been developed that consider a Bayesian formalism (e.g., bayes-by-backprop [7] , dropout during test [19] ) to quantify different uncertainties. 
 
 Appendix: ABNN [18] introduces Bayesian normalization layers after training of deep learning models, and requires additional training of these layers in a post-hoc manner. Deterministic neural network-based approaches [57, 41, 10] extend the existing neural network to become uncertainty-aware and enable the networks to quantify fine-grained uncertainties with a single forward pass through the network. Evidential deep learning models [56, 5, 75, 10, 10, 62] , an instance of deterministic approaches, introduce a conjugate higher-order evidential prior for the likelihood distribution to enable the model to express the fine-grained uncertainties in both classification [56, 11] and regression problems [3, 47] . Towards classification, evidential models [56, 5, 75] introduce higher-order evidential Dirichlet prior to the multinomial likelihood that enables the deterministic neural network model to capture different uncertainty characteristics. In what follows, we first provide some additional details on using evidential learning model to perform classification. We then highlight some important advantage of using evidential models over standard Bayesian models in uncertainty quantification.Evidential Deep Learning models, based on Subjective Logic theory [33] , aim to train the model such that for any new input sample, the model can make predictions, as well as output fine-grained uncertainty information (via vacuity [56] and dissonance [34] ). Towards capturing fine-grained uncertainty for classification problems, EDL models assume that the label for each sample is obtained from a generative process with a Dirichlet prior and a multinomial likelihood. The parameters for the Dirichlet prior express the vacuity and belief masses for uncertainty estimation. The conjugacy between the Dirichlet prior and the multinomial likelihood is explored, and different evidential losses are introduced for model training and inference [48] . In this work, we consider Type-II Maximum Likelihood-based evidential loss L Log (x, y) [56] with incorrect evidence regularization L inc reg (x, y) given by [56] EQUATIONWe replace the softmax layer in the head of the VPT model with exp activation function. To avoid zero evidence regions, we also include the correct evidence regularization L cor (x, y) = -λ cor log(α gt -1) (The incorrect evidence regularization guides the model to minimize the evidence for all classes other than the ground truth class and can take one of the following forms 1. KL-based incorrect evidence regularization term as in EDL [56] L EDL reg (x, y) = KL Dir(p| α)||Dir(p|1)EQUATIONWhere α = y + (1 -y) ⊙ α = (α 1 , α2 , ...α N ) parameterize a dirichlet distribution, αi=gt = 1, αi = α i ∀i ̸ = gt, and ⊙ represents element-wise product. Here, the KL regularization term encourages the Dirichlet distribution based on the incorrect evidence i.e., Dir(p| α) to be flat which is possible when there is no incorrect evidence.As compared with the Bayesian-inspired models, evidential learning offers two key properties that allow us to formulate a principled solution to address the unique under-confident behavior of the PEFT methods. First, thanks to its evidence-based fine-grained uncertainty decomposition capability, we can separate two distinct sources of second-order uncertainty, including vacuity and dissonance.Different from the commonly used first-order uncertainty (e.g., entropy), these two second-order uncertainty serve as a key tool to understand why PEFT methods are both accurate (with a low dissonance) while being under-confident (with a high vacuity). This key insight suggests that these methods systematically under-estimate the contribution from the prior knowledge to the downstream task. While the classical Bayes' theorem offers a principal idea to address the issue, which is to strengthen the prior belief, there is a lack of practical way to achieve this. As the second key property, evidential learning allows us to leverage the base rate, which is rooted in the subjective logic theory as an effective vehicle to adjust the prior belief gained through pre-training. To this end, we propose a transformation function in Eq. ( 6) to adjust the base rate that leads to the increase of the model confidence while maintaining the predictive accuracy of the model as guaranteed by our theoretical results in Lemma 2 and Theorem 3. Furthermore, we develop belief-based diversity for ensemble of evidential models leading to the the B-PEFT model. In theory, evidential deep learning model could be augmented with the Bayesian normalization layers [18] or Bayesian neural networks [7] as an alternative to belief-based diversity of B-PEFT. We leave exploration of different techniques for diversity for Bayesian evidential model as a potential future work.In this section, we provide the proofs of the major theoretical results presented in the main paper. For the evidential model with fixed base rate of a i = 1 N ∀i ∈ [1, N ], the model's predicted class is given by c pred = arg max(e 1 + a 1 × W, e 2 + a 2 × W, ..., e N + a N × W ) = arg max(e 1 + 1, e 2 + 1, ..., e N + 1) = Index(e max ). For the calibrated model with learnable χ = (χ 1 , χ 2 , ..., χ N ) ⊤ , the model's predicted class is given by c new pred = arg max(α 1 , α 2 , ..., α N ) = arg max(e 1 + χ 1 × W, e 2 + χ 2 × W, ..., e N + χ N × W ) = arg max(e 1 + N ( e1 emin ) -N, e 2 + N ( e2 emin ) -N, ..., e N + N ( e N emin ) -N ). Since e max ≥ e i ≥ e min ∀i ∈ [1, N ], α max ≥ α i ≥ α min ∀i ∈ [1, N ],a i = 1 N ∀i ∈ [1, N ],the difference between the Dirichlet parameters for class with maximum evidence and class with the second maximum evidence is given by α max -α 2nd = e max + a max W -e 2nd -a 2nd W = e max -e 2nd as a i = 1 N ∀i ∈ [1, N ].For the calibrated model with learnable χ = (χ 1 , χ 2 , ..., χ N ), the difference between the Dirichlet parameters for class with maximum evidence and class with the second maximum evidence is given by α max -α 2nd = e max + χ max W -e 2nd -χ 2nd W = (e max -e 2nd ) + (χ max -χ 2nd )W . Now, Since ei emin ≥ 1∀i ∈ [1, N ], and e max ≥ e 2nd ≥, ..., ≥ e min , (χ max -χ 2nd ) ≥ 0∀m > 0. For e max > e 2nd , &m > 0, χ max -χ 2nd > 0. Thus, with the proposed learnable base rate, the gap between the two largest Dirichlet parameters is maintained whenever m = 0 and/or e max = e 2nd . Moreover, whenever m ≥ 1 and e max > e 2nd , the Dirichlet parameter gap between the two classes is increased by a factor of emax emin -1 m -e2nd emin -1 m .We first carry out an analysis of the update in Stein Variational Gradient Descent (SVGD) based ensembling [12, 13] that reveals the repulsive force acting among the ensemble components that pushes the particles away and introduces diversity. We then consider ensemble components with different strengths of incorrect evidence regularization L inc reg and analyze the update to the evidential model in the evidence space that reveals a repulsive diversity-enforcing force acting identical to the SVGD based ensemble. SVGD update involves randomly initializing the particles and iteratively updating the particles to match the target distribution, which is summarized below.Algorithm 1 SVGD Update Input: {x 0 i } N i=1 : A set of initial parameters, and target distribution density function p(x) For L iterations, iteratively update the particles as• x l+1 i = x l i + ϵ l φ * (x l i ), where φ * (x) = 1 E E e=1 k(x l e , x)∇ x l e log p(x l e ) + ∇ x l e k(x l e , x)Here, ϵ l is the step size at iteration l, and k(•, •) is the kernel function that measures similarity. Output: {x 0 i } N i=1 : A set of initial parameters, and target distribution density function p(x)For given incorrect evidence regularization L inc reg , and P ensemble components with regularization strengths λ p , p ∈ [1, P ], the ensemble components in the evidence space are implicitly pushed away from each other by a force λ p ∇L inc reg that acts identical to the repulsive force in Stein Variational Gradient Descent (SVGD) based ensembles.Proof. For simplicity, consider RBF kernel for k(•, •) i.e., k(a, b) = exp -1 h (a -b) 2 , two particles x 1 , x 2 , and analyze their updates in SVGD based ensembling. At iteration l, the update to particle x 2 is given byφ * (x l 2 ) = 1 2 k(x l 2 , x l 1 )∇ x l 1 log p(x l 1 ) + k(x l 2 , x l 2 )∇ x l 2 log p(x l 2 ) + ∇ x l 1 k(x l 1 , x l 2 ) + ∇ x l 2 k(x l 1 , x l 2 )In the above update,- → Q = k(x l 2 , x l 1 )∇ x l 1 log p(x l 1 ) + k(x l 2 , x l 2 )∇ x l 2 log p(x l2 ) aims to guide the particles in the direction that maximizes the likelihood, and the update directionEQUATION2 ) acts as the repulsive force. Considering the repulsive force- → R = ∇ x l 1 k(x l 1 , x l 2 ) + ∇ x l 2 k(x l 1 , x l 2 ) = ∇ x l 1 exp -1 h (x l 1 -x l 2 ) 2 + ∇ x l 2 exp -1 h (x l 1 -x l 2 ) 2 = 2 h (x l 2 -x l 1 )k(x l 1 , x l 2 )As can be seen, the repulsive force -→ R pushes the particle x l 2 in the direction away from particle x l 1 that introduces diversity. With more particles, each particle is updated in the direction that maximizes the likelihood, and the particle is pushed away from all other particles (by force R).Next, consider ensemble components with different strengths of incorrect evidence regularization L inc reg to analyze the update to the evidential model in the evidence space. For simplicity, we consider the incorrect evidence sum-based regularization similar to ADL without correct evidence regularization (The analysis is valid for all incorrect evidence regularization and for models with correct evidence regularization). For a model with incorrect evidence regularization, the overall evidential loss is given by:L evid (x, y) = L Log (x, y) + λ × L ADL reg (x, y) = log S - K k=1 y k log α k + λ × K k=1 e k × (1 -y k )The gradient of the loss with respect to logits (the output head layer, where e k = exp(o k )) is given byEQUATIONConsider K class classification problem. The gradient update to the logit layer for the evidential model is given byEQUATIONEQUATIONHere, y k ∈ [0, 1], y k = 1 if k = gt, and y k = 0 otherwise. Moreover, the λ value is varied, and different λ values lead to different evidential models.A pushes the evidential model in the direction that maximizes the likelihood (similar to -→ Q in SVGD-based update), and the force -→ B implicitly pushes the ensemble components away from each other (similar to the repulsive force -→ R in SVGD-based update). Each component moves in -→ B direction with a different force determined by the incorrect evidence regularization strength λ that ensures that the ensemble components are diverse. Due to the different strengths of incorrect evidence regularization, each ensemble component places a different priority level for minimization of incorrect evidence over acquiring correct evidence, which ensures that the ensemble components remain diverse.We consider ViT model backbone [15] that is pre-trained in a supervised fashion, and 4 benchmark datasets of Cifar10 [1] , Cifar100 [1] , Food101 [8] , and Flowers102 [45] . We consider few-shot adaptation with K-shot classification problem (We experiment with K values of 1, 2, 5, 10, and 20). The few-shot training set is constructed by randomly selecting K samples per class from the training set of the benchmark datasets. We consider 2-shot validation set for all datasets and settings. We train the model on the few-shot training set, use the 2-shot validation set for hyperparameter tuning, and evaluate all models on the benchmark test set with all the test set samples. We augment the few-shot training set and the few-shot validation set with resize, random horizontal flip, and cropping. The dataset details are also presented in Table 5 . We train all the models for 50 epochs on the few-shot training dataset with a batch size of 64 samples at each iteration and evaluate the model on the benchmark test set. The evidence is in the range [0, infinity], and some stability issues could potentially arise in extreme cases when the logit output is extremely low (i.e. close to negative infinity). In our experiments, we did not observe the stability issue. Still, the issue can arise in some extreme cases for which a small delta in the denominator could be introduced or the network's logits could be bounded to be greater than a small negative value. For the calibration baseline model of Parameterized Temperature Scaling (PTS) [61] , we consider a 2-layer neural network with 128 nodes in the hidden layer (we carry out hyperparameter tuning with 1-layer, 2-layer, and 3-layer networks and select the best model), and train with a learning rate of 0.00001. For Temperature Scaling [26] , we optimize for the temperature hyperparameter using Adam optimizer, and a learning rate of 0.01 (We also experiment with SGD optimizer, and learning rates of 0.1, 1.0, 0.01, and 0.0001 to select the best performing model). The evidential models use incorrect evidence regularization strength of (0, 0.1, 1.0, 10.0, 100.0, and 1000.0). For Isotonic Regression [6] , we consider multi-class setting and sklearn package. We use VPT [32] as the representative PEFT where not specified due to its superior performance. The key model performance results are averaged across 5 different runs to present the mean and the standard deviation. The experiments use Pytorch and are carried out on a workstation with NVIDIA RTX A6000 GPU. We carry out a grid search using the few-shot validation dataset and select the optimal m for the evidential models. In this section, we carry out detailed ablation to study the impact of m with few-shot Cifar100 datasets. We consider m values of (0.5, 1.0, 1.5, 2.0, 2.5, 3.0), incorrect evidence regularization strengths of (0.0, 0.1, 1.0, 10.0, 100.0, 1000.0), and few-shot values of (1, 2, 5, 10, 20) (see Table 6 and Figure 9 , Figure 10 ). Across all the experiments, we observe that as we increase m, the model transforms the evidence to maximize the evidence gap making the model more confident on its knowledge. With increased confidence, the model's calibration performance improves. However, a large increase in m values starts to make the model overconfident leading to increased ECE and poor calibration. The strength of incorrect evidence regularization also impacts the optimal value of m. For large values of incorrect evidence regularization, a smaller m value suffices to make the model well-calibrated. The trend is seen across all few-shot classification settings. Evidential deep learning models introduce incorrect evidence regularization to minimize the evidence of classes other than the ground truth class. In this work, we use KL divergence-based incorrect evidence regularization (see Section C.2), and introduce a hyperparameter λ that controls the priority the model places on minimizing the incorrect class evidence over maximizing the correct class evidence. In this section, we study the impact of the hyperparameter λ on the model performance with K-shot Cifar100 and Flowers102 experiments (We experiment with K values of (1, 2, 5, 10, 20)).We observe that the model's calibration performance (the ECE) is optimal when no incorrect evidence regularization is used i.e., λ = 0 (see Table 7 and Table 8 ). However, no incorrect evidence regularization hurts the model's generalization performance. With the increase in incorrect evidence regularization, the model's generalization performance (indicated by accuracy) improves albeit with ECE tradeoff. However, very large incorrect evidence regularization misguides the model to only focus on minimizing incorrect class evidence hurting generalization. The optimal λ value leads to the best generalization performance while hurting the ECE performance. Moreover, with a larger number of shots in training, the optimal λ value is generally smaller (For instance, in 1-shot Cifar100, optimal λ = 1000.0, in 2-shot Cifar100, optimal λ = 10.0, and in 5-shot Cifar100, optimal λ = 0.1).Once the optimal λ value is determined, we can resort to our evidential base-rate adjustment that leads to a calibrated evidential model with good generalization performance. We present the experiment to study the impact of ensemble components in Figure 12 . The experiment is performed on 1 one-shot Cifar100 dataset using prompt-based adaption for vision transformer. There is a performance gain of both accuracy and ECE with the use of all ensemble components. We also note that as we increase the number of components from 1 to 3, both the generalization and calibration performance increase significantly. For instance, the ECE with a single ensemble component is 0.088 which improves to 0.077 with 3 ensemble components while the accuracy improves by almost 3%.. However, with a further increase in the number of ensemble components, the generalization/calibration performance improvement is not significant. Thus, for our B-PEFT model, we carry out an ensemble of 3 evidential models that is a good balance between the number of ensemble components and the gain in generalization/calibration performance.In this set of experiments, we apply our model to the 5-way 1-shot mini-ImageNet testing tasks and compare it with representative meta-learning models. The results are summarized in Table 9 . Benefiting from the knowledge acquired during pre-training, the proposed PEFT-based model outperforms the episodic meta-learning models by a large margin, demonstrating the potential of large vision foundation models for effective few-shot learning. However, the VPT model is underconfident as shown in Table 10 and Figure 13 . Our B-PEFT model improves on the VPT model's generalization and calibration leading to promising few-shot adaptation results.Table 9 : 5-Way 1-Shot Mini-ImageNetMAML [16] 48.70 Matching Networks [64] 43.56 LLAMA [24] 49.40 VERSA [23] 53.40 PLATIPUS [17] 50.13 Bayesian-MAML [69] 53.30 Bayesian-TAML [37] 71 In this section, we study the under-confidence behavior of PEFT methods w.r.t. data size, number of classes, and number of unfrozen parameters. In this work, we focus on vision foundation models that are pre-trained in a supervised learning paradigm. These models have shown remarkable effectiveness in a wide range of areas including image classification, video understanding, and visual recognition among others. Alternatively, foundation models have been developed that pre-train in a self-supervised fashion (e.g., CLIP [52] ). These models demonstrate good zero-shot performance on various datasets. As a representative of the self-supervised models, we use CLIP in our experiments. Parameter-efficient methods [77, 76, 21] have been proposed to adapt the CLIP model for downstream tasks. We use the popular methods: adapter and prompt for few-shot adaptation to study the calibration behavior. The reliability plot of different shot adaptations (1,2 and 5 shots per class) for cifar100 along with accuracy and ECE is presented in Figure 17 . The accuracy behavior as we increase the number of shots from 1 to 20 is shown in Figure 16 . In both methods, we observe for few-shot adaptations, the accuracy is either lower or comparable to zero-shot performance. More specifically, for adapter-based adaptation, even with 5 shots per class, the accuracy does not reach zero-shot accuracy. On the contrary, the parameter-efficient fine-tuning of supervised foundation models shows consistent improvement in accuracy as we increase the number of shots.Towards calibration performance, we observe that prompt-based adaptation, along with zero-shot generally show good calibration performance with some degree of overfitting (see Figure 17 ). In contrast, prompt adaptation for supervised models that are severe, and prompt adaptation for CLIP models have no such issue. However, the adapter-based adaptation is mostly over-confident and hence has a higher ECE value than our proposed model. Considering these results for parameter-efficient few-shot adaption of pre-trained self-supervised models, the calibration and uncertainty behavior of such pre-trained models pose an interesting direction for further investigation.We study the parameter-efficient fine-tuning techniques for large pre-trained vision foundation models, where we identify two key issues: the under-confidence of the fine-tuned models in their 


Decision: The paper proposes a Bayesian Parameter Efficient Fine-Tuning framework for training large transformer-based foundation models and reducing underconfidence while ensuring reliable prediction in few-shot settings. The reviewers were broadly positive about the paper. They highlighted that they were generally convinced by the empirical evaluation presented in the paper and agreed that the problem of underconfidence due to PEFT is interesting and important. While no reviewer strongly recommends acceptance, I agree that the problem studied in the paper and the proposed solution are of interest to the NeurIPS community and that the paper should be presented at the conference.

Reviews: review: paper_summary: paper_summary: This paper advances the Bayesian Parameter Efficient Fine-Tuning (Bayesian-PEFT) approach: a strategy to fine-tune large pre-trained foundation models with well calibrated classifier and ability to gracefully deal with out-of-distribution (OOD) data thanks to uncertainty quantification from ensembling.
First the authors study the behavior of a fine-tuned foundation model with several PEFT methods (bias fine-tuning, adapters, side-tuning, visual prompt-tuning) in the few-shot learning regime (defined here as reducing the number of training samples per class, but keeping the original number of classes) and show that even though they achieve good accuracy, they all display under-confident predictions. The authors attempt to trace back the source of this behavior with evidential learning tools and conclude that the models are accurate thanks to relatively greater evidence of the correct class compared to other classes, and that models are under-confident because all classes are being assigned very low evidence, including the correct ones.
Inspired by this finding, the authors devise an evidential learning based PEFT strategy that adjusts the base rates of the evidential head such that prior belief from pre-training knowledge is strengthened while maintaining accuracy. Secondly the authors propose an evidential ensemble with a technique to induce diversity with different incorrect belief regularization strengths that penalize the model differently when assigning high beliefs to non-correct classes.
The proposed Bayesian-PEFT is evaluated on 4 datasets  (CIFAR10, CIFAR100, Food101, Flowers102) under different few-shot regimes, showing boosted accuracy and calibration.
strength_weakness: strength_weakness: ### Framing of few-shot learning task
- the few-shot learning setting considered here is quite different from the one considered in most of the few-shot learning/meta-learning literature with many small sets of support images and classes and query images (sampled from miniImageNet, Omniglot, CIFAR, etc.), sometimes called _"episodes"_. In contrast the proposed few-shot learning setting proposed here keeps the original number of classes and reduces the number of samples to 1-20 samples per class. This ends up as a different setting as the model is finetuned on 200 to 4000 images for CIFAR100 for instance from the 1-shot to the 20-shot setting, which is not a big dataset but still not that few images to train on.

- I'm wondering whether the proposed setup of having many classes, but few samples, may be the source of the under-confidence behavior of PEFT methods for the few-shot learning setup that is proposed here. In such cases, simple solutions from the few-shot learning literature that do not require training: e.g., cosine classifier [b], [c]  (i.e., averaging features from the same class, do dot-product with query features, then softmax on similarities)

- the authors could refer to this setting that they propose by a different name to avoid confusion with few-shot learning where models can be finetuned with very few samples and there are several Bayesian methods there already. A potential alternative could be: low-shot regime.


### Motivation for evidential learning approach and few baselines
- the formalism based on evidential learning is interesting, but is not always trivial to implement and previous works [d],[e] reported difficulties in scaling evidential methods and the Dirichlet predictor to larger number of classes and encoders.

- while the approach itself is clearly described, the motivation for using such a line of approach instead of others that are more simple, generic and with a wider adoption, e.g., last-layer methods like Laplace-Redux [f], Bayesian last layer [g]. Since the endeavor of this work is to look at more practical settings with few labeled samples and reliability constraints, it may make sense to compare this work with some simple baselines to show the benefits of the approach and its trade-offs.
Other potentially relevant baselines would be the cosine classifier from few-shot learning, LoRA ensembles [h], linear probing with test-time augmentation [i]

- It would be also interesting to expand the list of PEFT methods studied with more recent and potentially more parameter efficient ones like LoRA [j] or VeRA [k] that have been shown useful to large transformer networks and could expand the scope of this work beyond image classification.


### Missing related works
- as mentioned above, the field of few-shot learning/meta-learning has seen several Bayesian methods with uncertainty quantification. Often these meta-learning methods fine-tune the network at test-time with just a few labeled samples (from classes not seen during pre-training). Here is a non-exhaustive list of relevant works from this area: PLATIPUS [m], VERSA [n], LLAMA [o], Bayesian MAML [p], Bayesian TAML [q], etc.

- a discussion on how the current B-PEFT is different from them and what type of problems it addressed and with which benefits would be useful here.

- the idea of converting a pre-trained network into a Bayesian Neural Network and training over a short period of a few epochs (not a low-shot setting though) has been recently proposed in ABNN [l]. The diversity-inducing mechanism with different incorrect belief regularization strengths can also be related with the random prior mechanism from ABNN [l] where an ensemble of networks was obtained from a pre-trained network and the diversity was induced by introducing a different class-level prior for each ensemble member. ABNN is pretty recent so it can be considered as concurrent work. 


### Misc.
- the OOD experiment could benefit from the inclusion of typical metrics used in literature (e.g., AUROC, FPR95, AUPR, etc.) to allow easier comparison and putting the performance in context of other works.
 

### Minor - Novelty
- the evidential learning choice seems to be inspired to some extent by the multidimensional belief quantification work [37] that used uncertainty for vanilla few-shot learning / meta-learning. The authors however expand the idea further with belief strenghtening and incorrect belief regularization for ensembling.




**References:**

[a] Vinyals et al., Matching networks for one shot learning, NeurIPS 2016

[b] Gidaris et al., Dynamic Few-Shot Visual Learning without Forgetting, CVPR 2018

[c] Qi et al., Low-Shot Learning with Imprinted Weights, CVPR 2018

[d] Joo et al., Being Bayesian about Categorical Probability, ICML 2020

[e] Franchi et al., Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification, arXiv 2020

[f] Daxberger et al., Laplace Redux -- Effortless Bayesian Deep Learning, NeurIPS 2021

[g] Kristiadi et al., Being bayesian, even just a bit, fixes overconfidence in relu networks, ICML 2020

[h] Balabanov et al., Uncertainty quantification in fine-tuned LLMs using LoRA ensembles, arXiv 2024

[i] Ashukha et al. Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning, ICLR 2020

[j] Hu et al., Low-rank adaptation of large language model, ICLR 2022

[k] Kopiczko et al., VeRA: Vector-based Random Matrix Adaptation, ICLR 2024

[l] Franchi et al., Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models, CVPR 2024

[m] Finn et al., Probabilistic Model-Agnostic Meta-Learning, NeurIPS 2018

[n] Gordon et al., Meta-Learning Probabilistic Inference For Prediction, ICLR 2019

[o] Grant et al., Recasting gradient-based meta-learning as hierarchical bayes, ICLR 2018

[p] Kim et al., Bayesian Model-Agnostic Meta-Learning, NeurIPS 2018

[q] Lee et al., Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks, ICLR 2020
questions: questions: This paper takes an interesting direction of study: ensuring reliability of PEFT foundation models in the low-shot learning regime.
I find the endeavor of the authors nice, with a good story and several PEFT methods. However I do have several concerns regarding the chosen few-shot learning setup (different from the common practices in the literature) and not necessarily realistic, the limited motivation brought for the use of evidential learning as well as the limited baseline. In addition, the relatively rich literature of Bayesian methods for few-shot learning is ignored.

My current rating is leaning towards reject at this time (a bit on the fence), but I'm looking forward for the rebuttal.

Here are a few questions and suggestions that could be potentially addressed in the rebuttal or in future versions of this work (please note that suggested experiments are not necessarily expected to be conducted for the rebuttal):

1. Please argue why using evidential learning should be used for this setting rather than last-layer or other Bayesian inspired methods

2. What is the performance of a cosine classifier in this setting? What about Laplace-Redux or LoRA ensembles?

3. How is this method working with LoRA-like methods, e.g., VeRA?

4. Why considering the proposed few-shot learning setup instead of the one from legacy few-shot learning / meta-learning literature

5. Can the authors discuss the difference and benefits of Bayesian-PEFT w.r.t. Bayesian few-shot learning methods?

6. Inclusion of OOD specific metrics in the OOD experiment.
limitations: limitations: The authors addressed some limitations in the supplementary.
paper_summary: paper_summary: This paper studies the problem of parameter efficient fine-tuning. The authors pointed out mis-calibration issues caused by parameter efficient fine-tuning, which is, more specifically, under-confident estimation when fine-tuning data is limited. To solve this issue, the author proposed Bayesian PEFT, a Bayesian parameter efficient fine-tuning method that introduces two components, i.e., a base rate adjustment to strengthen the prior belief of pre-trained knowledge, and a evidential ensemble with diverse ensemble components. The authors provided both theoretical justification as well as extensive experiments across diverse datasets.
strength_weakness: strength_weakness: 1. It would be interesting to see more in depth discussion and empirical analysis on the observation of uncertainty of PEFT, particularly, related to data size and number of unfrozen parameters. 

2. More discussion related to OOD generalization might be needed. Particularly, how does the proposed method help OOD generalization in terms of prediction accuracy.
questions: questions: 1. Is fully fine-tuned model also have low uncertainty compared to PEFT model? 

2. Will the proposed method improve robustness such as OOD generalization in terms of model accuracy? 

3. How does the proposed approach work on other modalities, such as audio and language?
limitations: limitations: Yes.
paper_summary: paper_summary: This submission presented a lightweight Bayesian Parameter Efficient Fine-Tuning (Bayesian-PEFT) framework for large transformer-based foundation models. Experiments across diverse datasets demonstrated the improved calibration performance by Bayesian-PEFT on multiple other PEFT techniques.
strength_weakness: strength_weakness: * The dependence on Evidential Deep Learning is quite significant. To some extent, the proposed method extends EDL, rather than being designed specifically for PEFT of foundation models.

* Some editorial errors (e.g., "we we") exists, the submission may need further proofreading.
questions: questions: * In Eq. (6), would there be a numerical stability problem when dividing e_{min}, as this value can be very small?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper focuses on the parameter efficient fine-tuning (PEFT) for foundation models in the few-shot learning settings. The authors reveal that the adapted models lack accurate fine-grained uncertainty quantification capabilities. Specifically, the few-shot tuned models perform remarkable accuracy but with large expected calibration errors. To address this problem, they propose a lightweight Bayesian PEFT framework by designing two Bayesian components to improve models' confidence. Theoretical analysis and experiments on 4 visual benchmarks prove the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.The title may be a bit inappropriate. If the Bayesian-PEFT is claimed to be used for foundation models, experiments on more types of foundation models are required. For example, I am curious whether it still works well on large language models (LLMs), e.g., LLaMA3. Also, it is unclear whether the under-confidence issue observed in the ViT also exists in LLMs.
2.Although it may be beyond the scope of the paper's research, I am still curious whether the under-confidence issue studied in the paper disappears when there is sufficient training data.
3.It would be better if a schematic diagram of the proposed Bayesian-PEFT could be provided.
questions: questions: Does the proposed Bayesian-PEFT work well in LLMs, e.g., LLaMA3?
limitations: limitations: Yes

Comments: title: A kind reminder
Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
Post-rebuttal comments
comment: Dear Reviewer 7WZR,

We noticed that the updated score has not been reflected in openreview yet. Since the discussion deadline is approaching, we are grateful if you could make this update soon. Many thanks!
Dear Reviewer CbTK,

We’d like to express our gratitude again towards the reviewer for evaluating the proposed work and providing constructive comments. In our rebuttal, 

- We analyze under-confidence behavior with varying sizes of data.

- We provide the schematic diagram of B-PEFT method.

- We discuss the potential extension to LLM models. 

We are happy to answer any further questions that you may have regarding our work.
Dear Reviewer YJfL, 

We’d like to express our gratitude again towards the reviewer for evaluating the proposed work and providing constructive comments. In our rebuttal, 

- We emphasize the novelty of the proposed method to address the unique under-confidence behavior of the PEFT methods.

- We clarify the numerical instability issue that can arise during experiments and how to tackle the issue.

We are happy to answer any further questions that you may have regarding our work.
Dear Reviewer DMeg, 

We’d like to express our gratitude again towards the reviewer for evaluating the proposed work and providing constructive comments. In our rebuttal, 

- We conduct an in-depth analysis on the under-confidence behavior of PEFT methods w.r.t number of classes, data size, and the number of unfrozen parameters.

- We fully fine-tune the model to analyse the calibration behavior.

- We discuss the possible extension of the proposed method to other modalities, such as audio and language.

We are happy to answer any further questions that you may have regarding our work.
Dear Reviewer 7WZR,

Many thanks for carefully checking our rebuttal. We are happy that it has adequately addressed the main concerns and we will make sure to incorporate these changes into the revised paper. We also appreciate that the reviewer increases the rating to 6! 

We would like to take this opportunity to address the two additional inquiries from the reviewer. First, in the table below, we present the performance of the vanilla PEFT method for the OOD experiments, which is worse than ours. 

|     Setting    | Method | AUROC | AUPR | FPR95 |
| ------------- | --------- | --------- | ------- | -------- |
| 100 way 1 shot |  PEFT  | 79.53 | 80.09| 70.58 |
| 100 way 1 shot | B-PEFT | 81.24 | 81.98| 68.15 |
| 100 way 5 shot |  PEFT  | 90.93 | 90.75| 39.82 |
| 100 way 5 shot | B-PEFT | 92.58 | 92.85| 35.24 |

Second, for the few-shot protocol on CIFAR100, we have 3 settings. For 5-way 1-shot and 10-way 1-shot, we formulate a task by randomly selecting 5 and 10 classes, respectively. In each such task, the support set contains 1 sample and the query set contains 50 samples for each of the selected classes. For 100-way 1-shot, we formulate a task by randomly selecting 1 sample per class in the support set. The query set contains $100$ samples per class. The result for each setting is obtained by averaging the performance across 50 tasks.
I would like to thank the authors for the detailed and informative rebuttal. I imagine they invested a lot of effort into clarifying the concerns from the 4 reviewers and I'm confident they will improve this work.

I've read the other reviews and the responses of the authors to them and skimmed through the paper again.

The strong points I see in this rebuttal are:
- several clarifications on the motivation of using evidential models for this tasks and observed under-confidence phenomenon, clarification on few-shot learning setting
- new studies to better understand the under-confidence trend w.r.t. prompt size, number of tuned parameters, dataset size
- new results comparing with simple baselines that don't need training at all (cosine classifier, test-time adaptation) or limited training (LoRA, Laplace Redux) highlighting the effectiveness of this method but also the presence of under-confidence or low calibration to them
- results on typical few-shot learning/meta-learning protocols on mini-Imagenet
- reporting of typical OOD metrics in OOD experiments

A few minor things that could be improved:
- for the OOD experiments (Table 4 in the submitted PDF), only the performance of the proposed model is shown. It would be useful to report the performance of the vanilla PEFT method
- the description of the few-shot protocol for the CIFAR100 results could be added (number of sampled episodes in the evaluation)


Overall, I think the authors did a nice job in the rebuttal and provided convincing responses. 
I will raise my rating to 6 - Weak Accept.