Be Confident in What You Know: Bayesian Parameter Efficient Fine-Tuning of Vision Foundation Models
Authors: Deep Shankar Pandey
Spandan Pyakurel
Qi Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Model calibration
uncertainty quantification
few-shot adaptation
Parameter-Efficient Fine-Tuning
Primary area: machine_vision

Abstract: Large transformer-based foundation models have been commonly used as pre-trained models that can be adapted to different challenging datasets and settings with state-of-the-art generalization performance. Parameter efficient fine-tuning ($\texttt{PEFT}$) provides promising generalization performance in adaptation while incurring minimum computational overhead. However, adaptation of these foundation models through $\texttt{PEFT}$ leads to accurate but severely underconfident models, especially in few-shot learning settings. Moreover, the adapted models lack accurate fine-grained uncertainty quantification capabilities limiting their broader applicability in critical domains. To fill out this critical gap, we develop a novel lightweight {Bayesian Parameter Efficient Fine-Tuning} (referred to as $\texttt{Bayesian-PEFT}$) framework for large transformer-based foundation models. The framework integrates state-of-the-art $\texttt{PEFT}$ techniques with two Bayesian components to address the under-confidence issue while ensuring reliable prediction under challenging few-shot settings. The first component performs base rate adjustment to strengthen the prior belief corresponding to the knowledge gained through pre-training, making the model more confident in its predictions; the second component builds an evidential ensemble that leverages belief regularization to ensure diversity among different ensemble components.
Our thorough theoretical analysis justifies that the  Bayesian components can ensure reliable and accurate few-shot adaptations with well-calibrated uncertainty quantification. Extensive experiments across diverse datasets, few-shot learning scenarios, and multiple $\texttt{PEFT}$ techniques demonstrate the outstanding prediction and calibration performance by $\texttt{Bayesian-PEFT}$.

Be Confident in What You Know: Bayesian Parameter Efficient Fine-Tuuing of Vision Foundation Models
Deep Shankar Pandey f Spandan Pyakurel f Qi Yu"
Rochester Institute of Technology
spp797,, 11.4468,.i.yuuu. yu)@ri..
Abstract
Large transformer-ban foundation models have been commonly used as pre- trained models that be adapted to different challenging datasets and settings with state-of oftee-of-hha-rr generalization performance. Parameter efficient fine-tuning (PEFT) provides promising generalization performance in adaptation while incur- ring minimum computational overhead. However, adaptation of these foundation models through PEFT leads accurate but severely underconfident models, espe- cially in few-shot learning settings. Moreover, the adapted models lack accurate fine-grained uncertainty quantification capabilities limiting their broader applicabil- ity in critical domains. To fill out this critical gap, we develop a novel lightweight Bayesian Parameter Efficient Fine-Tuning (referred as Bayesian- -PEFT) frame- work for large transformer based foundation models. The framework integrates state-of-the fit-thherrt art PEFT techniques with two Bayesian components address the under-confidence issue while ensuring reliable prediction under challenging few- shot settings. The first component performs base rate adjustment to strengthen the prior belief corresponding to the knowledge gained through pre-training, making the model more confident in predictions; the second component builds an ev- idential ensemble that leverages belief regularization to ensure diversity among different ensemble components. Our thorough thcoretical analysis justifics that the Bayesian components ensure reliable and accurate few-shot adaptations with well-calibrated uncertainty quantification. Extensive experiments across diverse datasets, few-shot learning scenarios, and multiple PEFT techniques demonstrate the outstanding prediction and calibration performance by Bayesian-PEFT.
Introduction
Transformer- basedd foundation models have been developed as general models with state-of-the-as generalization performance [32, 66, 52, 28]. These models leverage the rich meta-knows weta-knowledge acquired during the pre-training stage to effectively adapt to complex downstream tasks [32], where pre- training is usually performed massive-scale annotated datasets (e.g., [35, 55]) through supervised [33, 66, 28] or self-supervised learning [52]. To achieve effective adaption, various parameter- efficient fine-tuning (PEFT) approaches have been developed [38, 27, 9, 54] that introduce small number of tunable parameters either within or outside of the backbone architecture to ensure good generalization capability while incurring little computational overhead because most parts of (or the entire) backbone architecture is frozen during fine-tuning [25, 59, 67]. Bias-fine tuning [9], representative partial tuning-based PEFT, only fine-tunes the bias parameters to downstream tasks. Adapter [51] and side-tune [72] fine-tuning techniques are instances of extra module-based PEFT that introduce extra parameterized modules and fine-tune them based on the downstream tasks. Visual Prompt-tuning [32] (VPT) follows the popular prompt learning paradigm by introducing a learnable
*Corresponding author, t equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Head
Frozen
Learnable
Head
Add
Backbone
Side Network
Transformer Encoder Layer
aywrNorm
Transformer Encoder Layer
LayerNorm
ECE Vs. Accaracy from different PEFT methods performing few (1-20)-hhe adaption
Add Nor
Transformer Encoder Layer
0.50
Biss
Multinead Attention
0.75
Sid Tune
Acapter
0.80
Prompi
B-PEРT
U 0.45
shot
Bins Tuning
BLearnable Prompts
0.30
shot
shat
weight
shot
Input
20 shot
bias
Accuracy
(a)
(b)
Figure 1: Accuracy Vs. ECE on CIFARI00 few-shot adaptation from different PEFT methods
prompt that is fine-tuned on the downstream task knowledge keeping the pre- pre-trained backbone frozen.
Despite the attractive generalization performance, most foundation models adapted to downstream few-shot tasks through PEFT exhibit somewhat surprising and undesirable behavior that may prevent them from being applied to many critical domains. Figure (b) shows the predictive accuracy versus the Expected Calibration Error (ECE) of foundation model after performing few-shot adaptation on CIFAR-100 using a series of representative PEFT methods, including VPT [32]. Adapter [51], Bias Fine Tuning [71], and Side-tune [72]. is clear that the adapted model is able provide accurate predictions even after fine-tuned on limited training samples. For example, all PEFT methods help to boost the model's accuracy to over 75% using just 5-shot fine-tuning and the accuracy reaches 80% after 10-shot fine-tuning. However, the adapted model is very poorly calibrated as shown by the large ECE scores, which consistently over 0.3 across all the fine-tuning methods. Increasing the fine-tuning size does not show clear improvement and sometimes even hurts the calibration performance. While one may expect the poor ECE to caused by over-confidence as we fine-tur large foundation model using very limited training samples, the detailed ECE plots as shown in Figure (a)-(c) reveal that the model to the 1-shot training dataset, the VPT fine-tuned model can already achieve test accuracy close to 50% but under-confident under-confidence issue is observed for all representative PEFT methods across different datasets and various few-shot learning settings as evidenced by our experiments.
fact severely under-confident, For example, after adapting
almost all its predictions leading to an ECE score over 0.45. The
The under-confident few-shot adaptation behavior of foundation models closely mimics how human experts with rich domain knowledge their own disciplines tend make conservative decisions when facing new tasks that deviate from their own expertise. Analogous to their human counterparts, the rich prior knowledge gained through the pre- pre-training stage foundational models overshadows the relatively limited knowledge obtained through few-shot fine-tuning, which prevents them from making more confident predictions in the downstream tasks. Unreliable uncertainty (i.e., confidence) quantification makes the predictions provided by these models less trustworthy, which may limit applying the promising "pre-train- meuiin-thenffine sice-ffietune" paradigm to many critical domains. As shown in Figure 2 (a)-(c), the fine-tuned model seldom makes any predictions with confidence over 80%, making it difficult to leverage these predictions in any high-stakes decision-malee process.
The need to balance between the rich prior knowledge gained through pre-training and the new knowledge obtained through few-shot adaptation inspires us investigate the under-confidence issue from the Bayesian perspective. In particular, we propose to look into the predictive behavior of the few-shot adapted foundation model through the lenses of evidential learning [56], which is built upon Bayesian theorem and subjective logic (SL) theory [33]. As part of the recent advances in modern Bayesian modeling, evidential learning provides a cost-effective way to perform Bayesian inference with the capability quantify fine-gra men-grained second-order uncertainty [57]. By leveraging the important

---

Accuracy 48, 19.155 ECE 0.451
Accuracy 47.435, ECE: 0.396
10
Acturaty: 34,863, ECE: 0.327
10
Acouracy: 49 49.155, ECE: 0.064
06
AccurFCI
0.8
0.
0.4
004
0.
04
10
0.
3.0
0.0
0A Confidence
0.
066
30
Confidence
Confidence
(a) VPT
(b) Adapter
(c) Bias
(d) B- B-PEFT
Figure 2: PEFT on 1-shot CIFAR dataset: all existing PEFT techniques exhibit severe under-
confidence while the proposed B-PEFT reduces the ECE by almost order of magnitude.
theoretical connection between fine-grained uncertainty and model accuracy [46], we unveil the underlying reason that supports the good predictive performance of few-shot adapted foundation model and the root cause for the under-confident behavior. Drawing from this important insight as outlined above, we propose to integrate the modern PEFT techniques into novel lightweight Bayesian framework, referred to as Bayesian-PEFT (B-PEFT), aiming to achieve highly reliable and accurate few-shot adaptations with well-calibrated and trustworthy uncertainty quantification.
The proposed Bayesian framework offers two important components to address the under- confidecne issue while ensuring reliable prediction under challenging few-shot settings. The first component makes novel adjustments to the base rates introduced by the SI. theory to strengthen the prior belief corresponding the knowledge gained through pre-training. Meanwhile, the adjustment does not change the relative order of the belief assigned to different classes, which ensures that the model accuracy is maintained. Our theoretical analysis shows that the proposed base rate adjustment strategy leads to more confident predictions (by increasing the gaps between the belief assigned to the ground- truth class and the rest) without compromising the model's accuracy. Figure 2d shows that B-PEFT significantly improves the model calibration. To further enhance the reliability of both prediction accuracy and uncertainty quantification when performing few-shot adaptation, the second component performs Bayesian model averaging by building diversity-inducing evidential ensemble. In addition using different random initialization of the PEFT components, diversity further enhanced through incorrect belief regularization that penalizes model for assigning high belief to non-ground-tree class. By controlling the strength of belief regularization, different ensemble components are guided to learn from diverse features in the data space, where light penalty allows an ensemble component to learn the common discriminative features while a heavy one will force an ensemble component to learn rare features to avoid errors on the difficult data samples. A deeper theoretical analysis of the proposed diversity-induces evidential ensemble is equivalent to Stein Variational Gradient Descent (SVGD) based ensembles [13]. Experiments on multiple challenging few-shot learning tasks justify the superior performance of B-PEFT over state-of-the-ar PEFT baselines, in terms of both prediction accuracy and uncertainty calibration performance. Our contributions can be summarized as follows:
We identify the severe under-confidence issue of pre-trained foundation models after perform- ing parameter-effiieet fine-tuning over few-shot datasets. The une-grained uncertainty analysis through evidential learning and SL. theory reveals the root cause for their good predictive perfor- mance while being under-confident.
We develop a novel lightweight Bayesian framework that integrates state-of-the-arr PEFT tech- niques with two Bayesian components to address the under-confidence issue while ensuring reliable prediction under challenging few-shot settings. The first component performs base rate adjustment strengthen the prior belief corresponding to the knowledge gained through pre- pre-training, making the model more confident in its predictions; the second component builds an evidential ensemble that leverages belief regularization to ensure diversity among different ensemble components.
We perform thorough theoretical analysis to justify why the proposed Bayesian components can ensure reliable and accurate few-shot adaptations with well-calibrated uncertainty quantification.
We carry out experiments with 4 benchmark datasets, different few-shot settings, and 3 parameter efficient fine-tuning techniques that demonstrate the effectiveness of the developed model.
2 Related Works
Parameter Efficient Fine Tuning of Foundation Models. Transformer-basee based foundation models [63, 15] have been developed as an improvement over traditional convolution- architectures
3

---

[29,3]] for computer vision tasks. The transformer-basee models achieve strong generalization performance [40] after training on large datasets. Moreover, the pre-trained transformers can be fine-tuned in limited data settings leading to state-of-the-art performance [32, 27]. As computation, memory, and parameter-efficient alternative to full fine-tuning of such large pre-trained transformers, different Parameter Efficient Fine Tuning (PEFT) approaches have been developed. PEFT techniques freeze most of the large transformer backbone, fine-tune the remaining backbone parameters and/or introduce lightweight extra modules for adapting to the downstream task. Existing approaches can broadly be categorized as extra-module-based [72, 54], partial--uing--bssed [71, 91, and visual prompt tuning- based [68, 68, 28] methods. satra-module--iiee
methods (e.g., Adapter [51], side-tune[72)) introduce small additional Iearnable modules and keep the pre-rained backbone frozen. Partial-tuning based methods (e.g.. Bias [9]) keep large portion of the backbone frozen, and fine-tune only part of the foundation model to downstream tasks. Visual prompt-based PEFT methods (VPT) introduce learnable prompt variable along with learnable classification head over the fixed pre-trained backbone to be adapted to the downstream task. VPT [32] has shown significant improvement over other PEFT techniques, and can even outperform full fine-tuning in multiple datasets/settings [28].
Calibration in Deep Learning Models Calibration methods have been increasingly explored to achieve trustworthy deep-learning models. Post-hoc calibration methods [26, 43, 73] aim to learn a calibration map for a standard trained deep learning model such that the map can transform the poorly calibrated probabilities to calibrated probabilities. Regularization-base calibration approaches introduce explicit regularization (such as with L2 regularization [26], entropy regularization [50]), or implicit regularization (such as with focal loss (39]) during training to ensure that the trained model is calibrated. Data augmentation methods such as Label smoothing [44], and mixup training [60][74] have also been explored for developing calibrated deep learning models. Recent survey [65] provides a discussion of the most relevant works towards developing calibrated deep learning models. Most existing calibration methods are designed to tackle the over-confidence issue, which is more commonly observed for large models trained from limited data due to overfitting. We observe that fine-tuned foundation models exhibit severe under-confidence in their predictions, where existing calibration techniques are less effective. To this end, we propose a lightweight Bayesian framework that fills this critical gap.
Few-Shot Adaptation and Relationship with Meta-Learning. In this work, consider few-shot adaption with a focus on N-way K-shot classification [64, 22], where the model is presented with a few-shot training set with N-class, each having K examples. For instance, 1-shot Cifar100 training set consists of sample from each of the 100 classes. The model is then evaluated on the test set, which is identical to the query set in the meta-testing tasks [58]. It is worth to note that meta-learning (e.g., matching networks [64], MAML [16]. VERSA [23], PLATIPUS (17I) leverages an episodic learning paradigm to achieve few-shot adaptation, where both meta-training and meta-testing are done on the task level in an episodic fashion (69,7] with a large number of N-way K-shot training tasks. In this work, we consider more challenging few-shot adaptation tasks (e.g., 100-way 1-shot in Cifar 100 100 and 102-way 1-shot in Flowers 102) compared to the commonly used 5-way 1-shot meta-learning tasks. We leverage the power of the pre-trained foundation models, which eliminates the need of task based episodic meta-training. From a weta-learning perspective, the pre-training phase for the foundation model could be viewed as performing meta-knowlest acquisition similar to meta- trainnga The pre-trained model can be seen as an expert equipped with the meta- weeta-nnowlddge and sarameter-efficieen fine-tuning performs quick adaptation to the downstream tasks, analogous to the support-set based adaptation done in meta-testing.
3 Bayesian Parameter- Fine- ToeeTTnnnggaun of Foundation Models
We start by introducing some fundamental concepts from evidential learning, which will serve as key building blocks in the proposed Bayesi an PP-PEFT framework. We then detail the two Bayesian components: base rate adjustment to address under-confidence and inversiy--inducin evidential ensemble improve the reliability on both prediction accuracy and uncertainty quantification.
3.1 Preliminaries
Evidential Deep Learning (EDL) [56] introduces a computationally efficient framework to transform deterministic deep learning (DL.) models into uncertainty-an models. The key idea is to introduce a higher-order conjugate prior distribution over the predicted likelihood distribution and train the DI. model to output parameters of the higher-order distribution. Towards classification, EDL models

---

[56, 11] introduce Dirichlet prior distribution for the multinomial likelihood distribution. Specifically, the output softmax layer of the DL model is replaced by a monotonic, non-negative transformation function (e.g., ReL.U, SoftPlus, or exp) to obtain the evidence for different classes that are transformed into the Dirichlet parameters. Mathematically, for a DL model fo(-), and an input sample x, we have
C = = a, = c4 + a; x W
(1)
where e is the output evidence for the Nt class from the model fo(-) and input sample x, az is the base rate for the jh class, W is the non-informative prior weight usually set the number of classes, E is the non-negative transformation function, and a; parameterizes Dirichlet distribution.
Existing EDL works usually adopt non-informative base rate of a; 1. Vi E [1, .N.. Furthermore,
a multinomial distribution Mult(y/p) over labels is parameterized as E Pil = S where the total
Dirichlet Strength = >:=1 Ox..
An evidential model can be trained via Type-1l Maximum Likelihood- evidential loss fLog (r, y) [56] with KL regularization that penalizes evidence assigned non-ground-truth classes:
Lav y) L (x: y)
where ā y (1 y) @ Once trained, the evidential model can predict an evidence vector (e1, e2; ENN)..N) for given test sample æ. From the predicted evidence, we obtain the model's belief (b) over different classes, the correct belief beors incorrect belief dinc. and vacuity u as
b b=
boor Eyrb
dinc E(1 y)
u S'
N
(3)
where vacuity a is a second-order uncertainty [33] that captures the model's lack of knowledge in its prediction; Deor and bine quantify model accuracy and error, respectively. However, neither beor nor Dine can be evaluated without the ground-trus label, which is not available in the testing phase. Existing theoretical work has established an important connection between bine and dissonance [46], which is another second-order uncertainty [33] that can be quantified without the ground-truth label. More specifically, dissonance dis can be evaluated
N dis Σ
(nn Eitnn by (yBalbb,bb) Ejjn bj
min(by b..) by
if b,b;
Bal(b;, bn)
0,
otherwise
where B))(;,) is the relative mass balance function between two belief masses. The dissonance essentially captures the conflicting belief assigned to different classes [57].
3.2 Strengthening the prior belief through base rate adjustment
To gain a deeper understanding the under-confident few-shot adaptation behavior of foundation models, we perform fine-grained uncertainty analysis using the predicted evidence from an evidential model. To this end, we replace the softmax layer in a VPT fine-tuned transformer model with an exponential-basee evidential head that outputs non-negative evidence for different classes. We analyze the output evidence from the evidential model that reveals some interesting insights.
Why is the model accurate? First, we observe that the relative order of the evidence assigned to different classes is accurate. This implies that the model outputs relatively greater evidence for the correct class compared to all other classes that ensure the model's strong predictive performance. To more precisely quantify the model's s accuracy, we adapt the lower bound of incorrect belief theorem developed for meta-learning [46] to evaluate the model accuracy through its predicted dissonance:
Theorem Consider an evidential model that outputs incorrect belief of benc and the dissonance in the beliefs is dis. Then, the incorrect belief of the model will be at least half of the dissonance for all predictions from the evidential model.
2 dis < Dine where 0 S dís < & 0 0.5 Dinc S
(5)
Figure 3a shows the test accuracy dissonance curve, which is aligned with the relationship
between the incorrect belief and the dissonance given in the theorem above, where a low dissonance implies a low binc (or a high accuracy). From all the testing samples, we observe relatively low dissonance and the highest is only slightly above 0.7 as shown the figure. We further evaluate the (1 - dis) and obtain AUC of 0.82 shown in Figure 3b. This implies the model is able to clearly discriminate the ground- -tunn label from the rest without much confusion (i.e. low dissonance) which ensures its good prediction accuracy.
Area Under the Curve of the Accuracy
5

---

1. Accuracy Dissonance Trend
AUC Curve
Confitence: 6,025, Venuly:: 0.875
Veruty Count
Tue Fosltvee Raae
1,00
Ground
Number of Samples
8000
Eddaancs
6000
0,50
4006
AUC 0.62
2006
3.00
Dissonance Threshold
0:25
0.50
D:75
5.00
0.25
2.50
0:75
1.00
20
60
60
80
100
8.6
0.2
0.4
Vacity
0.6
0.8
1.0
(a) Dissonance-Acurreyy
(b) AUC Curve
(c) Evidence of classes
(d) Vacuity Distribution
Figure 3: 1-shot Cifar10 results and evidence vacuity trends
Why is the model under-confident? Despite being able to assign relatively more evidence to the correct label over the rest, it also interesting to observe that the model generally assigns very low evidence to all the labels, including the correct one. Figure 3c shows the evidence distribution of one representative test data sample from 100 As can be seen, most classes are assigned very low evidence that is close to zero. The ground-truth class is assigned higher evidence, but it is far from sufficient to make the prediction confident. The resultant confidence is only 0.025 while the vacuity is extremely high 0.875, implying that the model believes it has very limited knowledge of the data sample despite it correctly identifying the correct label. Figure 3d shows the predicted vacuity over all the test samples, most of which are assigned very high vacuity. This confirms the overly conservative behavior of the model, where the low confidence is primarily due to the insufficient allocation of evidence in its predictions. On the other hand, since the model is fairly accurate, it is reasonable to believe that the model underestimates the contribution of the rich prior knowledge gained through pre-training.
Base rate adjustment to strength the prior belief. The fine-grained uncertainty analysis through the lenses of evidential learning not only explains the good predictive performance of the few-shot adapted model through PEFT but also unveils the root cause for its under-confident behavior, which is under-estimation of the contribution from the prior knowledge to the downstream task. While the classical Bayes' theorem offers a principal idea to address the issue, which is to strengthen the prior belief, there is a lack of practical way to achieve this. To this end, we propose to leverage the base rate introduced by the subjective logic theory [33] as an effective vehicle to adjust the prior belief gained through pre-training According to (I), adjusting the base rate has the effect of changing the Dirichlet parameter a, which will change the confidence for the prediction given by max; Elp.]. However, base rate adjustment needs to meet two key requirements: (1) the relative order of the Dirichlet parameters assigned to different classes should be preserved so that the predictive performance of the model remains unaffected, (2) the gap between the Dirichlet parameters for different classes is transformed such that the model becomes more confident in its predictions, making it well-calibrated. To meet these requirements, we we propose a transformation function Am to the model's s output evidence such that the model is well calibrated without any compromise in the generalization performance:
a = Am Am (x)() = e WX X; = a,
adj
64
Gmin
(6)
Emin
where X = (x1, ) is the adjusted base rate, and m 2 1 controls the base rate transformation. The adjusted base rate X considers evidence of all classes as a reference via Cmins and transforms the gap between different class evidences such that the model is well calibrated.
Lemma 2. The base-rate adjusted model that uses learnable base rate X = (X1, X2.... X T has the same generalization performance compared to the model using fixed base rate of a, = N Vi E [1, N] Theorem 3. For any m. 2 1, the transformation function Am transforms the base rate for the class with the highest evidence Cean and class with the second highest evidence Gand such that the gap in Dirichlet parameters between the two classes is non non-decreasing.
Remark. Theorem 3 ensures that the expected probability E'p.] for the predicted class has an increased gap with the rest of the classes, which results in an increase of the model's confidence. Therefore, if the prediction is accurate, the model's 's calibration performance will be improved. Meanwhile, Lemma ensures that the good prediction accuracy of the model is maintained by the proposed base rate adjustment strategy. The detailed proofs are given in Appendix D.
3.3 Building A Diversity Induced Evidential Ensemble
The second Bayesian component of the proposed B-PEFT framework aims to further improve the reliability of both prediction accuracy and uncertainty quantification when performing sew-shot adap-

---

Observed Variable
Latent Variable
Ensemble
Dirichlet Parameters
Base-rate adjustment
8
p(0)zr)
Ensemble
Input
Dirichlet Parameters
Base-rate adjustment
Ensemble
a
Dir(al0)
Predictions
Accuracy
P
Mult(pla)
ECE
Engemble E
Dirichlet Parameters
Base-rate adjustment
y
(a)
(b)
Figure 4: (a) Schematic diagram and (b) Graphical model of the B-PEFT model
tation. It performs Bayesian model averaging by building diversity-inducing evidential ensemble. The ensemble of deep learning models (i.e., deep ensemble) [20, 36] can effectively improve the gen- eralization performance of deep learning models. Moreover, deep ensembles capture the model uncertainty [36, 53] via the sniremment-disggeemmen uncertainty essentially captures the uncertainty in the model parameters, which is denoted graphical model of B-PEFT as shown Figure 4(b). The schematic diagram of B-PEFT is shown in Figure 4(a). The model uncertainty can be leveraged evaluate the reliability of fine-grained uncertainty output by the evidential model.
between the ensemble components. Model
of the
The effectiveness of the ensembles has been empirically demonstrated across multiple datasets/scttings [30] with theoretical guarantees [2]. However, standard deep ensembling leads to limited diversity among the ensemble components as it only considers the random initialization of components. We propose novel diversity-inducing ensembling scheme for the evidential models. Similar to the deep ensemble [36], we also consider randomly initialized evidential models. We additionally train each ensemble component with different strengths for incorrect evidence regularization along with evidential loss objective. The overall objective for each ensemble component is identical (2).
However, each ensemble component trained with different incorrect evidence (or belief) regulariza- tion strengths (i.e., different components place different priorities for the minimization of incorrect evidence over the maximization of correct evidence) which leads to diversity among the components. Since each component's 's priority for minimizing the incorrect evidence is different, the components focus on different attributesffeatures in the data that help the model avoid overfitting to an identical set of discriminative features. As result, the proposed evidential ensembling scheme implicitly pushes the ensemble components away from each other, making it equivalent to the repulsive force in the Stein Variational Gradient Descent (SVGD) [12, 13].
Lemma For given incorrect evidence regularization Cine and ensemble components with regularization strengths Xp. P E [1, P), the ensemble components in the evidence space are implicitly pushed away from each other by a force Xp V Lsse that acts identical to the repulsive force in Stein Variational Gradient Descent (SVGD) based ensembles.
Remark. The detailed proof is given in the Appendix. We present intuitive visualization of the update of the eviden- tial ensemble model for different strengths (A1 < A2 < < Xr) of incorrect evidence regulariza- tion for different seeds in Figure Each ensemble component aims to maximize the likelihood (direc- tion A) and minimize incorrect
Seed
A
Seed
Seed 2
A
4
Seed
ApB
ApB
Ensemble Components
Seed
Seed
Figure 5: Illustration of ensemble diversity achieved through incorrect belief regularization with different strength
7

---

Table 1: Prediction accuracy and ECE performance on few-shot adaptation
Cifar10
Cifar100
Food101
Flowers102
K (Shol)
Accuracy
ECB
Accuracy
1
ECE J
Accuracy
ECE
Accuracy
ECE 4
(a) Standard Model
1-Shot
69.378
0.037=0.010
48. 637 78777
0.393
35. 1.095
0.263_0.000
68.161
0.61
2-Shot
$1.771
0.400
64.501
300
0.494.0.002
53.95442..559
0.394
93.462 21.07
0.55
5-Shot
88.70729..22
0.255=..008
76.758 525
0.007
65.58620.14r
0. 0.424
97.363
165
0.47220.013
10-Shot
91.061 0.212=0.005
60.720 s20
0.501
71.566=0.000
0. 0.444_0.003
98. 98.244 20..244
0.439
20-Shot
92.678-0..7
0.166=000004
82.608 60.6008 266
0.487=0.004
74.914.0.178
0. 0.460=0.003
98. 08.43110.100
0. 0.42540.017
(b) Evidential Model
1-Shot
0.537=2..01 10.197=1.013
51.127
36.297 0.009
0.3499.0.014
89. 19.225511.0
0.840
2-Shot
81.613=1.796
0.553
65.545 330
52.855 0.620
0.485__0.006
95.071 +0.44
0.87440.006
5-Shot
88. 88.764
0.391 10.015
77.561 1...7....
0.744.0.006
65. 13.
0.536
97.602210.
0.68610.02 199
10-Shot
92.014
0.388
81.661+0
291
70.863-2..20 0.76520-002
0.673
98.326 2.:32
0.44420.00s
20-Shot
93.029 20
0.360
83. 03.1100a...
72.060 0.782g00001
0.599
98. g8.708
0.411go:0
(c) Base-rate adjusted Evidential Model (Calibrated Evidential Model)
1-Shot
10.
0.027=2.002
+1.127
0.077
36,297
0.081=0.011
89. 29.2225g1.0
0.025+00004
2-Shot
11.613
0.0400g..013
65.54510.330
0.08_.o000
52.855=o.0o1
0.063_00000
95.071 20.43
0.023
5-Shot
88. 88.7644.0.00
0.028=20006
77.561
0.04420.002
65, 25.133
0.037=0.00a
97 67..6022..10
0.015=:.002
10-Shot
92.014
0.019
81.561
0.034_0.002
70.863 10.261
0.054_a.00r
98.3261...
0.023_..000 233
20oShot
93.029
0.016=2.002
83. 00,1000+.. 184
0.031=0.00
72.060+0.500
0.050-a..022
98.778 98.708
(d) B-PEFT Model (Ours)
1-Shot
74.6742..0us
0.024go.0zz
152.335.0.00
0.067 20.001
38.775 20.
0.021
90.238 2.
0.023
2-Shot
83.865=0.735
0.022200002
67.563
0.056:g0001
5-1.661=0.017
0.020=0.001
95.715
0.021
5-Shot
90.556.20100
80.081 0.017=0.001
20.067
0.036.000000
66.548
10.10
0.034_0.0m
97.807
07.00720.006
20.066
0.014.0.000
10-Shol
92.9566..oes
0.014_00000
83.038
20,045
0.031
0.000
71.661
=0.212
0.038_..000
98. 0500
0.011:a.00
20-Shot
93.833 00.021
0.014 00
83.748 10.00s
0.030 200000
75.495
101
0.043 mmm
98.193 uay
0.010 go.0t
evidence (direction B). The strengths of incorrect evidence regularization (direction B) are different for each ensemble component that acts as an implicit repulsive force among the ensemble components, ensuring that they are diverse from each other. Different from the SVGD-based ensemble, in our proposed model, the particles do not need to explicitly communicate with each other making our proposed approach computationally efficient, scalable, and generalizable.
Experiments and Results
Experiment setup, datasets, and baselines. We consider K-shot adaptation (i.e., the dataset has K examples per class in the training set) with Cifar 10 [1], Cifar100 [1], Food101 [8], and Flowers 102 [45] datasets. For instance, the 2-shot Cifar100 dataset has 2 examples per class leading to a total of 200 labeled training samples. For all datasets and experiments, the training set is few-shot dataset, and the evaluation is done on the standard test set available with benchmark datasets. Details of the few-shot training datasets along with additional experiment details are presented in the Appendix E. We consider large pre-trained vision transformer with VIT backbone [155] and consider Visual Prompt Tuning (VPT) [32], along with bias fine-tuning [9] and adapter fine-tuning [72] as the PEFT techniques (We use VPT as the representative PEFT where not specified due to its superior performance). We consider eccuracy-preserving post-hoe calibration techniques including Temperature Scaling (TS) [26], Parameterized Temperature Scaling (PTS) [61], and Isotonic Regression (IR-MC) [6] as the baseline calibration techniques.
Prediction and calibration performance. We first consider standard cross-entropy (CE)-based PEFT of the supervised pre-trained ViT model on few-shot datasets. We present the accuracy and calibration results of VPT in Table 1 (a). We observe that the straightforward adaption of the models leads to accurate but under-confident models as indicated by a high ECE. The evidential models as shown in Table (b) have comparable or better generalization performance across the datasets/settings. However, these models are also severely under-confien confideentdeet similar to CE-based models indicated by high ECE and accuracy-cone confidence trends (see Figure 1 11a, 11b in the Appendix). The overall performance of the calibrated evidential model using base-rate adjustment is presented in Table 1 (c). As can be seen, the accuracy remains the same as the adjusted base rate expands the gap between evidence of the class and preserves the relative order in the predicted class evidence. It effectively tackles the under-confidence issue, which leads to a significant improvement in the overall ECE performance across the datasets and settings (also see Figure 1lc in the Appendix). Table (d) shows
8

---

the results of the proposed B-PEFT model that introduces a evidential models trained with different strengths of incorrect evidence regularization. performance. Such a learning signal helps the model avoid overfitting (addressing the potential overcon- fidence issue) and leads to further improve- ment in generalization and the calibration.
ensemble of calibrated
Table 2: Adapter and bias fine tuning results
Bias
Adapter
K (Shot)
Accuracy
f
ECE!
Accuracy 1
ECE 1
We further carry out experiments with addi- tional PEFT techniques of bias and adapter fine-tuning and on the K-shot Cifar dataset in Table 2. We observe that these PEFT techniques lead to a lower generaliza- tion performance (as indicated by a lower test accuracy) compared to the VPT-hased technique. Nevertheless, the same under- confidence issue remains as shown in Table 2 (a) for standard cross-entropy trained model performance, and in Table 2 (b) for their ev- idential extensions. We then carry out ex- periments using base-rate adjusted evidential model and our proposed B-PEFT model. The results are shown in Table 2 (c) and Table 2 (d). Base-rate adjusted evidential model and B-PEFT are equally effective with these PEFT techniques, and they address the under- confidence issue, which leads improve- ment in both generalization and calibration.
(a) Standard Model
1-Shot
35.514 420
0 0.296 023
46. 46.150 1.00
0.38610.010
2-Shot
55.098 44.92
0.38440.om one
66.78 20.554
0.513g0.00
5-Shot
74.203
267
0.383 002
78. 73.200022
0.503 000
10-Shot
79.141
1o. 203
0.336 to 002
81.589 10.001
0.470110.000
(b) Evidential Model
1-Shot
36.243-4.115
0.34982a.011
47 21.
0. 0.463400014
2-Shot
58. 58.258 s8a
0.516 o32
67 277
0 0.65420.006
5-Shol
75. 20.643
0.509 one
79.875=n.051
0. 0.670=0.o0
10-Shot
80. 80.158#0.284 0.45440..0 001
82. 82.67440.014
0.731 +0.00
(c) Base-rate adjusted Evidential Model
I-Shot
36. 36.24314.113
47.391 0.061++0.011
0.081 0.08110.000 1.0000
2-Shot
58.258 2..288...884
67. 67.523 0.077
0.070
5-Shot
75.643 098
0.069 002
79.87520.001
0.057220000
10-Shot
80. 15. 284
0.063 001
82.674.0.04s
0.052.0.001
(d) B-PEFT Model (Ours)
1-Shot
0.050 10.002
48.
10.225 18.722.0.225
0.076_a.008
2-Shot 62.796
0.065
69.187 10.15 0.068_0.002
S-Shol
77.181
0.062 20.001
79.918
20.010 0.051_a.001
10-Shot 80.788+0.064 0.059
82.748+4.o01
0.049-.0.00
Comparison with existing calibration tech- niques. We compare our proposed eviden- tial calibra tion technique with existing calibra- tion techniques using representative Cifar 100 dataset on different few-shot classification set- tings. We consider Isotonic Regression (IR-
Table 3: ECE performance comparison
Model
I Shot
Shot
Shot
10 Shot
CE Model [32]
0.393
0.494
0.517
0.501
Evidential Model [56]
0.499
0.620
0.744
0.765
TS [26]
0.092
0.074
0.043
0.036
[C) Temperature Scaling (TS) [26]. and Parameterized Temperature Scaling (PTS) [61] as baselines. Overall results are presented in Table 3. The baseline model and its evidential extension are poorly calibrated. All calibration techniques transform the logits to address the under-confidence issue of the model. Our proposed Base-rate adjusted Evidential model (BR-Evid addresses the model leading to significant improvement in the ECE. B-PEFT model further improves on the BR- Evid model it effectively addresses both the under-confidence issue (via the base rate adjustment) and overconfidence issue (via the Bayesian ensembling) that leads to superior calibration results as indicated by the lowest ECE in all few-shot settings.
PTS [61]
0.145
0.129
0.096
0.083
IR-MC [6]
0.091 0.104
0.103
0.085
BR-Evid (Ours)
0.077
0.080
0, 044
0.034
B-PEFT (Ours)
0.067 0.056
0.036
0.031
issue in the evidential
Uncertainty quantification results. We investigate the uncertainty quantification performance of our proposed calibrated evidential ensemble model. The developed model can reflect the model uncer- tainty (i.e., the model's confidence in its predictions) through the ensemble agreemen//issgreemmen and the distributional uncertainty through the sharpness of evidential prior distribution (i.e., the vacuity). For the analysis, we use C0 as in-distribution (ID) dataset for different shots, and Cifar100 as OOD dataset. As shown in Figure 6, we plot vacuity and variance distribution for I and 5 shots. A single model outputs the vacuity that can be used detect OOD samples. We can see ID samples are in the lower vacuity region and OOD samples are in the higher vacuity region. As the number of shots increases, the region becomes more separate. However, for I shot, there are some OOD samples in the lower vacuity region as well. Since we can not trust vacuity alone for uncertainty, we utilize the variance of ensemble components to quantify model uncertainty. As we see in Figure 6(c), the variance of OOD samples mostly lies in the high variance region. As we increase the number of shots, the variance shifts towards the lower region. High variance indicates that model-predicter vacuity can not be totally trusted. This behavior is qualitatively shown in Figure
9

---

1. in Ocmtbutiion www.n CTF10
3.hhet Destuuionooori
1h In Obunbution ClbrLo
shott In-iitrrttttton
8000
68000
Namber San
4000
OCB: Phr200
6000
7000
000: Chr/00
E3000
Char130
67700
68000
500 c00
n-bentarion reomect
2000
5500
2000
1.00
4000
2000
2000
2000-
10000
1000
8
1000
30:00
0.00
0.200
0,800
0.000
0.000
0.800
1,000
0.000
0.000
0010
eat
6023
0.030
0.035
0.000
0.010
0dp0
0,800
0.040
Hicutty
Hellnne
Marience
(a) Vacuity for 1 shot
(b) Vacuity for 5 shot
(c) Variance for 1 shot
(d) Variance for 5 shot
Figure 6: (a-b): Vacuity distribution of a single model and (c-d): variance distribution of ensemble models for 1/5 shots c0 In-Distribution and Cifar100 as Out-of-Distribuun dataset
Component vacuity: 0.379
Component vacuity: 0.308
Component vaculty: 0.857
Component 2 vacuity: 0.511
Component vacuity: 0.780
Component vaculty: 0.882
Component vacuity: 0.587
Component 3 vaculty: 0.601
Component vaculty:
Ensemble Variance: 0.015
Ensemble Variance: 0.018
Ensemble Variance: 0,002
(a) High Variance
(b) High Variance
Low Variance
Figure 7: Qualitative analysis of OOD samples for 1-shot adaptation of Cifar as ID dataset and Cifar100 as OOD dataset
7. We observe that when only single component outputs low vacuity for the OOD samples in Figure 7 (a-b), the variance of the ensemble is high, implying a high model uncertainty. When all the components output a high vacuity to the OOD sample in Figure (c), the variance is also low, implying low model uncertainty.
shot Cifar100
Ablation study. We carry out ablation with 1-shot Ci
0.45
dataset to study the impact of the base rate transformation order
0.40
kl=0.1
m (Section 3.2) for different strengths of incorrect evidence reg- ularization the calibration performance. As we increase m, the probability gap between classes improves, leading to more confident predictions. However, the model starts to become overconfident for large m values (see Figure Morevvr,,,, with an increase in incorrect evidence regularization strength, the optimal m value decreases to a smaller value (e.g., optimal m 2.0 for 0.1, and optimal m 1.5 for À 10). Choosing a proper m leads to the best-calibrated evidential model. We present additional results studying the impact of in Appendix F.I.
0.35
kl-1.0
0.30
kd-10.0
kl-100.0
0.25
kl=1000.0
0.20
0.15
0.10
0.5
1.0
1.5
2.0
2.5
3.0
Figure 8: Impact of m
Limited by space, we provide results comparing meta-learning methods on standard few-shot tasks in Appendix F.4, discuss impact of various components (number of classes, data size, and unfrozen parameters) in Appendix F.5, and include additional experiments and comparisons, including OOD settings, in Appendix F.6. Moreover, we carry out additional ablation experiments study the impact of incorrect evidence regularization strength (Appendix F.2), and the impact of ensemble components (Appendix F.3). We further carry out experiments and discuss applying PEFT to foundation models pre-trained in a self-supervised fashion (Appendix G). We discuss the societal impact and limitations of the work in Appendices H and I, respectively.
Conclusion
In this work, we focus on transformer- baaserdme--baedd large vision foundation models and investigate different parameter-efficien fine-tuning techniques for effective few-shot adaptation. We observe that the existing models are severely under-confident, especially in challenging datasets and settings. More- over, existing models lack fine-grained uncertainty quantification capabilities. We extend the models to uncertainty-awar evidential models, and resort the evidential framework to develop a novel Bayesian parameter efficient fine-tuning (B-PEFT) framework that integrates evidence- wridence-based base rate adjustment addresses the under-confidence and a diversity inducing evidential ensemble technique to further improve the reliability in model prediction and uncertainty quantification. The B- PEFFT framework possesses theoretically sound properties to ensure its superior generalization capability and robust calibration behavior. We carry out intensive experiments across different benchmark datasets and diverse few-shot settings that demonstrate the outstanding performance of B-PEFT.
10

---

Acknowledgments
This research was supported in part by an NSF IIS award IIS-18 IIS-1814450. The views and conclusions contained in this paper those of the authors and should not be interpreted representing any funding agency. We would like to thank the anonymous reviewers for their constructive comments.
References
[1] Krizhevsky Alex. Learning multiple layers of features from tiny images. https://www. cs. toronto. ectukri//larriingeeeuuree features-200-----T- pdf. 2009.
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and eeff-dssillatiin deep learning. arXiv preprint arXiv:2012.09816, 2020.
[3] Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential
regression. Advances in Neural Information Processing Systems, 33:14927-14937, 2020.
[4] Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.066470 2020.
[5] Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13349- 13358, 2021.
[6] Eugene Berta, Francis Bach, and Michael Jordan. Classifier calibration with roc-regularized isotonic regression. In International Conference on Artificial Intelligence and Statistics, pages 1972-1980. PMLR, 2024.
[7] Charles Blundell, Julien Corneiise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pages 1613-1622, PMLR, 2015.
[8] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference Computer Vision, 2014.
[9] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 33:1 1285- 11297, 2020.
[10] Bertrand Charpentier, Oliver Borchert, Daniel Zügner, Simon Geisler, and Stephan Günne- mann. Natural posterior network: Deep bayesian predictive uncertainty for exponential family distributions. In International Conference on Learning Representations, 2022.
[11] Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Posterior network: Uncer- tainty estimation without ood samples via density- pseudo-counts. Advances in Neural Information Processing Systems,
2020.
[12] Francesco D' Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems, 34:3451-3465 2021.
[13] Francesco D' Angelo, Vincent Fortuin, and Florian Wenzel. On stein variational neural network ensembles. arXiv preprint arXiv:2106. 10760, 2021.
[14] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen. Mathias Bauer, and Philipp Hennig. Laplace redux-efforrteess bayesian deep learning. Advances in Neural Information Processing Systems, 34:20089-20103, 2021.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 2010..190...99 2020.

---

[16] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volaan Volume 70, pages 1136-1135. JMLR. org, 2017.
[17] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems, pages 9516-9521 2018.
[18] Gianni Franchi, Olivier Laurent, Maxence Leguéry, Andrei Bursuc, Andrea Pilzer, and Angela Yao. Make me a bnn: simple strategy for estimating bayesian uncertainty from pre-trained models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pages 2219-112204 2024.
[19] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
[20] Mudasir A Ganaie, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthu- rai N Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:10515,, 2022.
[21] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang. Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581-59) 2024.
[22] Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4367-4375, 2018.
[23] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta-learning probabilistic inference for prediction. arXiv preprint arxiv: 1805.0992 2018.
[24] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. International Conference on Learning Representations, 2018.
[25] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang, Ppt: Pre-trained prompt tuning for few-shot learning, arXiv preprint arXiv:2109.0453 2021.
[26] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
[27] Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. E' 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint arXiv:2307.1377 13770, 2023.
(28) Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang, Siyuan Qi, and Dongfang Liu. Facing the elephant in the room: Visual prompt tuning or full finetuning? arXiv preprins arXiv:2401.1290 12902, 2024.
[29] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[30] Gao Huang. Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get m for free. arXiv preprint 1rXiv:1704.00110 2017.
[31] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[32] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
12

---

[33] Audun Jøsang, Subjective logic, volume 3. Springer, 2016.
[34]
Audun Josang, Jin-Hee Cho, and Feng Chen. Uncertainty characteristics of subjective opinions. In 2018 21st International Conference on Information Fusion (FUSION), pages 1998-2005. IEEE, 2018.
[35] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- Tusee, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, (88())1956-198) 2020.
[36] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[37] Hae Beom Lee, Hayeon Lee, Donghyun Na, Saehoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-of- distribution tasks. In Eighth International Conference on Learning Representations, ICLR 2020. ICLR, 2020.
[38] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. arXiv preprint arXiv:2104.0800 08691, 2021.
[39] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[41] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 2018.
[42] Aryan Mobiny, Pengyu Yuan, Supratik K Moulik, Naveen Garg, Carol C Wu, and Hien Van Nguyen. Dropconnect effective in modeling uncertainty of bayesian deep networks. Scientific reports, 11(1):5488, 2021.
[43] Azadeh Sadat Mozafari, Hugo Siqueira Gomes, Wilson Leão, Steeven Janny, and Christian Gagné. Attended temperature scaling: a practical approach for calibrating deep neural networks. arXiv preprini arXiv: 1586, 2018.
[44] Rafael Müller, Simon Korblith, and Geoffrey Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
[45] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008
[46] Deep Shankar Pandey and Qi Yu. Multidimensional belief quantification for label-efficient meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14391-14400 June 2022.
[47] Deep Shankar Pandey and Qi Yu. Evidential conditional neural processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages
2023.
[48] Deep Shankar Pandey and Qi Yu. Learn to accumulate evidence from all training samples: Theory and practice. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26999926999.... PMLR, 23 29 Jul 2023.
[49] Tim Pearce, Felix Leibfried, and Alexandra Brintrup. Uncertainty in neural networks: Approxi- mately bayesian ensembling. International conference on artificial intelligence and statistics, pages 234-244, PMLR, 2020.
13

---

[50] Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Reg- ularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv: 170100648,, 2017.
[51] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.0777
2020.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[53] Rahul Rahaman et al. Uncertainty quantification and deep ensembles. Advances in Neural Information Processing Systems, 34:20063-20075, 2021.
[54] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
[55] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manc imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104. 100222200..0972, 2021.
[56] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems, 31, 2018.
[57] Weishi Shi, Xujiang Zhao, Feng Chen, and Qi Yu. Multifaceted uncertainty estimation for label-efficient deep learning. Advances in neural information processing systems, 33, 2020.
[58] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In
I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[59] Zhao Song, Ke Yang, Naiyang Guan, Junjie Zhu, Peng Qiao, and Qingyong Hu. Vppt: Visual pre-trained prompt tuning framework for few-shot image classification. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[60] Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances Neural Information Processing Systems, 32, 2019.
[61] Christian Tomani, Daniel Cremers, and Florian Buettner. Parameterized temperature scaling for boosting the expressive power in post-hoc uncertainty calibration. In In European Conference on Computer Vision (ECCV), 2022.
[62] Dennis Ulmer. A survey on evidential deep learning for single-pass uncertainty estimation. arXiv preprint arXiv:2110.03051, 2021.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[64] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 29, 2016.
[65] Cheng Wang. Calibration in deep learning: A survey of the state-of-the-art. arXiv preprint arXiv:2308.0 01222, 2023.
[66] Xiao Wang. Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao- Yong Wei, Yaowei Wang. Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research, 407442447-482 2023.
14

---

[67] Jingyuan Wen, Yutian Luo, Nanyi Fei, Guoxing Yang, Zhiwu Lu, Hao Jiang, Jie Jiang, and Zhao Cao. Visual prompt tuning for few-shot text classification. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5560-5570, 2022.
[68] Liqi Yan, Cheng Han, Zenglin Xu, Dongfang Liu, and Qifan Wang. Prompt learns prompt: ex- ploring knowledge-awar aware generative prompt collaboration for video captioning. In Proceedings of international joint conference artificial intelligence (IJCAI), pages 1622-1630, 2023.
[69] Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems, pages 7332-7342, 2018.
[70] Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen, Prashanth G Shivakumar, Yile Gu, Sungho Ryu Roger Ren, Qi Luo, Aditya Gourav, I-Fan Chen, Yi-Chieh Liu, et al. Low-rank adaptation of large language model rescoring for parameter-efficient speech recognition. 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1-8. IEEE, 2023.
[71] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efffcient fine-tuning for transformer-based masked language-models. arXiv preprint vrXiv:2106.10199 2021.
[72] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 698-714. Springer, 2020.
[73] Jize Zhang. Bhavya Kailkhura, and T Yong-Jin Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference on machine learning, pages 1 11117-11128 PMLR, 2020.
[74] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. In International Conference on Machine Learning, pages 26135-26166 PMLR, 2022.
[75] Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems, 33:12827-12836. 2020.
[76] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-lange Iasion-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816-16825, 2022.
[77] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009)23337-2348, 2022.
15

---

Supplementary Material
Appendix
Table of Contents
Organization of the Appendix
B Summary of the Symbols
17
C Further Discussion Uncertainty- Deep Learning
17
C.I Existing Uncertainty Quantification Methods in Deep Learning
17
C.2 Evidential Deep Learning Models for Classification
18
C,3 Evidential models vs. Standard Bayesian Models
19
Proofs Theoretical Results
D.1 Proof of Lemma
19
D.2 Proof of Theorem
19
D.3 Connection with SVGD-based Bayesian Ensembling and Proof of Lemma
20
E Dataset and Implementation Details
21
Additional Experiments
F.1
Impact of on Expected Calibration Error
22
F2
Impact of Incorrect Evidence Regularization Strength (A)
................................
25
F3
Impact of Ensemble Components
26
F4
Few Shot Learning Results
26
F55
Impact of Different Components
26
F6
Additional Experiments and Comparison
27
G Calibration Behavior of Self-Supervised Model
28
H Societal Impact
28
Limitations and Future Work
29
16

---

A Organization of the Appendix
In Section B, present the table with summary of the key symbols used in this work.
In Section C, present related works in uncertainty-awar deep learning, describe eviden- tial deep learning for classification in details, and discuss some key advantages of using evidential models to address the under-confidence issue over standard Bayesian models.
In Section D, we present proof of all our theoretical claims.
In Section E, we present the details of hyperparameters and additional experiment details.
In Section F, we present additional experiment results including the impact of m for calibra- tion performance, comparison results with meta-learning methods, the impact of the number classes, data size, and unfrozen parameters, comparison in OOD settings, the impact of incorrect evidence regularization strength, the results of additional PEFT methods, and the impact of ensemble components.
In Section G, discuss the calibration and accuracy behavior for PEFT of large foundation models pre-trained in a self-superviser fashion.
In Section H, we discuss societal impact of our work.
In Section I, we discuss the limitations of our work and present potential future direction.
The source code for the experiments carried out in this work is attached in the supplementary materials and is available at the link: http:///gghabb.ooo/iiinnnggggg PPEFT
B Summary of the Symbols
Table 4: Summary of the symbols and their definitions
Symbol
Definition
Input sample vector
y
Ground truth label as one hot vector
e,
Cx
The evidence vector, and the evidence for class i
Fixed base rate for class usually set a; N
04
Dirichlet parameter value for class
-i=1 0i
The Dirichlet Strength
(b,
Ucory Uina)
The belief vector, Correct belief, and the Incorrect belief
N
Number of classes
Belief class
2
Vacuity output by the model
dis
Dissonance output by the model
Non-negative evidential transformation function (we use exp)
X²
Leamable base rate for class i
Learaalle base rate vector
Incorrect evidence regularization strength
Flement wise multiplication hetween two vectors
W
Non-informative prior weight
C Further Discussion Uncertainty-Aware Deep Learning
C.1
Existing Uncertainty Quantification Methods in Deep Learning
Accurate quantification of predictive uncertainty is essential for the development of trustworthy Deep Learning (DL) models. To this end, DL models have been augmented to become uncertainty- aware using a variety of approaches such as ensemble-based approaches [36, 49], bayesian neural networks based approaches [42, 19, 7], and deterministic neural network based approaches [56, 11, 3]. Deep ensemble techniques [36, 49] construct an ensemble of neural networks, and the agrremmen/ddisageeemmntt across the ensemble components is used to quantify different uncertainties. Alternatively, Bayesian neural networks [19][]][42] have been developed that consider a Bayesian formalism (e.g.,
[7], dropout during test [19]) to quantify different uncertainties.
17

---

ABNN [18] introduces Bayesian normalization layers after training of deep learning models, and requires additional training of these layers in a post-hoc manner. Deterministic neural network-based approaches [57, 41, 10] extend the existing neural network to become uncertainty- the networks to quantify fine-grained uncertainties with a single forward pass through the network. Evidential deep learning models [56, 5, 75, 10, 10, 62], an instance of deterministic approaches, introduce a conjugate higher-order evidential prior for the likelihood distribution to enable the model to express the fine-grained uncertainties in both classification 11] and regression problems [3, 47J. Towards classification, evidential models [56, S, 75] introduce higher-order evidential Dirichlet prior to the multinomial likelihood that enables the deterministic neural network model to capture different uncertainty characteristics. In what follows, we first provide some additional details on using evidential learning model to perform classification. We then highlight some important advantage of using evidential models over standard Bayesian models in uncertainty quantification.
and enable
C.2 Evidential Deep Learning Models for Classification
Evidential Deep Learning models, based on Subjective Logic theory [33], aim to train the model such that for any new input sample, the model can make predictions, as well as output fine- ine-grained uncertainty information (via vacuity [56] and dissonance [34]). Towards capturing fine- fine-grained uncertainty for classification problems, EDL models assume that the label for each sample is obtained from generative process with Dirichlet prior and a multinomial likelihood. The parameters for the Dirichlet prior express the vacuity and belief masses for uncertainty estimation. The conjugacy between the Dirichlet prior and the multinomial likelihood is explored, and different evidential losses are introduced for model training and inference [48]. In this work, we consider Type-Il Maximum Likelihood-bal based evidential loss CLoB (x, y) [56] with incorrect evidence regularization Crog (z. y) given by [56]
Covia (x, y) = C y) + X x Ciag( y)
(7)
We replace the softmax layer in the head of the VPT model with exp activation function. To avoid zero evidence regions, we also include the correct evidence regularization (eo y) ->cor log( -)) ( Aooor is the magnitude of the model's vacuity) in model training objective. The evidential model outputs evidence vector (c1, E2, ...e) ) for given input æ and corresponding ground truth label of y. Based on the evidence, Dirichlet parameters are obtained as a, = 6 + 1. The Type-II Maximum likelihood-beg based evidential loss is given by
K
K
CLo8 (2., y) - In Mult(yy)) = log S
Σ
yk log on
(8)
k=1
The incorrect evidence regularization guides the model to minimize the evidence for all classes other than the ground truth class and can take one of the following forms
1. KL-based incorrect evidence regularization term in EDL [56] Cro (x.y) = (DDir(pl@)|Dir((pp]]
K rΣK k= 6%
K
K >
(9)
log
+ >>aay
v Cās 1) (((((()))))
(
Г(K) II k= rā
Where ā = + (1 3) 3 a = (ā) (āj,āā,.... aā.)) parameterize a dirichlet distribution, Öi=gt 1, a. ViVi + gt, and C represents element-wise product. Here, the KL regularization term encourages the Dirichlet distribution based on the incorrect evidence i.e., Dir Dir(p)ā) to be flat which is possible when there is no incorrect evidence.
2. Incorrect evidence sum based regularization as in ADL [57]
K
K
(a, y)
Σ
Σ
ck x (1 yk)
(10)
1=1
k=1
3. Incorrect belief-sum based regularization as in Units-ML [46]
K
CUnite (x. y) =
(11
Σ
ek
x (1 yk)
(11)
s
1=1
k=
18

---

All three regularizations guide the model to minimize the incorrect evidence(ideally close to zero). In our experiments, we consider KL-based incorrect evidence regularization.
C.3 Evidential models vs. Standard Bayesian Models
As compared with the Bayesian-inspired models, evidential learning offers two key properties that allow us to formulate principled solution to address the unique under-confident behavior of the PEFT methods. First, thanks to its evidence-based fine-grained uncertainty decomposition capability, we can separate two distinct sources of second-order uncertainty, including vacuity and dissonance. Different from the commonly used first-order uncertainty (e.g., entropy), these two second-orde uncertainty serve a key tool to understand why PEFT methods are both accurate (with a low dissonance) while being under-confident (with high vacuity). This key insight suggests that these methods systematically under-estimate the contribution from the prior knowledge to the downstream task. While the classical Bayes' theorem offers a principal idea to address the issue, which is to strengthen the prior belief, there is lack of practical way to achieve this. As the second key property, evidential learning allows us to leverage the base rate, which is rooted in the subjective logic theory as an effective vehicle to adjust the prior belief gained through pre-training. To this end, we propose transformation function in Eq. (6) to adjust the base rate that leads to the increase of the model confidence while maintaining the predictive accuracy of the model as guaranteed by our theoretical results in Lemma 2 and Theorem 3. Furthermore,
develop belief-baset diversity for ensemble of evidential models leading to the the B-PEFT model. In theory, evidential deep learning model could be augmented with the Bayesian normalization layers [18] or Bayesian neural networks [7] as an alternative to belief-based diversity of B-PEFT. We leave exploration of different techniques for diversity for Bayesian evidential model as a potential future work.
D Proofs of Theoretical Results
In this section, we provide the proof' of the major theoretical results presented in the main paper.
D.1 Proof of Lemma
Proof. Consider an input sample for which the model outputs the evidence (c1, E2,
c2. .... eN) T Let
Cnax max c,, ex), and Cmin = min(e1, C2, .... Cx). Here, 6nax 2 6; 2 emin Vi E [1, N].
For the evidential model with fixed base rate of 0; vi [1, N.. the model's predicted class is given by (pred = arg max(e1 + 01 x W.ez + 02 x W...,e + aN x W) = arg max(e1 1 + 1, C2 1,...,e, EN + 1) (ndeex(max). ). For the calibrated model with learnable X = (x), X2, ... XN the model's predicted class is given by caey arg max 022, "e.,......., aN) = arg max(e1 XI W, e2
X2 W, ...., eN XN XW) arg max(e1 + ((())) N2 Cetn - N, ..., eN + N EM
N).
Cata
Since 6nax 2 e, 2 Emin E [1, NJ. O'max 2 a 2 Omnn Vi E [1, N). and eex
ppred = Cprad
D.2 Proof of Theorem
Proof. Consider an input sample x for which the model outputs the evidence (e1, E2, ... eN) Let Gpax max(e1, NN,)., Euin min(e1, E2; ...EN) and Emax 2 Ezua 2.... Emin- For the evidential model with fixed base rate of = N parameters for class with maximum evidence and class with the second maximum evidence given
1 Vi E [1, NJ. the difference between the Dirichlet
by Omaz andd
Gzaz + Gpax W E2dd Q2aa / = Graz E2ad as a;
N Vi E [1.
For the calibrated model with learnable x (x1, X2; XN)), the difference between the Dirichlet parameters for class with maximum evidence and class with the second maximum evidence is given
by Omax
0'2dd Csax
Xmax W
62nd
X2na W (enax 62nd)
(Xmax X2nd Now,
Xaax = (Crax
Cmin
m
Cnax
-
&
X2nd
(22nd
Cnin
G2nd
1)
(12)
Epin
Emin
emin
Emin
Or, (Xnox X2nd)
Cnax
m
62nd
-1
(13)
Cain
Gmin
Since > Ivi E [I. and Emax
Eznd 2, ..., Ensn, (Xeax
X2na)
Ovn > 0. For
Cuin
Cuax EZad,
&m > 0, Xmax X2od 0. Thus, with the proposed learnable base rate, the gap between
19

---

the two largest Dirichlet parameters is maintained whenever m = 0 and/or Caa = 2244. Moreover, whenever m and Gnax > C2nd, the Dirichlet parameter gap between the two classes is increased m
by a factor of
fus
1)
Siad
1)
m
Fein
Cein
D.3
Connection with SVGD-based Bayesian Ensembling and Proof of Lemma
We first carry out an analysis of the update in Stein Variational Gradient Descent (SVGD) based ensembling [12, 13] that reveals the repulsive force acting among the ensemble components that pushes the particles away and introduces diversity. We then consider ensemble components with different strengths of incorrect evidence regularization Line and analyze the update to the evidential model in the evidence space that reveals a repulsive diversity-enforeing force acting identical to the SVGD based ensemble.
SVGD update involves randomly initializing the particles and iteratively updating the particles to match the target distribution, which is summarized below.
Algorithm 1 SVGD Update
Input: [ A set of initial parameters, and target distribution density function p(z) For L iterations, iteratively update the particles
o x + E
where o (x)) - ( Σ k(x'. x:) V: log + VV kork
Here, is the step size at iteration I, and :): is the kernel function that measures similarity. set of initial parameters, and target distribution density function p(x)
Output: {a] |==1 N
For given incorrect evidence regularization Creg. and P ensemble components with regularization strengths Xp. E [1, Pl. the ensemble components in the evidence space are implicitly pushed away from each other by force >p V Line that acts identical to the repulsive force in Stein Variational Gradient Descent (SVGD) based ensembles.
Proof. For simplicity, consider RBF kemel for k(;;-) i.e., k(a, b) exp (=) ())),, two particles di, 32, and analyze their updates in SVGD based ensembling. At iteration 1, the update to particle 22 given by
1 (aaah logp(z²) + k( V logg(a)) + V h(1 wn)) + k(ah
G (x-))
In the above update, Q k(al) m1 V log)(a)) + k(a m5 V log p(g)( aims guide the particles in the direction that maximizes the likelihood, and the update direction F V xb)), acts the repulsive force. Considering the repulsive force V (01 2h))
π k(
Vay exp ( (of xb))) V exp
( (1f zbb))) h
2 h (1d2 2 (1)
As can be seen, the repulsive force R pushes the particle r/ in the direction away from particle 2| that introduces diversity. With more particles, each particle updated the direction that maximizes the likelihood, and the particle is pushed away from all other particles (by force Rt).
Next, consider ensemble components with different strengths of incorrect evidence regularization Ling to analyze the update to the evidential model in the evidence space. For simplicity, we consider the incorrect evidence sum-based regularization similar to ADL without correct evidence regularization (The analysis is valid for all incorrect evidence regularization and for models with correct evidence regularization). For model with incorrect evidence regularization, the overall evidential loss given by:
K
K
Lovia( y) = (xx y) + A X CADL
(2c, (e, y) log S
yk log Or + A x
Ek X (1 4k)
20

---

The gradient of the loss with respect to logits (the output head layer, where exp(ox)) is given by
(x,
a dryy (x,
dos = (s
yk
Sek
de: (14)
grad
+
+ A (1 yk) x
dok
akk
dok
dok
yk
+ A(1 wk)) EK =
(s
1
yk
+ A(1 - y11) )
(15)
S
Or
Consider K class classification problem. The gradient update to the logit layer for the evidential model is given by
ML
/11
V/2
1 - 1/2
grad Ek X
8
+ X
Ax
(16)
ax
A
x x B
(17)
Here. 1/s E 10, yk if = gt, and y/s = 0 otherwise. Moreover, the À value is varied, and different À values lead to different evidential models.
The update force À pushes the evidential model in the direction that maximizes the likelihood (similar to Q in SVGD-based update), and the force B implicitly pushes the ensemble components
away from each other (similar to the repulsive force R in SVGD-based update). Each component moves in B direction with a different force determined by the incorrect evidence regularization strength A that ensures that the ensemble components are diverse. Due to the different strengths of incorrect evidence regularization, each ensemble component places different priority level for minimization of incorrect evidence over acquiring correct evidence, which ensures that the ensemble components remain diverse.
E Dataset and Implementation Details
We consider ViT model backbone [15] that is pre-trained in a supervised fashion, and 4 benchmark datasets of Cifar10 [i], Cif [1]. Food101 [8], and Flowers102 [45]. We consider few-shot adaptation with K-shot classification problem (We experiment with K values of 1, 2, 5, 10, and 20). The few-shot training set constructed by randomly selecting K samples per class from the training set of the benchmark datasets. We consider 2-shot validation set for all datasets and settings. We train the model on the few-shot training set, use the 2-shot validation set for hyperparameter tuning, and evaluate all models on the benchmark test set with all the test set samples. We augment the few-shot training set and the few-shot validation set with resize, random horizontal flip, and cropping. The dataset details are also presented in Table 5. We train all the models for 50 epochs on the few-shot training dataset with a batch size of 64 samples at each iteration and evaluate the model on the benchmark test set. The evidence is in the range [0, infinity], and some stability issues could potentially arise in extreme cases when the logit output is extremely low (i.e. close to negative infinity). In our experiments, we did not observe the stability issue. Still, the issue can arise in some extreme cases for which a small delta in the denominator could be introduced or the network's 's logits could be bounded to be greater than small negative value. For the calibration baseline model of Parameterized Temperature Scaling (PTS) [61], we consider a 2-layer neural network with 128 nodes in the hidden layer (we carry out hyperparameter tuning with I-layer. 2-layer, and 3-layer networks and select the best model), and train with a learning rate of 0.00001. 1. For Temperature Scaling [26], we optimize for the temperature hyperparameter using Adam optimizer, and a learning rate of 0.01 (We also experiment with SGD optimizer, and learning rates of 0.1, 1.0, 0.01, and 0.0001 to select the best performing model). The evidential models use incorrect evidence regularization strength of (0, 0.1, 1.0, 10.0, 100.0, and 1000.0). For Isotonic Regression [6], we consider multi-class setting and sklearn package. We use VPT [32] as the representative PEFT where not specified due to its superior performance. The key model performance results are averaged across 5 different runs to present the mean and the standard deviation. The experiments use Pytorch and are carried out on workstation with NVIDIA RTX A6000 GPU.
21

---

Dataset Name
Number of Classes
Training Sumples
Validation Samples
Test Samples
Cifarl0 [1]
10 x K
20
10, 000
Cifar [1]
100
100 x
200
10, 000
Food101 [8]
101
101 X K
202
25, 250
Flowers102 [45]
102
102 x K
204
6, 149
Table 5: Dataset Details for K-Shot Classification problem
Acturacy 48.155, ECE: 0.272
10
49.155 ECE, 0.103
10
Accuraty: 49.155, ECE: 0.0:0.070
9.6
0.6
0.6
Confidence
Confidence
Confidence
(a) 1 Shot, 0.10, m 1.0 (b) Shot, =L= 0.1, m= 1.5 (c) Shot, =L= 0.1, I. m 2.0
1.0 Acturacy 50.905, ECE 0.214
1.0
Accuracy 50.905 ECE: 0.083
10
Accuragy 50.305, ECE: 0.124
9.6
06
0.6
02
02
0.2
Confidence
Confidence
Confidence
(d) Shot, KL = 100.0, m = (e)
Shot, KL 100.0, m = (0) Shot, KL 100.0, m
1.0
1.5
2.0
Figure 9: Visualization of the impact of m for 1-Shot Cifar 100 dataset using reliability plots
F
Additional Experiments
F1
Impact of m on Expected Calibration Error
The developed B-PEFT model introduces the post-hoc calibration technique that adjusts the base rate in the evidential model with one additional hyperparameter m (see Section 3.2 for details). We carry a grid search using the few-shot validation dataset and select the optimal m for the evidential models. In this section, we carry out detailed ablation to study the impact of m with few-shot Cifar100 datasets. We consider m values of (0.5, 1.0, 1.5, 2.0, 2.5, 3.0), incorrect evidence regularization strengths of (0.0, 0.1, 1.0, 10.0, 100.0, 1000.0). and few-shot values of 2,,2,5, 10, 20) (see Table 6 and Figure 9, Figure 10). Across all the experiments, we observe that as we increase m, the model transforms the evidence to maximize the evidence gap making the model more confident on its knowledge. With increased confidence, the model's 's calibration performance improves. However, large increase in values starts to make the model overconfident leading to increased ECE and poor calibration. The strength of incorrect evidence regularization also impacts the optimal value of m. For large values of incorrect evidence regularization, a smaller value suffices to make the model wall-calibrated. The trend is seen across few-shot classification settings.
22

---

Table 6: Impact of m
Reg. Strength (A) (Eqn
0.5
1.0
= 1.5
2.0
2.5
3.0
Shot
0.0
0.401 =0.007
+0.099
0.094
0.0044
00.477
0.21 40.000
0.423
0.2614
0.092
0.079=0.003
0.156
0.22
1.0
0.252 0.444420.00
0.082+0.000
0.169 20.000
0.23g0.007
0.429
0..01
0.077
0.186 =0.000
0.245+0.000
100.0
0.212 0.433_0.000
0.08
0.199 0.125
0os
0.253 ooa
1000.0
0.422=0.006
0.182+a.003
0.077+0.003
0.148=0.000
0.221 +0.002
0.271 +0.003
Shot
0.033
0.33440.000
0.120
0.041
0.100
0.154g0.00s
0.35920.006
0.32840.010
0.103
0.060=0.005
0.113g0.00 006
0.160
LO
0.065
0.30810.000 10.000
0.08610.004
0.07220.003
0.125g0.000 0003
10.000
10.0
0.560_0.005
0.28210.00 005
0.074
0.081
0.136 005
0.177:10.000 004
0.2644..000 0.550_o.002
0.066
0.1-46g0.00 20.00
0.1864...00z
1000.0
0.047
1.25710.001
1.06410.001
0.090
20.000000
0. 0.186100.00
Shot
= 0.0
0.507=0.010
0, 0.125 0.35440.017
0.031=0.004
0.073
0.117
= 0.1
0.639
0.349
0. 20.00
0.04220.004
0.084
0.11920.003
= 1.0
0.634_o.002
0.340
0.11 #0.002
0.047
01.085g0.003
0.122++0.00
10.0
0.362 0.645_0.000
03138 003
0.0711 0.04912.004
0.107
100.0
0.652_00000
0.35742.001
0. 0.11920.001
0.04222.002
0.075 003
0. 0.10940..00
1000.0
0.642=0.006
0.31440.000
0.081
0.051=0.002
0.091
0.122++0005
10 Shot
= 0.0
0.325 0.604
0.09010.004
0.041
010955.0.00 000
0.13110.007
= 0.1
0.31440.000 0.62910.009
0.077 #0.005
0.100 0.0544.0.004
0.13-440.00s
= 1.0
0.657
0.367 1.0.000000000
0. 0.13610.000 00.002
0.053_0.000
01.668...00
0. 0.10221.00c
10.0
0.685_o.004
0.404
00.001
0.03-1
0.057=0.000
0.090
= 100.0
0.673
0.363
0.109
0.041 20.001
0.075
000
0. 0.
1000.0
0.651 + .0.004
0.335
0.099
0.049
(1082+...o)
0. 0.113+0.001
20 Shot
A = 0.0
0. 0.413=0.000
0.141
0.060
0.162
0.226
0.270+a.010
A = 0.1
0.409
0.099 005
0.090
0.18820.00
0.248
0.288
A LO
0.399
0.091
0.08640.002
0.183
0.244
0.284++.00
A = 10.0
0.387 002
0.075 804
0.106
0.202
0.259
0.297
100.0
0.401
0.097
0.093
0. 0.166
0.259
0.299
23

---

Acturacy: 71.497, ECE: 0.346
10
Accuracy: 71.97, ECE: 0.106
10
Accuracys 71,497, ECE: O. 041
0.8
0
0.6
0.2
02
0.2
06
04
DR
05
04 0.6
15
022
0.6
1.5
Confidence
Confidence
Confidence
(a) 5 Shot, KL. 0.1, m 1.0 (b) Shot, =L 0.1, m = 1.5 (c) Shot, =L 0.1, 2.0
1.0
Accuracy 74,177 ECE: 0.350.357
1.0
Accuracy. 74.347 ECE: 0.216
1.
Accuracy: 74.147, ECE: 0.041
Accuracy
9.6
Accuracy
6
0.6
00
0.4
DB
04
0.5
1.2
3.0
02
0.6
1.0
Confidence
Confidence
Confidence
(d) Shot, KL. 100.0, (e) 1.5
Shot, KL. 100.0, (f) Shot, KL 100.0, 2.0
1.0
Figure 10: Visualization of the impact of m for 5-Shot 10 dataset using reliability plots
10
ECE:
1.
1.0
1.2
0.0GB
Accurniyy
Accuracyy
Accuragyy
0.7
0.2
0.2
3.2
000
0.4
2.5
5.
24
05
03
0.5
0.5
08
0.0
0.
08
Confidence
Confidunce
Confidencce
Confidence
(a) Standard CE model
(b) Evidential Model
(c) Calibrated Evidential
(d) B-PETT (Ours)
Figure 11: Ccnfracy-Coffidence trends in 1-shot Cifar100 Results
24

---

F22 Impact of Incorrect Evidence Regularization Strength (X)
Evidential deep learning models introduce incorrect evidence regularization to minimize the evidence of classes other than the ground truth class. In this work, evidence regularization (see Section C.2), and introduce a hyperparameter À that controls the priority the model places on minimizing the incorrect class evidence over maximizing the correct class evidence. In this section, we study the impact of the hyperparameter À on the model performance with K-shot C0f and Flowers 102 experiments (We experiment with K values of (1, 5, 10, 10,20), 20)). We observe that the model's 's calibration performance (the ECE) is optimal when no incorrect evidence regularization is used i.e., X 0 (see Table 7 and Table 8). However, no incorrect evidence regularization hurts the model's generalization performance. With the increase in incorrect evidence regularization, the model's generalization performance (indicated by accuracy) improves albeit with ECE tradeoff. However, very large incorrect evidence regularization misguides the model to only focus on minimizing incorrect class evidence hurting generalization. The optimal À value leads to the best generalization performance while hurting the ECE performance. Moreover, with a larger number of shots in training. the optimal À value generally smaller (For instance, in 1-shot C0far100, optimal À = 1000.0, in 2-shot Cifar100, optimal X = 10.0, and in 5-shot Cifar 10,, optimal À = 0.1). Once the optimal À value is determined, we can resort to our evidential base-rate adjustment that leads to a calibrated evidential model with good generalization performance.
KL divergence-basee incorrect
Table 7: Different Shot Classification - Accuracy and ECE in Cifar100
Shots
KL
KL
KL
KL 10.0
KL 100.0
KL 1000.0
(a) Accuracy f
1
45. 40.30210.402
48. 18.75410.67 17.707
49.967
50.12710.962
51.127
2
60.49441.507 64.00640.834
20.06
65.545
65.439 295
35.531
5
11900 77.391 11.033
77.238.0.094
76.760
77.561 20.711
77.52520.085
10
80.56620.3mm 81.51240.165
81. 11.
81.561=00291
81.559
81.344-0.2
20
82.111 82.96710.2
83.01220.12
23.100..0.14.
83.01410.16s
81.96620.7
(b) ECE
1
0.40440.005
0. 0.440
0.460=0.007
0.479
0.48720.0m
0. 0.490
2
0.480
0.56220.007
0.596 005
0. 0.62010.000
20.003
0.637 6637=0.008 003
5
0.513
0. 0.632
0.686_a.00m
0. 0.7
0.7 0.704
0.756 1.756_0.00e
10
0.449
0.644200.004
0.71 0011
0. 00.022
0.787 #0.002
0.740
20
0.483+0.003
0. 0.647=0..00
0.733
0. 0.782
0. 20.04...0m
0.490=a.007
Table 8: Different Shot Classification Accuracy and ECE in Flowers
Shots
KL 0.0
KL. 0.1
KI. 1.0
KI. 10.0
KL 100.0
KI. 1000.0
(a) Accuracy 1
1 Shot
84.314g1m77
86, 4374
88.48120.92
19.225
89.475 .046
80.
2 Shot
91. 91.416+1 296
93.
94.575 49.795
5.544
94.800020.200
94
05.071+0.11
5 Shot
07.471+0.1m
97.
97.51=0.279
97.602_a.199
97. 97.130=o.185
96.95 96.953+0.19
07.235+0.248
10 Shot
98. 08.32240..0 zua
97.
97.9644a.198
97.804_..093
97.847 20.044
98. 00.30901..4
98 01144..111
Shot
98. 18.708
98.37 10.072
97.953_a.115
98.086_0.776
98. 08.40620.167
97. 27.7540
(b) ECE
1 Shot
0.66220.015
0.722
0.766 008
0.801
0.82620.008
0.846.20.00
2 Shot 0.60820.00
0.69620.003
0. 20.08
1.7994g0.00s
0.835ga.00m
0.87420.006
5 Shot
0.487
0.398
0.68640.02
20.005
0.83
0.89620.001
10 Shot
0.444=0.00s
0.57
0.641 20
0. 0.733
0.827=00018
0.908
20 Shot 0.411
0.54420.009
0. 0.634g0.012
0.75+0.046
0. 0.831=0.023
0.808
25

---

F.3 Impact of Ensemble Components
We present the experiment to study the impact of ensemble components in Figure 12. The ex- periment is performed on one-shot Cifar100 dataset using prompt-based adaption for vision transformer. There is performance gain of both accuracy and ECE with the use of all ensemble components. We also note that as we increase the number components from 1 to 3, both the gen- eralization and calibration performance increase significantly. For instance, the ECE with single ensemble component is 0.088 which improves 0.077 with 3 ensemble components while the ac- euracy improves by almost 3%.. However, with further increase in the number of ensemble compo- nents, the gemeraliizationcallibttooo performance improvement is not significant. Thus, for our B-PEFT model, we carry out an ensemble of 3 evi- dential models that is good balance between the number of ensemble components and the gain in generalizatiio//alibraaii performance.
Ensemble Components
1 components
0.086
2 components components
0.084
components 5 components
0.082
0.080
0.078
49
50
51
52
Accuracy
Impact of ensemble components as number of component increases
Figure 12:
F.4 Few Shot Learning Results
In this set of experiments, we apply our model to the 5-way 1-shot mini-ImageNet testing tasks and compare with representative meta-leaming models. The results are summarized in Table 9. Benefiting from the knowledge acquired during pre-training, the proposed PEFT- basee model outperforms the episodic meta-learning models by large margin, demonstrating the potential of large vision foundation models for effective few-shot learning. However, the VPT model is under- confident as shown Table and Figure 13. Our B-PEFT model improves on the VPT model's generalization and calibration leading to promising few-shot adaptation results.
Table 9: 5- 5-y 1-Shot Mini- Mini-ImageNet
0.8
Model
Accuracy
Accuracy
MAML [16]
48.70
Matching Networks [64]
43.56
0.2
LLAMA [24]
49.40
VERSA [23]
53.40
0.5
9-2
0.2
9.6
6.5
Confidence
PLATIPUS [17]
50.13
Bayesian- MAML [69]
Merssian-MMMLLL
53.30
Figure 13: VPT reliability plot for 5-Way Shot Mini-ImageNet
Coyesian-TAMi [37]
71.46
VPT
89.50
B-PEFT (Ours)
90.09
Accurscy
3.0
Table 10: Calibration Results
Model
Accuracy
ECE
VPT
89.50
0.418
5.0
0.6
0.6
B-PEFT (Ours)
90.09
0.065
Confidence
Figure 14: Full fine tuning reliability plot for 100-way 1-Shot Cifar100
F.5 Impact of Different Components
In this section, we study the under-confidence behavior of PEFT methods w.r.t. data size, number of classes, and number of unfrozen parameters.
26

---

Data size. We observe that the model's accuracy increases with more training samples (see Table I where we vary shots from to 20). The confidccne remains, even with further increase in training samples. To this end, we conduct additional experiments on Cifar 100 by increasing the training samples per class to 500 and observe the under-confidence issue despite the increase the accuracy. The trend is summarized in Figure 15 (a-b). We see an increase in accuracy and a decrease in ECE. However, even with 500 samples per class, the under-confidence issue remains. Further, we report the accuracy and ECE of the fully fine-tuned model (fine-tuning of all the parameters) for I shot c0 Table 11 where we observe decrease in accuracy while the calibration issue remains. We observe that full fine-tuning leads overconfidence behavior, hurting the generalization performance, as seen in Table 11 and reliability plot as presented in Figure 14.
Accuracy Thend
Trand
Cifar100
15
00.
0.50
0.52
10-
0.42
12
70
20
0.44
Shall Prongt
Dees inoerct
Poonut
160 250 316
Shots
100
200
2011
Number Prompt
Number ofoommp
(a) Data size trend
(h) Data size trend
(c) Prompt size trend
(d) Prompt size trend
Figure 15: (a-b) Accuracy-EEC trend with the number of parameters, (c-d): Accuracy-ECC Trend with data size
Number of classes: To study the impact of number of classes, we formulate 5-way I shot, 10-way 1 shot, and 100-way shot tasks using 100 The results are presented in Table 12. As we decrease the number of shots from 100 to 5, that the model is more accurate as tasks become casier (indicated by fewer classes i.e., lower N value in Table 12). However, the under-confidence issue remains.
increase in accuracy and a decrease in ECE. We observe
Number of unfrozen parameters: We conduct additional experiments on Cifar100 100-way 1-shot tasks by varying the number of prompts for 1) shallow prompt: prompt added to the input only and 2) deep prompt: prompt added to all Transformer encoder layers' input as well. The accuracy and ECE trends are presented in Figure 15 (c-d). As can be seen, with the increase in the number of prompts for both shallow and deep prompts, there are fluctuations in accuracy and ECE performance. However, the under-confidence issue persists for all the cases.
F.6 Additional Experiments and Comparison
In this section, we carry out additional experiments to study the OOD performance of our model, and compare model with additional methods present in literature.
The current work, being an instance of fine-grained uncertainty quantification works, could potentially help in OOD detection. To this end, we present the OOD results of Cifarl0 as in-distribution dataset and Cifar 100 as diitr--distiiuttion dataset with AUROC, FPR95, AUPR metrics for our model on 100-way 1-shot and 100-way 5-shot Cifar100 00 tasks in Table 13. As seen, B-PEFT performs better than PEFT, and with more training data, the model's 's OOD detection capabilities improve. Even with only 5 samples/class (i.e. 100-way 5-shot Cifar 100 task), the model can achieve an AUROC of 92.58.
OOD Performance:
We first carry out experiments with cosine classifier [22] without training for the 100-way 1-shot task on Cifar100. The cosine classifier has comparable generalization performance (accuracy) in comparison to VPT based model (see Table 14). However, looking at the ECE, the miscalibration issue is even higher than VPT based model. Hence, the simple solution (cosine classifier) does not ensure calibrated predictions. We also carry out experiments with test time augmentation [4] and VPT fine tuning with LoRA [70], on 100-way 1-shot C0 dataset. Towards comparison with Bayesian inspired methods [14], we use Laplace approximation on last layer of the model using Kronecker Product and Diagonalization represented by KronLaplace and DiagLaplace in Table 14. As can be seen, these methods also suffer from the under-confidence issue when straightforwardly extended to the VPT. B-PEFT achieves much better generalization and calibration performance than these basclines.
Model Comparison:
27

---

Table 11: Full fine-tuning results
Table 12: N-Way 1-Shot Cifar100 calibration
Model
Accuracy
ECE
Task
Accuracy
ECE
Full Fine Tuning
25.75
0.118
5 way shot 63.60
0.324
FT + Base rate adj.
25.75
0.037
10 way shot 56.35
0.370
VPT B-PEFT
48.63
0.393
100 way shot 48.63
0.393
52.34
0.067
Table 13: 100-way Cifar100 OOD experiments
Table 14: Comparison with baselines
PEFT
AUROC
AUPR
FPR95
Model
Accuracy
ECE
shot
79.53
80.09
70.58
Cosine Classifier
47.99
0.493
shot
90.93
90.75
39.82
VIT + LoRA
48.19
0.243
Test Time Aug.
50.10
0.271
(B-PEFT)
AUROC
AUPR
FPR95
VPT KronL
50.26
0,475
shot
81.24
81.98
68. 15
VPT DiagLaplace
50.20
0.474
shot
92.58
92.85
35.24
VPT
48.63
0.393
B-PEFT
52.34
0.067
G Calibration Behavior of Self-Supervised Model
In this work, we focus on vision foundation models that are pre-trained in supervised learning paradigm. These models have shown remarkable effectiveness in a wide range of areas including image classifi- cation, video understanding, and visual recognition among others. Alternatively, foundation models have been developed that pre- pre-train a self-supervised fashion (e.g., CLIP [52]). These models demonstrate good zero-shot performance on various datasets. As representative of the self-supervised models, use CLIP in our experiments. Parameter-efficie methods [77, 76, 21] have been proposed to adapt the CLIP model for downstream tasks. We use the popular methods: adapter and prompt for few-shot adaptation to study the calibration behavior. The re- liability plot of different shot adaptations (1,2 and shots per class) for cifar 100 along with accuracy and ECE is presented in Figure 17. The accuracy behavior as we increase the number of shots from to 20 shown in Figure 16. In both methods, we observe for few-shot adaptations, the accuracy is either lower or comparable to zero-shot perfor- mance. More specifically, for adapter-basse based adaptation, even with shots per class, the accuracy does not reach zero-shot accuracy. On the contrary, the parameter-efficiee fine-tuning of supervised foundation models shows consistent improvement in accuracy as we increase the number of shots.
Num Shots Accuracy
60
55
zero-shot prompt adapter
Shots
Figure 16: Accuracy trends of CLIP on few- shot adaptation
Towards calibration performance, we observe that prompt-based adaptation, along with zero-shot generally show good calibration performance with some degree of overfitting (see Figure 17). In contrast, prompt adaptation for supervised models that are severe, and prompt adaptation for CLIP models have such issue. However, the adapter-based adaptation mostly over-confident and hence has higher ECE value than our proposed model. Considering these results for parameter-efficiee few-shot adaption of pre-trained self-supervised models, the calibration and uncertainty behavior of such pre-trained models pose an interesting direction for further investigation.
H Societal Impact
We study the parameter-efficient fine-tuning techniques for large pre-trained vision foundation models, where identify two key issues: the under-confidence of the fine-tuned models in their
28

---

Accuracy: 41.960, ECE: 0.091
Accuracy, 44.370, ECE: 0.076
Accuracy. 49.050, ECE: 0.056
1.0
1.0
1.0
0.8
0.8
Accuracy
0.6
Accuracy
1.0
0.0
0.2
0.4
0.4 0.8
Confidence
Confidence
Confidence
Shot using prompt
(b) using prompt
(c) 5 Shot using prompt
1.0
Accuracy: 21.370, ECE: D.225
1.0
Accuracy: 29.150, ECE: 0.212
1.0
Accuracy. 43, 340 ECE: 0.152
0.8
0.0
0.8
Accuracy
0.6
0.6
Accuracy
0.6
1.0
Contidence
Confidence
Confidence
(d) Shot using adapter
(e) Shot using adapter
(f) 5 Shot using adapter
Figure 17: Different Shot adaptation:
Calibration performance CLIP based model on cifar100
dataset using prompt and adapter-based fine-tuning
predictions, and lack of fine- fine-grained uncertainty quantification capabilities. We develop novel Bayesian Evidential model: B-PEFT that addresses the weaknesses existing PEFT for pre-trained foundation models. Being instance of the PEFT, our developed model enables the large foundation models to be adapted to challenging few-shot problems in a parameter-efficient and computationally efficient manner with limited memory requirements and energy footprint. Moreover, the developed model improves the generalization performance and the model's predictions are calibrated ensuring trustworthiness. Finally, the model has fine one-grained uncertainty quantification capabilities which are highly desirable when applying these models in real-world -rittccll scenarios. Overall, our developed B-PEFT expected to have a strong positive societal impact.
I Limitations and Future Work
In this work, investigate the calibration of cransformer-basee foundation models under few-shot adaptation using various parfmeter-efficient fine-tuning methods. We focus on fine-tuning supervised pre-trained models for few-shot learning. We note that there other supervvue pre- pre-trained models that show promising results for various benchmark datasets. We investigate the calibration of CLIP, a representative method, under few-shot adaptation using the two popular parameter- efficient fine-tuning methods: prompt and adapter. Our preliminary results demonstrate that the few-shot performance does not consistently increase in comparison zero-shot. Similarly, prompt- based fine-tuning has relatively better calibration than adapter-based fine-tuning. As an extension of this work, we will investigate the calibration performance of self-supervised foundation models. Additionally, it could be interesting to study the calibration performance of PEFT for tasks beyond image classification, e.g., to other modalities such audio and language foundation models. For instance, the ideas developed in this work could potentially be used in situations where the PEFT leads to mis-calibrated models and the developed model requires trustworthy fine- fine-grained uncertainty quantification capabilities. If these data modalities are also modeled using transformers and follow the parameter-efficient fine-tuning paradigm in performing downstream tasks, we expect the proposed approach can benefit them in similar way.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction present the paper's contribution and scope, and match the theoretical and empirical results the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the proposed work have been discussed in Section I.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Complete proof of all the theoretical claims is presented.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details for reproducibility along with link to the code is provided.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Only benchmark datasets and publicly available models are used for training and evaluation.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental setting and details provided.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All the key results present the mean and standard deviation of five trials.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Details of all compute resources used in experiments is provided
9. Code Of Ethics
30

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurr:: httppp///aarrpp..ppppptttt////// httpp///ww....ccccccciiaooooooaa
Question:
Answer: [Yes]
Justification: The work confirms to the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The socictal impact is discussed in Section H.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work poses obvious high risk for misuse.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: References and citations are provided as required.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The link to the source code and resources is provided.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing and research with human subjects involved in this research work.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification; No crowdsoureing and research with human subjects involved in this research work.
31


Reviews:
[{"review_id": "BIiTDBF303", "reviewer": "Reviewer_7WZR", "summary": "This paper advances the Bayesian Parameter Efficient Fine-Tuning (Bayesian-PEFT) approach: a strategy to fine-tune large pre-trained foundation models with well calibrated classifier and ability to gracefully deal with out-of-distribution (OOD) data thanks to uncertainty quantification from ensembling.\nFirst the authors study the behavior of a fine-tuned foundation model with several PEFT methods (bias fine-tuning, adapters, side-tuning, visual prompt-tuning) in the few-shot learning regime (defined here as reducing the number of training samples per class, but keeping the original number of classes) and show that even though they achieve good accuracy, they all display under-confident predictions. The authors attempt to trace back the source of this behavior with evidential learning tools and conclude that the models are accurate thanks to relatively greater evidence of the correct class compared to other classes, and that models are under-confident because all classes are being assigned very low evidence, including the correct ones.\nInspired by this finding, the authors devise an evidential learning based PEFT strategy that adjusts the base rates of the evidential head such that prior belief from pre-training knowledge is strengthened while maintaining accuracy. Secondly the authors propose an evidential ensemble with a technique to induce diversity with different incorrect belief regularization strengths that penalize the model differently when assigning high beliefs to non-correct classes.\nThe proposed Bayesian-PEFT is evaluated on 4 datasets  (CIFAR10, CIFAR100, Food101, Flowers102) under different few-shot regimes, showing boosted accuracy and calibration.", "questions": "This paper takes an interesting direction of study: ensuring reliability of PEFT foundation models in the low-shot learning regime.\nI find the endeavor of the authors nice, with a good story and several PEFT methods. However I do have several concerns regarding the chosen few-shot learning setup (different from the common practices in the literature) and not necessarily realistic, the limited motivation brought for the use of evidential learning as well as the limited baseline. In addition, the relatively rich literature of Bayesian methods for few-shot learning is ignored.\n\nMy current rating is leaning towards reject at this time (a bit on the fence), but I'm looking forward for the rebuttal.\n\nHere are a few questions and suggestions that could be potentially addressed in the rebuttal or in future versions of this work (please note that suggested experiments are not necessarily expected to be conducted for the rebuttal):\n\n1. Please argue why using evidential learning should be used for this setting rather than last-layer or other Bayesian inspired methods\n\n2. What is the performance of a cosine classifier in this setting? What about Laplace-Redux or LoRA ensembles?\n\n3. How is this method working with LoRA-like methods, e.g., VeRA?\n\n4. Why considering the proposed few-shot learning setup instead of the one from legacy few-shot learning / meta-learning literature\n\n5. Can the authors discuss the difference and benefits of Bayesian-PEFT w.r.t. Bayesian few-shot learning methods?\n\n6. Inclusion of OOD specific metrics in the OOD experiment.", "limitations": "The authors addressed some limitations in the supplementary.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "### Significance\n- this work deals with an important problem in the current landscape of machine learning: how to leverage existing foundation models to downstream tasks with few labels while ensuring their reliability\n \n\n### Originality\n- I find the use of evidential learning for this setting to be original. The idea of preserving prior knowledge and increasing the evidence for all classes with evidential learning for this use-case is novel\n\n- the strategy of inducing diversity of ensembles with different strengths for incorrect evidence regularization is novel for the evidential learning literature and has the benefit of not needing communication between ensemble particles during training -> improved computational efficiency\n\n### Clarity\n- the paper is overall clearly written conveying a well argued story about the behavior observed for different PEFT methods in the few-shot regime, then studying it with evidential learning lens and then proposing a few solutions to mitigate the encountered non-desirable behaviors\n\n- the code comes to support information potentially missing from the paper\n\n### Quality\n\n- the authors conduct numerous experiments under several few-shot learning regimes, showing improved performances. They also conduct several ablations and sensitivity studies towards better understanding the impact of the building blocks of this method and the choices made for them\n\n- code is provided in the supplementary for reproducibility of results\n\n- the authors study their work on several PEFT methods from computer vision literature, mostly VPT in the main paper and the others in supplementary", "weaknesses": "### Framing of few-shot learning task\n- the few-shot learning setting considered here is quite different from the one considered in most of the few-shot learning/meta-learning literature with many small sets of support images and classes and query images (sampled from miniImageNet, Omniglot, CIFAR, etc.), sometimes called _\"episodes\"_. In contrast the proposed few-shot learning setting proposed here keeps the original number of classes and reduces the number of samples to 1-20 samples per class. This ends up as a different setting as the model is finetuned on 200 to 4000 images for CIFAR100 for instance from the 1-shot to the 20-shot setting, which is not a big dataset but still not that few images to train on.\n\n- I'm wondering whether the proposed setup of having many classes, but few samples, may be the source of the under-confidence behavior of PEFT methods for the few-shot learning setup that is proposed here. In such cases, simple solutions from the few-shot learning literature that do not require training: e.g., cosine classifier [b], [c]  (i.e., averaging features from the same class, do dot-product with query features, then softmax on similarities)\n\n- the authors could refer to this setting that they propose by a different name to avoid confusion with few-shot learning where models can be finetuned with very few samples and there are several Bayesian methods there already. A potential alternative could be: low-shot regime.\n\n\n### Motivation for evidential learning approach and few baselines\n- the formalism based on evidential learning is interesting, but is not always trivial to implement and previous works [d],[e] reported difficulties in scaling evidential methods and the Dirichlet predictor to larger number of classes and encoders.\n\n- while the approach itself is clearly described, the motivation for using such a line of approach instead of others that are more simple, generic and with a wider adoption, e.g., last-layer methods like Laplace-Redux [f], Bayesian last layer [g]. Since the endeavor of this work is to look at more practical settings with few labeled samples and reliability constraints, it may make sense to compare this work with some simple baselines to show the benefits of the approach and its trade-offs.\nOther potentially relevant baselines would be the cosine classifier from few-shot learning, LoRA ensembles [h], linear probing with test-time augmentation [i]\n\n- It would be also interesting to expand the list of PEFT methods studied with more recent and potentially more parameter efficient ones like LoRA [j] or VeRA [k] that have been shown useful to large transformer networks and could expand the scope of this work beyond image classification.\n\n\n### Missing related works\n- as mentioned above, the field of few-shot learning/meta-learning has seen several Bayesian methods with uncertainty quantification. Often these meta-learning methods fine-tune the network at test-time with just a few labeled samples (from classes not seen during pre-training). Here is a non-exhaustive list of relevant works from this area: PLATIPUS [m], VERSA [n], LLAMA [o], Bayesian MAML [p], Bayesian TAML [q], etc.\n\n- a discussion on how the current B-PEFT is different from them and what type of problems it addressed and with which benefits would be useful here.\n\n- the idea of converting a pre-trained network into a Bayesian Neural Network and training over a short period of a few epochs (not a low-shot setting though) has been recently proposed in ABNN [l]. The diversity-inducing mechanism with different incorrect belief regularization strengths can also be related with the random prior mechanism from ABNN [l] where an ensemble of networks was obtained from a pre-trained network and the diversity was induced by introducing a different class-level prior for each ensemble member. ABNN is pretty recent so it can be considered as concurrent work. \n\n\n### Misc.\n- the OOD experiment could benefit from the inclusion of typical metrics used in literature (e.g., AUROC, FPR95, AUPR, etc.) to allow easier comparison and putting the performance in context of other works.\n \n\n### Minor - Novelty\n- the evidential learning choice seems to be inspired to some extent by the multidimensional belief quantification work [37] that used uncertainty for vanilla few-shot learning / meta-learning. The authors however expand the idea further with belief strenghtening and incorrect belief regularization for ensembling.\n\n\n\n\n**References:**\n\n[a] Vinyals et al., Matching networks for one shot learning, NeurIPS 2016\n\n[b] Gidaris et al., Dynamic Few-Shot Visual Learning without Forgetting, CVPR 2018\n\n[c] Qi et al., Low-Shot Learning with Imprinted Weights, CVPR 2018\n\n[d] Joo et al., Being Bayesian about Categorical Probability, ICML 2020\n\n[e] Franchi et al., Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification, arXiv 2020\n\n[f] Daxberger et al., Laplace Redux -- Effortless Bayesian Deep Learning, NeurIPS 2021\n\n[g] Kristiadi et al., Being bayesian, even just a bit, fixes overconfidence in relu networks, ICML 2020\n\n[h] Balabanov et al., Uncertainty quantification in fine-tuned LLMs using LoRA ensembles, arXiv 2024\n\n[i] Ashukha et al. Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning, ICLR 2020\n\n[j] Hu et al., Low-rank adaptation of large language model, ICLR 2022\n\n[k] Kopiczko et al., VeRA: Vector-based Random Matrix Adaptation, ICLR 2024\n\n[l] Franchi et al., Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models, CVPR 2024\n\n[m] Finn et al., Probabilistic Model-Agnostic Meta-Learning, NeurIPS 2018\n\n[n] Gordon et al., Meta-Learning Probabilistic Inference For Prediction, ICLR 2019\n\n[o] Grant et al., Recasting gradient-based meta-learning as hierarchical bayes, ICLR 2018\n\n[p] Kim et al., Bayesian Model-Agnostic Meta-Learning, NeurIPS 2018\n\n[q] Lee et al., Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks, ICLR 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your constructive comments, thorough review, and suggestions. Here are the responses to the clarifying questions: \n\n**Q1: Framing of few-shot learning problem**\n\nIn this work, we consider few-shot classification problem, i.e., the $N$-way $K$-shot classification problem, where the $N$-class classifier has $K$ examples per class to learn from. For instance, 1-shot Cifar100 is a 100-way 1-shot classification problem where the support set has a total of 100 examples (i.e., 1 example per class), and the model is evaluated on the test set (identical to the query set in the meta-testing tasks). We clarify that the current work does not rely on the episodic learning paradigm of meta-learning (matching networks, MAML, PLATIPUS), where both meta-training and meta-testing are done on task level in an episodic fashion with a large number of N-way K-shot tasks.  In fact, we consider more challenging few-shot learning tasks (e.g., 100-way 1-shot in Cifar100 and 102-way 1-shot in Flowers102) compared to the commonly used 5-way 1-shot meta-learning tasks. We leverage the power of the pre-trained foundation models eliminating the need of task based episodic meta-training. From a meta-learning perspective, the pre-training phase for the foundation model could be viewed as the meta-knowledge acquisition phase (i.e., meta-training). The pre-trained model can be seen as an expert equipped with meta-knowledge, and parameter-efficient fine-tuning technique performs quick adaptation to the downstream tasks, analogous to the support-set based adaptation done in meta-testing.  To test our method in the standard 5-way 1-shot tasks, we apply our model to the mini-ImageNet dataset test tasks and compare it with representative meta-learning models. The results are summarized in Table 1 in the attached PDF. The PEFT-based model outperforms the episodic training meta-learning models demonstrating the potential of large vision foundation models for effective few-shot learning.\n\n**Q2: Number of classes is the source of Under-confidence**\n\nPlease refer to the answer to **Q1** in the overall response. \n\n**Q3: Performance of simple solutions that do not require training**\n\nAs suggested by the reviewer, we carry out experiments with cosine classifier without training for the 100-way 1-shot task on Cifar100. The results are shown in Table 6 of the PDF file. The cosine classifier has comparable generalization performance (accuracy) in comparison to VPT based model. However, looking at the ECE, the miscalibration issue is even higher than VPT based model. Hence, the simple solution (cosine classifier) does not ensure calibrated predictions.\n\n**Q4: Low-shot regime suggestion**\n\nThank you for the suggestion. We will add a detailed discussion of the few-shot learning setup, clearly explaining that we differ from  the existing meta-learning works as we no longer rely on the meta-training phase and episodic learning paradigm.\n\n**Q5: Scaling evidential methods and the Dirichlet predictor to larger number of classes and encoders.**\n\nThe lack of scalability of EDL has been observed and addressed by Ref [38] (Learn to Accumulate Evidence from All Training Samples: Theory and Practice, ICML 2023) that enables the evidential models to be scaled to ResNet for Cifar100, and we further extend the evidential framework to Vision Transformers in challenging 100-way, 101-way and 102-way few-shot tasks further demonstrating the scalability of evidential models. \n\n**Q6: Comparison with baselines, e.g., last-layer methods like Laplace-Redux, Bayesian last layer, and linear probing with test-time augmentation**\n\nWe carry out experiments with the suggested models on 100-way 1-shot Cifar100 dataset. The results are presented in Table 6. We observe that these methods also suffer from the under-confidence issue when straightforwardly extended to the VPT. We resort to the Evidential Deep Learning framework to address the under-confidence issue of the VPT model. Similarly, we also fine-tune the ViT model using Lora as another parameter efficient fine-tuning method. The fine-tuning results in under-confidence issue as well. Further expanding the scope of the work to tasks beyond image classification with LoRA and VeRA based methods can be an interesting followup work. Thank you for the suggestion. \n\n**Q7: Discuss the difference and benefits of Bayesian-PEFT w.r.t. Bayesian few-shot learning methods?**\n\nPlease refer to the answer to **Q2** in the general response for a detailed discussion of the proposed method over Bayesian inspired models, which include all the Bayesian few-shot learning methods. As a result, these Bayesian models are inadequate to address the unique under-confidence behavior associated with the PEFT methods. Furthermore, benefiting from a large pre-trained vison foundation model, the proposed method is able to achieve much higher prediction performance as compared with the Bayesian few-shot learning methods trained through episodic meta-training with the specifically constructed few-shot tasks. In Table 1 of the attached PDF file, we show the results on 5-way 1-shot mini-ImageNet. As can be seen, our model outperforms the bayesian few-shot learning models in existing benchmark task by a large margin.\n\n**Q8: ABNN [l] Discussion**\n\nABNN introduces bayesian normalization layers after training of deep learning model, and requires additional training of these layers in a post-hoc manner. In theory, an evidential deep learning model could be augmented with the bayesian normalization layers/ABNN idea as an alternative to belief-based diversity. We will include discussion of ABNN in the updated draft, and leave exploration of ABNN based diversity for bayesian evidential model as a potential future work. \n\n**Q9: Inclusion of OOD specific metrics in the OOD experiment.**\n\nPlease see the answer to **Q3** in the overall response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "44OHajuUP5", "reviewer": "Reviewer_DMeg", "summary": "This paper studies the problem of parameter efficient fine-tuning. The authors pointed out mis-calibration issues caused by parameter efficient fine-tuning, which is, more specifically, under-confident estimation when fine-tuning data is limited. To solve this issue, the author proposed Bayesian PEFT, a Bayesian parameter efficient fine-tuning method that introduces two components, i.e., a base rate adjustment to strengthen the prior belief of pre-trained knowledge, and a evidential ensemble with diverse ensemble components. The authors provided both theoretical justification as well as extensive experiments across diverse datasets.", "questions": "1. Is fully fine-tuned model also have low uncertainty compared to PEFT model? \n\n2. Will the proposed method improve robustness such as OOD generalization in terms of model accuracy? \n\n3. How does the proposed approach work on other modalities, such as audio and language?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well written and easy to follow.\n\n2. The paper discusses an interesting observation on low confidence in few-shot setup for PEFT.\n\n3. The proposed method seems to be reasonable and solve the identified problem.", "weaknesses": "1. It would be interesting to see more in depth discussion and empirical analysis on the observation of uncertainty of PEFT, particularly, related to data size and number of unfrozen parameters. \n\n2. More discussion related to OOD generalization might be needed. Particularly, how does the proposed method help OOD generalization in terms of prediction accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your constructive comments and suggestions.  Here are the responses to the clarifying questions: \n\n**Q1: It would be interesting to see more in depth discussion and empirical analysis on the observation of uncertainty of PEFT, particularly, related to data size and number of unfrozen parameters.**\n\nPlease see the answer to **Q1** in the overall response.\n\n**Q2: Clarification on OOD performance**\n\nThe current work, being an instance of fine-grained uncertainty quantification works, can help in OOD detection. We further present the OOD performance of our model in the answer to **Q3** of the overall response, that shows the potential of our model in handling OOD.\n\n**Q3: Is fully fine-tuned model also have low uncertainty compared to PEFT model?**\n\nFull fine-tuning in few-shot learning tasks leads to overconfident models (see Table 5/Figure 1(a) in the attached pdf) and, due to overfitting, also hurts the model's generalization performance.\n\n**Q4: How does the proposed approach work on other modalities, such as audio and language?**\n\nIn this work, we focus on few-shot image classification task and extension to other modalities is beyond the scope of this work. \nIt could be interesting follow-up to study the calibration performance of PEFT for audio/language foundation models. For instance, the ideas developed in this work could potentially be used in situations where the PEFT leads to miscalibrated models and the developed model requires trustworthy fine-grained uncertainty quantification capabilities. If these data modalities are also modeled using transformers and follow the parameter efficient fine-tuning paradigm in performing downstream tasks, we expect the proposed approach can benefit them in a similar way.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "slXIBXxDaj", "reviewer": "Reviewer_YJfL", "summary": "This submission presented a lightweight Bayesian Parameter Efficient Fine-Tuning (Bayesian-PEFT) framework for large transformer-based foundation models. Experiments across diverse datasets demonstrated the improved calibration performance by Bayesian-PEFT on multiple other PEFT techniques.", "questions": "* In Eq. (6), would there be a numerical stability problem when dividing e_{min}, as this value can be very small?", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well written and the experiments are extensive.\n\n* The proposed transformation function $A_m$ is well motivated and the experiments supports authors' claim.", "weaknesses": "* The dependence on Evidential Deep Learning is quite significant. To some extent, the proposed method extends EDL, rather than being designed specifically for PEFT of foundation models.\n\n* Some editorial errors (e.g., \"we we\") exists, the submission may need further proofreading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your constructive comments and suggestions.  Here are the responses to the clarifying questions: \n\n**Q1: The dependence on Evidential Deep Learning is quite significant. To some extent, the proposed method extends EDL, rather than being designed specifically for PEFT of foundation models.**\n\nPlease refer to the answer to **Q2** in the general response, which makes it clear that the proposed approach is specifically designed to address the unique under-confidence behavior of PEFT methods. We extend evidential  learning in novel ways by performing fine-grained uncertainty decomposition and base rate adjustment, which allows us to seamlessly integrate it into parameter efficient fine-tuning paradigm. Such an integration has the potential to significantly broaden the usage of  foundation models in many critical domains by fixing their calibration issues. \n\n**Q2: Typos and minor mistakes**\n\nWe will proofread the draft, and correct typos/minor mistakes in the updated draft.\n\n**Q3: Numerical Stability Issue in Eqn. 6**\n\nYes, since the evidence is in the range [0, infinity] (the evidence is obtained by the exponential of the logit outputs), the stability issue can arise in extreme cases when the logit output is extremely low (i.e. close to negative infinity). In our experiments, we did not observe the stability issue. Still, the issue can arise in some extreme cases, and to address this challenge, we could introduce a small delta in the denominator or bound the network’s logits to be greater than a small negative value.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OrhAzUgFH2", "reviewer": "Reviewer_CbTK", "summary": "This paper focuses on the parameter efficient fine-tuning (PEFT) for foundation models in the few-shot learning settings. The authors reveal that the adapted models lack accurate fine-grained uncertainty quantification capabilities. Specifically, the few-shot tuned models perform remarkable accuracy but with large expected calibration errors. To address this problem, they propose a lightweight Bayesian PEFT framework by designing two Bayesian components to improve models' confidence. Theoretical analysis and experiments on 4 visual benchmarks prove the effectiveness of the proposed method.", "questions": "Does the proposed Bayesian-PEFT work well in LLMs, e.g., LLaMA3?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper is well-written and the storyline is clear.\n2.The proposed Bayesian-PEFT is simple-and-effective, and theoretical analysis in the paper can reveal the underlying mechanism of the method.\n3.Extensive experiments on ViT in several few-shot settings show the effectiveness of Bayesian-PEFT.", "weaknesses": "1.The title may be a bit inappropriate. If the Bayesian-PEFT is claimed to be used for foundation models, experiments on more types of foundation models are required. For example, I am curious whether it still works well on large language models (LLMs), e.g., LLaMA3. Also, it is unclear whether the under-confidence issue observed in the ViT also exists in LLMs.\n2.Although it may be beyond the scope of the paper's research, I am still curious whether the under-confidence issue studied in the paper disappears when there is sufficient training data.\n3.It would be better if a schematic diagram of the proposed Bayesian-PEFT could be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your constructive comments and suggestions.  Here are the responses to the clarifying questions:\n\n**Q1: LLM Clarification**\n\n\nThe proposed ideas in this work could potentially be extended to PEFT of LLM models for some downstream tasks. However, it is worth noting that the uncertainty quantification for LLM is challenging due to their generative nature, and is an open question. For instance, LLAMA3 is a generative model, and uncertainty quantification of the overall generated text response for an input query is difficult to interpret. In this work, we focus on PEFT of vision foundation models, where we identify the under-confidence issue for all 4 PEFT techniques considered. We introduce a novel evidence-guided method to address this challenge. We will update the title from foundation models to vision foundation models to make it more accurate.\n\n**Q2: Under-confidence with sufficient data**\n\nPlease refer to the answer to **Q1** in the general response for the results related to PEFT with sufficient data. \n\n\n**Q3: Schematic Diagram of B-PEFT**\n\n We present the schematic diagram of B-PEFT in Figure 3 of the attached pdf of the general response. We will include the schematic diagram in the revised draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Overall Response

We thank all reviewers for their valuable feedback and constructive suggestions. We identify some important questions raised by multiple reviewers and answer them together in our general response below. 

**Q1: Under-confidence behavior of PEFT methods w.r.t. number of classes (Reviewer 7WZR), data size (Reviewers 7WZR and CbTK), and number of unfrozen parameters (Reviewer 7WZR).**

In order to more thoroughly understand the under-confidence behavior of PEFT methods under diverse settings as suggested by the reviewers, we refer to some of already reported results in the paper while conducting additional experiments. The new results are presented in the attached PDF file. We summarize our main findings as follows:

- *Data size*: In Table 1 of the paper, we varied the number of training samples per class from 1 to 20. We observe  that the model's accuracy increases with more training samples. However, under-confidence remains, even with the increase in training samples. We conduct additional experiments on Cifar100 by increasing the training samples per class to 500  and observe the under-confidence issue despite the increase in the accuracy. The trend is summarized in Figure 2 (c-d) of the attached PDF. We see an increase in accuracy and a decrease in ECE. However, even with 500 samples per class, the under-confidence issue remains. Further, we report the accuracy and ECE of the fully fine-tuned model (fine-tuning of all the parameters) for 1 shot cifar100 in Table 5 of the attached pdf. We observe a decrease in accuracy. The miscalibration issue still remains. We observe the overconfidence behavior by looking at the reliability plot as presented in Figure 1 (a). The performance suggests the overfitting of the fully tuned model.


- *Number of classes*: To study the relationship between model accuracy, uncertainty, and the number of classes, we formulate standard 5-way 1-shot MiniImagenet tasks and apply the PEFT to these tasks. The results are summarized in Table 2 of the attached PDF. As can be seen, the model achieves high overall accuracy of 89.78\%. However, the model remains under-confident with an ECE of 0.418, where under-confidence is shown by the the reliability diagram in Figure 1. We further formulate 5-way 1 shot, 10-way 1 shot, and 100-way 1 shot tasks in Cifar100. The results are presented in Table 3. As we decrease the number of shots from 100 to 5, we see an increase in accuracy and a decrease in ECE. However, the under-confidence remains.

- *Number of unfrozen parameters*: We conduct additional experiments on Cifar100 via 100-way 1-shot tasks by varying the number of prompts for 1) shallow prompt: prompt added to the input only and 2) deep prompt: prompt added to all Transformer encoder layers' input as well. The accuracy and ECE trends are presented in Figure 2 (a)/(b) of the pdf. As can be seen, with the increase in the number of prompts for both shallow and deep prompts, there are fluctuations in accuracy and ECE performance. However, the under-confidence issue persists for all the cases. 

**Q2: Justification of using evidential learning to achieve calibrated PEFT (Reviewers 7WZR and  YJfL)**

As compared with the Bayesian-inspired models, evidential learning offers two key properties that allow us to formulate a principled solution to address the unique under-confident behavior of the PEFT methods. First, thanks to its evidence-based fine-grained uncertainty decomposition capability, we can separate two distinct sources of second-order uncertainty, including vacuity (line 172) and dissonance (line 176). Different from the commonly used first-order uncertainty (e.g., entropy), these two second-order uncertainty serve as a key tool to understand why PEFT methods are both accurate (with a low dissonance) while being under-confident (with a high vacuity). This key insight suggests that these methods systematically under-estimates the contribution from the prior knowledge to the downstream task. While the classical Bayes’ theorem offers a principal idea to address the issue, which is to strengthen the prior belief, there is a lack of practical way to achieve this. As the second key property, evidential learning allows us to leverage the base rate, which is rooted in the subjective logic theory as an effective vehicle to adjust the prior belief gained through pre-training. To this end, we propose a transformation function in Eq (6) to adjust the base rate that leads to the increase of the model confidence while maintaining the predictive accuracy of the model as guaranteed by our theoretical results in Lemma 2 and Theorem 3. 

 We also conduct additional experiments and compare our approach with Bayesian inspired models for 100-way 1-shot Cifar100 tasks. The results are reported in Table 6.  We use laplace approximation on last layer of the model using Kronecker Product and Diagonalization represented by KronLaplace and DiagLaplace in the table. As can be seen, the Bayesian-inspired models do not address the under-confidence issue, which still exhibits a fairly high ECE. This further confirms the effectiveness of the proposed evidential learning-based approach. 

**Q3:  OOD experiments and Clarification (Reviewers 7WZR and Dmeg)**

 We present the OOD results of Cifar10 as in-distribution dataset and Cifar100 as out-of-distribution dataset with AUROC, FPR95, AUPR metrics for our model on $100$-way $1$-shot and $100$-way $5$-shot cifar100 tasks in Table 4. As seen, with more training data, the model's ood detection capabilities improve. Even with only 5 samples/class (i.e. $100$-way $5$-shot cifar100 task), the model can achieve AUROC of 92.58.

Meta-review: The paper proposes a Bayesian Parameter Efficient Fine-Tuning framework for training large transformer-based foundation models and reducing underconfidence while ensuring reliable prediction in few-shot settings. The reviewers were broadly positive about the paper. They highlighted that they were generally convinced by the empirical evaluation presented in the paper and agreed that the problem of underconfidence due to PEFT is interesting and important. While no reviewer strongly recommends acceptance, I agree that the problem studied in the paper and the proposed solution are of interest to the NeurIPS community and that the paper should be presented at the conference.
Decision: Accept (poster)