Title: Be Confident in What You Know: Bayesian Parameter Efficient Fine-Tuning of Vision Foundation Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Model calibration

uncertainty quantification

few-shot adaptation

Parameter-Efficient Fine-Tuning
Abstract: Large transformer-based foundation models have been commonly used as pre-trained models that can be adapted to different challenging datasets and settings with state-of-the-art generalization performance. Parameter efficient fine-tuning ($\texttt{PEFT}$) provides promising generalization performance in adaptation while incurring minimum computational overhead. However, adaptation of these foundation models through $\texttt{PEFT}$ leads to accurate but severely underconfident models, especially in few-shot learning settings. Moreover, the adapted models lack accurate fine-grained uncertainty quantification capabilities limiting their broader applicability in critical domains. To fill out this critical gap, we develop a novel lightweight {Bayesian Parameter Efficient Fine-Tuning} (referred to as $\texttt{Bayesian-PEFT}$) framework for large transformer-based foundation models. The framework integrates state-of-the-art $\texttt{PEFT}$ techniques with two Bayesian components to address the under-confidence issue while ensuring reliable prediction under challenging few-shot settings. The first component performs base rate adjustment to strengthen the prior belief corresponding to the knowledge gained through pre-training, making the model more confident in its predictions; the second component builds an evidential ensemble that leverages belief regularization to ensure diversity among different ensemble components.
Our thorough theoretical analysis justifies that the  Bayesian components can ensure reliable and accurate few-shot adaptations with well-calibrated uncertainty quantification. Extensive experiments across diverse datasets, few-shot learning scenarios, and multiple $\texttt{PEFT}$ techniques demonstrate the outstanding prediction and calibration performance by $\texttt{Bayesian-PEFT}$.

Be Confident in What You Know: Bayesian Parameter Efficient Fine-Tuning of Vision Foundation Models

Abstract

Large transformer-based foundation models have been commonly used as pretrained models that can be adapted to different challenging datasets and settings with state-of-the-art generalization performance. Parameter efficient fine-tuning ( PEFT ) provides promising generalization performance in adaptation while incurring minimum computational overhead. However, adaptation of these foundation models through PEFT leads to accurate but severely underconfident models, especially in few-shot learning settings. Moreover, the adapted models lack accurate fine-grained uncertainty quantification capabilities limiting their broader applicability in critical domains. To fill out this critical gap, we develop a novel lightweight Bayesian Parameter Efficient Fine-Tuning (referred to as Bayesian-PEFT ) framework for large transformer-based foundation models. The framework integrates state-of-the-art PEFT techniques with two Bayesian components to address the under-confidence issue while ensuring reliable prediction under challenging fewshot settings. The first component performs base rate adjustment to strengthen the prior belief corresponding to the knowledge gained through pre-training, making the model more confident in its predictions; the second component builds an evidential ensemble that leverages belief regularization to ensure diversity among different ensemble components. Our thorough theoretical analysis justifies that the Bayesian components can ensure reliable and accurate few-shot adaptations with well-calibrated uncertainty quantification. Extensive experiments across diverse datasets, few-shot learning scenarios, and multiple PEFT techniques demonstrate the outstanding prediction and calibration performance by Bayesian-PEFT .

1 Introduction

Transformer-based foundation models have been developed as general models with state-of-the-art generalization performance [32, 66, 52, 28]. These models leverage the rich meta-knowledge acquired during the pre-training stage to effectively adapt to complex downstream tasks [32], where pretraining is usually performed on massive-scale annotated datasets ( e.g., [35, 55]) through supervised learning[32, 66, 28] or self-supervised learning [52]. To achieve effective adaption, various parameterefficient fine-tuning ( PEFT ) approaches have been developed [38, 27, 9, 54] that introduce a small number of tunable parameters either within or outside of the backbone architecture to ensure good generalization capability while incurring little computational overhead because most parts of (or the entire) backbone architecture is frozen during fine-tuning [25, 59, 67]. Bias-fine tuning [9], a representative partial tuning-based PEFT , only fine-tunes the bias parameters to downstream tasks. Adapter [51] and side-tune [72] fine-tuning techniques are instances of extra module-based PEFT that introduce extra parameterized modules and fine-tune them based on the downstream tasks. Visual Prompt-tuning [32] (VPT) follows the popular prompt learning paradigm by introducing a learnable

Head

Multihead

Attention

Add & Norm

LayerNorm

LayerNorm

Add & Norm

Frozen

Learnable

Head

Side Network

Adapter

Backbone

Input

weight

bias

ECE Vs. Accuracy from different PEFT methods

performing few (1-20)-shot adaption

(a)

(b)

1

0.90

0.75

SideTune

Adapter

0.60

Prompt

E

E

0.45

1 shot

B-PEFT

0.30

2shot

5 shot

0.15

10 shot

+

20 shot

0.00

10

30

50

70

90

prompt that is fine-tuned on the downstream task knowledge keeping the pre-trained backbone frozen.

Despite the attractive generalization performance, most foundation models adapted to downstream few-shot tasks through PEFT exhibit a somewhat surprising and undesirable behavior that may prevent them from being applied to many critical domains. Figure 1 (b) shows the predictive accuracy versus the Expected Calibration Error (ECE) of a foundation model after performing few-shot adaptation on CIFAR-100 using a series of representative PEFT methods, including VPT [32], Adapter [51], Bias Fine Tuning [71], and Side-tune [72]. It is clear that the adapted model is able to provide accurate predictions even after fine-tuned on limited training samples. For example, all PEFT methods help to boost the model's accuracy to over 75% using just 5-shot fine-tuning and the accuracy reaches 80% after 10-shot fine-tuning. However, the adapted model is very poorly calibrated as shown by the large ECE scores, which are consistently over 0.3 across all the fine-tuning methods. Increasing the fine-tuning size does not show clear improvement and sometimes even hurts the calibration performance. While one may expect the poor ECE to be caused by over-confidence as we fine-tune a large foundation model using very limited training samples, the detailed ECE plots as shown in Figure 2 (a)-(c) reveal that the model is in fact severely under-confident. For example, after adapting to the 1-shot training dataset, the VPT fine-tuned model can already achieve a test accuracy close to 50% but is under-confident in almost all its predictions leading to an ECE score over 0 . 45 . The under-confidence issue is observed for all representative PEFT methods across different datasets and various few-shot learning settings as evidenced by our experiments.

The under-confident few-shot adaptation behavior of foundation models closely mimics how human experts with rich domain knowledge in their own disciplines tend to make conservative decisions when facing new tasks that deviate from their own expertise. Analogous to their human counterparts, the rich prior knowledge gained through the pre-training stage of foundational models overshadows the relatively limited knowledge obtained through few-shot fine-tuning, which prevents them from making more confident predictions in the downstream tasks. Unreliable uncertainty ( i.e., confidence) quantification makes the predictions provided by these models less trustworthy, which may limit applying the promising 'pre-train-then-finetune' paradigm to many critical domains. As shown in Figure 2 (a)-(c), the fine-tuned model seldom makes any predictions with confidence over 80%, making it difficult to leverage these predictions in any high-stakes decision-making process.

The need to balance between the rich prior knowledge gained through pre-training and the new knowledge obtained through few-shot adaptation inspires us to investigate the under-confidence issue from the Bayesian perspective. In particular, we propose to look into the predictive behavior of the few-shot adapted foundation model through the lenses of evidential learning [56], which is built upon Bayesian theorem and subjective logic (SL) theory [33]. As part of the recent advances in modern Bayesian modeling, evidential learning provides a cost-effective way to perform Bayesian inference with the capability to quantify fine-grained second-order uncertainty [57]. By leveraging the important

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 49.155, ECE: 0.451

(a) VPT

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 47.435, ECE: 0.396

(b) Adapter

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 34.863, ECE: 0.327

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 49.155, ECE: 0.064

(d)

B-PEFT

theoretical connection between fine-grained uncertainty and model accuracy [46] , we unveil the underlying reason that supports the good predictive performance of a few-shot adapted foundation model and the root cause for the under-confident behavior. Drawing from this important insight as outlined above, we propose to integrate the modern PEFT techniques into a novel lightweight Bayesian framework, referred to as Bayesian-PEFT ( B-PEFT ), aiming to achieve highly reliable and accurate few-shot adaptations with well-calibrated and trustworthy uncertainty quantification.

The proposed Bayesian framework offers two important components to address the under-confidence issue while ensuring reliable prediction under challenging few-shot settings. The first component makes novel adjustments to the base rates introduced by the SL theory to strengthen the prior belief corresponding to the knowledge gained through pre-training. Meanwhile, the adjustment does not change the relative order of the belief assigned to different classes, which ensures that the model accuracy is maintained. Our theoretical analysis shows that the proposed base rate adjustment strategy leads to more confident predictions (by increasing the gaps between the belief assigned to the groundtruth class and the rest) without compromising the model's accuracy. Figure 2d shows that B-PEFT significantly improves the model calibration. To further enhance the reliability of both prediction accuracy and uncertainty quantification when performing few-shot adaptation, the second component performs Bayesian model averaging by building a diversity-inducing evidential ensemble. In addition to using different random initialization of the PEFT components, diversity is further enhanced through incorrect belief regularization that penalizes a model for assigning a high belief to a non-ground-truth class. By controlling the strength of belief regularization, different ensemble components are guided to learn from diverse features in the data space, where a light penalty allows an ensemble component to learn the common discriminative features while a heavy one will force an ensemble component to learn rare features to avoid errors on the difficult data samples. A deeper theoretical analysis of the proposed diversity-induced evidential ensemble is equivalent to Stein Variational Gradient Descent (SVGD) based ensembles [13]. Experiments on multiple challenging few-shot learning tasks justify the superior performance of B-PEFT over state-of-the-art PEFT baselines, in terms of both prediction accuracy and uncertainty calibration performance. Our contributions can be summarized as follows:

We identify the severe under-confidence issue of pre-trained foundation models after performing parameter-efficient fine-tuning over few-shot datasets. The fine-grained uncertainty analysis through evidential learning and SL theory reveals the root cause for their good predictive performance while being under-confident.

We develop a novel lightweight Bayesian framework that integrates state-of-the-art PEFT techniques with two Bayesian components to address the under-confidence issue while ensuring reliable prediction under challenging few-shot settings. The first component performs base rate adjustment to strengthen the prior belief corresponding to the knowledge gained through pre-training, making the model more confident in its predictions; the second component builds an evidential ensemble that leverages belief regularization to ensure diversity among different ensemble components.

We perform thorough theoretical analysis to justify why the proposed Bayesian components can ensure reliable and accurate few-shot adaptations with well-calibrated uncertainty quantification.

We carry out experiments with 4 benchmark datasets, 5 different few-shot settings, and 3 parameter efficient fine-tuning techniques that demonstrate the effectiveness of the developed model.

2 Related Works

Parameter Efficient Fine Tuning of Foundation Models. Transformer-based foundation models [63, 15] have been developed as an improvement over traditional convolution-based architectures

[29, 31] for computer vision tasks. The transformer-based models achieve strong generalization performance [40] after training on large datasets. Moreover, the pre-trained transformers can be fine-tuned in limited data settings leading to state-of-the-art performance [32, 27]. As a computation, memory, and parameter-efficient alternative to full fine-tuning of such large pre-trained transformers, different Parameter Efficient Fine Tuning (PEFT) approaches have been developed. PEFT techniques freeze most of the large transformer backbone, fine-tune the remaining backbone parameters and/or introduce lightweight extra modules for adapting to the downstream task. Existing approaches can broadly be categorized as extra-module-based [72, 54], partial-tuning-based [71, 9], and visual prompt tuning-based [68, 68, 28] methods. Extra-module-based methods ( e.g., Adapter [51], side-tune[72]) introduce small additional learnable modules and keep the pre-rained backbone frozen. Partial-tuningbased methods ( e.g., Bias [9]) keep a large portion of the backbone frozen, and fine-tune only part of the foundation model to downstream tasks. Visual prompt-based PEFT methods (VPT) introduce a learnable prompt variable along with a learnable classification head over the fixed pre-trained backbone to be adapted to the downstream task. VPT [32] has shown significant improvement over other PEFT techniques, and can even outperform full fine-tuning in multiple datasets/settings [28].

Calibration in Deep Learning Models Calibration methods have been increasingly explored to achieve trustworthy deep-learning models. Post-hoc calibration methods [26, 43, 73] aim to learn a calibration map for a standard trained deep learning model such that the map can transform the poorly calibrated probabilities to calibrated probabilities. Regularization-based calibration approaches introduce explicit regularization (such as with L 2 regularization [26], entropy regularization [50]), or implicit regularization (such as with focal loss [39]) during training to ensure that the trained model is calibrated. Data augmentation methods such as Label smoothing [44], and mixup training [60][74] have also been explored for developing calibrated deep learning models. Recent survey [65] provides a discussion of the most relevant works towards developing calibrated deep learning models. Most existing calibration methods are designed to tackle the over-confidence issue, which is more commonly observed for large models trained from limited data due to overfitting. We observe that fine-tuned foundation models exhibit severe under-confidence in their predictions, where existing calibration techniques are less effective. To this end, we propose a lightweight Bayesian framework that fills this critical gap.

Few-Shot Adaptation and Relationship with Meta-Learning. In this work, we consider few-shot adaption with a focus on N -way K -shot classification [64, 22], where the model is presented with a few-shot training set with N -class, each having K examples. For instance, 1-shot Cifar100 training set consists of 1 sample from each of the 100 classes. The model is then evaluated on the test set, which is identical to the query set in the meta-testing tasks [58]. It is worth to note that meta-learning ( e.g., matching networks [64], MAML [16], VERSA [23], PLATIPUS [17]) leverages an episodic learning paradigm to achieve few-shot adaptation, where both meta-training and meta-testing are done on the task level in an episodic fashion [69, 37] with a large number of N -way K -shot training tasks. In this work, we consider more challenging few-shot adaptation tasks ( e.g., 100-way 1-shot in Cifar100 and 102-way 1-shot in Flowers102) compared to the commonly used 5-way 1-shot meta-learning tasks. We leverage the power of the pre-trained foundation models, which eliminates the need of task based episodic meta-training. From a meta-learning perspective, the pre-training phase for the foundation model could be viewed as performing meta-knowledge acquisition similar to meta-training. The pre-trained model can be seen as an expert equipped with the meta-knowledge, and parameter-efficient fine-tuning performs quick adaptation to the downstream tasks, analogous to the support-set based adaptation done in meta-testing.

3 Bayesian Parameter-Efficient Fine-Tuning of Foundation Models

We start by introducing some fundamental concepts from evidential learning, which will serve as key building blocks in the proposed Bayesian-PEFT framework. We then detail the two Bayesian components: base rate adjustment to address under-confidence and diversity-inducing evidential ensemble to improve the reliability on both prediction accuracy and uncertainty quantification.

3.1 Preliminaries

Evidential Deep Learning (EDL) [56] introduces a computationally efficient framework to transform deterministic deep learning (DL) models into uncertainty-aware models. The key idea is to introduce a higher-order conjugate prior distribution over the predicted likelihood distribution and train the DL model to output parameters of the higher-order distribution. Towards classification, EDL models

[56, 11] introduce Dirichlet prior distribution for the multinomial likelihood distribution. Specifically, the output softmax layer of the DL model is replaced by a monotonic, non-negative transformation function ( e.g., ReLU, SoftPlus, or exp ) to obtain the evidence for different classes that are transformed into the Dirichlet parameters. Mathematically, for a DL model f θ ( · ) , and an input sample x , we have

where e i is the output evidence for the i th class from the model f θ ( · ) and input sample x , a i is the base rate for the i th class, W is the non-informative prior weight usually set to the number of classes, E is the non-negative transformation function, and α i parameterizes a Dirichlet distribution. Existing EDL works usually adopt a non-informative base rate of a i = 1 N ∀ i ∈ [1 , N ] . Furthermore, a multinomial distribution Mult ( y | p ) over labels is parameterized as E [ p i ] = α i S , where the total Dirichlet Strength S = ∑ N α i .

An evidential model can be trained via a Type-II Maximum Likelihood-based evidential loss L Log ( x , y ) [56] with KL regularization that penalizes evidence assigned to non-ground-truth classes:

where ˜ α = y +( 1 -y ) ⊙ α . Once trained, the evidential model can predict an evidence vector e = ( e 1 , e 2 , ...e N ) ⊤ for a given test sample x . From the predicted evidence, we obtain the model's belief ( b ) over different classes, the correct belief b cor , incorrect belief b inc , and vacuity u as

where vacuity u is a second-order uncertainty [33] that captures the model's lack of knowledge in its prediction; b cor and b inc quantify model accuracy and error, respectively. However, neither b cor nor b inc can be evaluated without the ground-truth label, which is not available in the testing phase. Existing theoretical work has established an important connection between b inc and dissonance [46], which is another second-order uncertainty [33] that can be quantified without the ground-truth label. More specifically, dissonance dis can be evaluated as

̸

̸

where Bal ( · , · ) is the relative mass balance function between two belief masses. The dissonance essentially captures the conflicting belief assigned to different classes [57].

3.2 Strengthening the prior belief through base rate adjustment

To gain a deeper understanding of the under-confident few-shot adaptation behavior of foundation models, we perform fine-grained uncertainty analysis using the predicted evidence from an evidential model. To this end, we replace the softmax layer in a VPT fine-tuned transformer model with an exponential-based evidential head that outputs non-negative evidence for different classes. We analyze the output evidence from the evidential model that reveals some interesting insights.

Why is the model accurate? First, we observe that the relative order of the evidence assigned to different classes is accurate. This implies that the model outputs relatively greater evidence for the correct class compared to all other classes that ensure the model's strong predictive performance. To more precisely quantify the model's accuracy, we adapt the lower bound of incorrect belief theorem developed for meta-learning [46] to evaluate the model accuracy through its predicted dissonance:

Theorem 1. Consider an evidential model that outputs incorrect belief of b inc and the dissonance in the beliefs is dis . Then, the incorrect belief of the model will be at least half of the dissonance for all predictions from the evidential model.

Figure 3a shows the test accuracy vs. dissonance curve, which is aligned with the relationship between the incorrect belief and the dissonance given in the theorem above, where a low dissonance implies a low b inc (or a high accuracy). From all the testing samples, we observe relatively low dissonance and the highest is only slightly above 0.7 as shown in the figure. We further evaluate the Area Under the Curve of the Accuracy vs. (1 -dis ) and obtain an AUC of 0.82 as shown in Figure 3b. This implies the model is able to clearly discriminate the ground-truth label from the rest without much confusion ( i.e., low dissonance) which ensures its good prediction accuracy.

(a) Dissonance-Accuracy

(b) AUC Curve

0

20

40

60

80

100

Class ID

0

1

2

3

4

5

Evidence

Confidence: 0.025, Vacuity: 0.875

Ground Truth Class

(c) Evidence of classes

0.0

0.2

0.4

0.6

0.8

1.0

Vacuity

0

2000

4000

6000

8000

Number of Samples

Vacuity Count

(d) Vacuity Distribution

Why is the model under-confident? Despite being able to assign relatively more evidence to the correct label over the rest, it is also interesting to observe that the model generally assigns very low evidence to all the labels, including the correct one. Figure 3c shows the evidence distribution of one representative test data sample from Cifar100. As can be seen, most classes are assigned very low evidence that is close to zero. The ground-truth class is assigned higher evidence, but it is far from sufficient to make the prediction confident. The resultant confidence is only 0.025 while the vacuity is extremely high at 0.875, implying that the model believes it has very limited knowledge of the data sample despite it correctly identifying the correct label. Figure 3d shows the predicted vacuity over all the test samples, most of which are assigned a very high vacuity. This confirms the overly conservative behavior of the model, where the low confidence is primarily due to the insufficient allocation of evidence in its predictions. On the other hand, since the model is fairly accurate, it is reasonable to believe that the model underestimates the contribution of the rich prior knowledge gained through pre-training.

Base rate adjustment to strength the prior belief. The fine-grained uncertainty analysis through the lenses of evidential learning not only explains the good predictive performance of the few-shot adapted model through PEFT but also unveils the root cause for its under-confident behavior, which is under-estimation of the contribution from the prior knowledge to the downstream task. While the classical Bayes' theorem offers a principal idea to address the issue, which is to strengthen the prior belief, there is a lack of practical way to achieve this. To this end, we propose to leverage the base rate introduced by the subjective logic theory [33] as an effective vehicle to adjust the prior belief gained through pre-training. According to (1), adjusting the base rate has the effect of changing the Dirichlet parameter α , which will change the confidence for the prediction given by max i E [ p i ] . However, base rate adjustment needs to meet two key requirements: (1) the relative order of the Dirichlet parameters assigned to different classes should be preserved so that the predictive performance of the model remains unaffected, (2) the gap between the Dirichlet parameters for different classes is transformed such that the model becomes more confident in its predictions, making it well-calibrated. To meet these requirements, we we propose a transformation function A m to the model's output evidence such that the model is well calibrated without any compromise in the generalization performance:

where χ = ( χ 1 , χ 2 , ...χ N ) ⊤ is the adjusted base rate, and m ≥ 1 controls the base rate transformation. The adjusted base rate χ considers evidence of all classes as a reference via e min , and transforms the gap between different class evidences such that the model is well calibrated.

Lemma2. The base-rate adjusted model that uses learnable base rate χ = ( χ 1 , χ 2 , ..., χ N ) ⊤ has the same generalization performance compared to the model using fixed base rate of a i = 1 N ∀ i ∈ [1 , N ]

Theorem 3. For any m ≥ 1 , the transformation function A m transforms the base rate for the class with the highest evidence e max and class with the second highest evidence e 2nd such that the gap in Dirichlet parameters between the two classes is non-decreasing.

Remark. Theorem 3 ensures that the expected probability E [ p i ] for the predicted class i has an increased gap with the rest of the classes, which results in an increase of the model's confidence. Therefore, if the prediction is accurate, the model's calibration performance will be improved. Meanwhile, Lemma 2 ensures that the good prediction accuracy of the model is maintained by the proposed base rate adjustment strategy. The detailed proofs are given in Appendix D.

3.3 Building A Diversity Induced Evidential Ensemble

The second Bayesian component of the proposed B-PEFT framework aims to further improve the reliability of both prediction accuracy and uncertainty quantification when performing few-shot adap-

Dirichlet

Parameters

Dirichlet

Parameters

Dirichlet

Parameters

Base-rate

adjustment

Base-rate

adjustment

Base-rate

adjustment

Ensemble

Predictions

ECE

Input

Observed Variable

Latent Variable

(a)

(b)

Ensemble E

Ensemble 2

Ensemble 1

A

p(0|α)

Dir(α|0)

p

Mult(pla)

y

tation. It performs Bayesian model averaging by building a diversity-inducing evidential ensemble. The ensemble of deep learning models ( i.e., deep ensemble) [20, 36] can effectively improve the generalization performance of deep learning models. Moreover, deep ensembles can capture the model uncertainty [36, 53] via the agreement-disagreement between the ensemble components. Model uncertainty essentially captures the uncertainty in the model parameters, which is denoted as θ of the graphical model of B-PEFT as shown in Figure 4(b). The schematic diagram of B-PEFT is shown in Figure 4(a). The model uncertainty can be leveraged to evaluate the reliability of fine-grained uncertainty output by the evidential model.

The effectiveness of the ensembles has been empirically demonstrated across multiple datasets/settings [30] with theoretical guarantees [2]. However, standard deep ensembling leads to limited diversity among the ensemble components as it only considers the random initialization of components. We propose a novel diversity-inducing ensembling scheme for the evidential models. Similar to the deep ensemble [36], we also consider randomly initialized evidential models. We additionally train each ensemble component with different strengths for incorrect evidence regularization along with evidential loss objective. The overall objective for each ensemble component is identical to (2).

However, each ensemble component is trained with different incorrect evidence (or belief) regularization strengths ( i.e., different components place different priorities for the minimization of incorrect evidence over the maximization of correct evidence) which leads to diversity among the components. Since each component's priority for minimizing the incorrect evidence is different, the components focus on different attributes/features in the data that help the model avoid overfitting to an identical set of discriminative features. As a result, the proposed evidential ensembling scheme implicitly pushes the ensemble components away from each other, making it equivalent to the repulsive force in the Stein Variational Gradient Descent (SVGD) [12, 13].

Lemma 4. For given incorrect evidence regularization L inc reg , and E ensemble components with regularization strengths λ p , p ∈ [1 , P ] , the ensemble components in the evidence space are implicitly pushed away from each other by a force λ p ∇L inc reg that acts identical to the repulsive force in Stein Variational Gradient Descent (SVGD) based ensembles.

Remark. The detailed proof is given in the Appendix. We present an intuitive visualization of the update of the evidential ensemble model for different strengths ( λ 1 < λ 2 < ... < λ P ) of incorrect evidence regularization for different seeds in Figure 5. Each ensemble component aims to maximize the likelihood (direction - → A ) and minimize incorrect

C

Seedo

AB

B

AB

A

C2

Seed2

入2B

入2B

SeedE

B

ApB

入pB

Seed0

EnsembleComponents

evidence (direction - → B ). The strengths of incorrect evidence regularization (direction - → B ) are different for each ensemble component that acts as an implicit repulsive force among the ensemble components, ensuring that they are diverse from each other. Different from the SVGD-based ensemble, in our proposed model, the particles do not need to explicitly communicate with each other making our proposed approach computationally efficient, scalable, and generalizable.

4 Experiments and Results

Experiment setup, datasets, and baselines. We consider K -shot adaptation ( i.e., the dataset has K examples per class in the training set) with Cifar10 [1], Cifar100 [1], Food101 [8], and Flowers102 [45] datasets. For instance, the 2 -shot Cifar100 dataset has 2 examples per class leading to a total of 200 labeled training samples. For all datasets and experiments, the training set is a few-shot dataset, and the evaluation is done on the standard test set available with benchmark datasets. Details of the few-shot training datasets along with additional experiment details are presented in the Appendix E. We consider large pre-trained vision transformer with ViT backbone [15] and consider Visual Prompt Tuning (VPT) [32], along with bias fine-tuning [9] and adapter fine-tuning [72] as the PEFT techniques (We use VPT as the representative PEFT where not specified due to its superior performance). We consider accuracy-preserving post-hoc calibration techniques including Temperature Scaling (TS) [26], Parameterized Temperature Scaling (PTS) [61], and Isotonic Regression (IR-MC) [6] as the baseline calibration techniques.

Prediction and calibration performance. We first consider standard cross-entropy (CE)-based PEFT of the supervised pre-trained ViT model on few-shot datasets. We present the accuracy and calibration results of VPT in Table 1 (a). We observe that the straightforward adaption of the models leads to accurate but under-confident models as indicated by a high ECE. The evidential models as shown in Table 1 (b) have comparable or better generalization performance across the datasets/settings. However, these models are also severely under-confident similar to CE-based models indicated by high ECE and accuracy-confidence trends (see Figure 11a, 11b in the Appendix). The overall performance of the calibrated evidential model using base-rate adjustment is presented in Table 1 (c). As can be seen, the accuracy remains the same as the adjusted base rate expands the gap between evidence of the class and preserves the relative order in the predicted class evidence. It effectively tackles the under-confidence issue, which leads to a significant improvement in the overall ECE performance across the datasets and settings (also see Figure 11c in the Appendix). Table 1 (d) shows

the results of the proposed B-PEFT model that introduces a diversity-enforcing ensemble of calibrated evidential models trained with different strengths of incorrect evidence regularization. performance.

Such a learning signal helps the model avoid overfitting (addressing the potential overconfidence issue) and leads to further improvement in generalization and the calibration.

We further carry out experiments with additional PEFT techniques of bias and adapter fine-tuning and on the K -shot Cifar100 dataset in Table 2. We observe that these PEFT techniques lead to a lower generalization performance (as indicated by a lower test accuracy) compared to the VPT-based technique. Nevertheless, the same underconfidence issue remains as shown in Table 2 (a) for standard cross-entropy trained model performance, and in Table 2 (b) for their evidential extensions. We then carry out experiments using base-rate adjusted evidential model and our proposed B-PEFT model. The results are shown in Table 2 (c) and Table 2 (d). Base-rate adjusted evidential model and B-PEFT are equally effective with these PEFT techniques, and they address the underconfidence issue, which leads to improvement in both generalization and calibration.

Comparison with existing calibration techniques. We compare our proposed evidential calibra tion technique with existing calibration techniques using a representative Cifar100 dataset on different few-shot classification settings. We consider Isotonic Regression (IRMC)[6], Temperature Scaling (TS) [26], and Parameterized Temperature Scaling (PTS) [61] as baselines. Overall results are presented in Table 3. The baseline model and its evidential extension are poorly calibrated. All calibration

techniques transform the logits to address the under-confidence issue of the model. Our proposed Base-rate adjusted Evidential model (BR-Evid) addresses the under-confidence issue in the evidential model leading to significant improvement in the ECE. B-PEFT model further improves on the BREvid model as it effectively addresses both the under-confidence issue (via the base rate adjustment) and overconfidence issue (via the Bayesian ensembling) that leads to superior calibration results as indicated by the lowest ECE in all few-shot settings.

Uncertainty quantification results. We investigate the uncertainty quantification performance of our proposed calibrated evidential ensemble model. The developed model can reflect the model uncertainty ( i.e., the model's confidence in its predictions) through the ensemble agreement/disagreement and the distributional uncertainty through the sharpness of evidential prior distribution ( i.e., the vacuity). For the analysis, we use Cifar10 as in-distribution (ID) dataset for different shots, and Cifar100 as OOD dataset. As shown in Figure 6, we plot vacuity and variance distribution for 1 and 5 shots. A single model outputs the vacuity that can be used to detect OOD samples. We can see ID samples are in the lower vacuity region and OOD samples are in the higher vacuity region. As the number of shots increases, the region becomes more separate. However, for 1 shot, there are some OOD samples in the lower vacuity region as well. Since we can not trust vacuity alone for uncertainty, we utilize the variance of ensemble components to quantify model uncertainty. As we see in Figure 6(c), the variance of OOD samples mostly lies in the high variance region. As we increase the number of shots, the variance shifts towards the lower region. High variance indicates that model-predicted vacuity can not be totally trusted. This behavior is qualitatively shown in Figure

0.000

0.200

0.400

0.600

0.800

1.000

Vacuity

0

1000

2000

3000

4000

5000

Number of Samples

1 shot In-Distribution:Cifar10

In Distribution: Correct

In-Distribution: Incorrect

OOD: Cifar100

(a) Vacuity for 1 shot

0.000

0.200

0.400

0.600

0.800

1.000

Vacuity

0

1000

2000

3000

4000

5000

6000

7000

8000

Number of Samples

5 shot In-Distribution:Cifar10

In Distribution: Correct

In-Distribution: Incorrect

OOD: Cifar100

(b) Vacuity for 5 shot

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0

1000

2000

3000

4000

5000

6000

Number of Samples

1 shot In-Distribution:Cifar10

In Distribution: Correct

In-Distribution: Incorrect

OOD: Cifar100

0.000

0.010

0.020

0.030

0.040

0

1000

2000

3000

4000

5000

6000

7000

8000

Number of Samples

5 shot In-Distribution:Cifar10

In Distribution: Correct

In-Distribution: Incorrect

OOD: Cifar100

Component 1 vacuity:  0.379

Component 2 vacuity:  0.811

Component 3 vacuity:  0.587

Ensemble Variance:  0.013

Component 1 vacuity:  0.308

Component 2 vacuity:  0.780

Component 3 vacuity:  0.601

Ensemble Variance:  0.018

Component 1 vacuity:  0.857

Component 2 vacuity:  0.882

Component 3 vacuity:  0.868

Ensemble Variance:  0.002

(a) High Variance

(b) High Variance

7. We observe that when only a single component outputs a low vacuity for the OOD samples in Figure 7 (a-b), the variance of the ensemble is high, implying a high model uncertainty. When all the components output a high vacuity to the OOD sample in Figure 7 (c), the variance is also low, implying a low model uncertainty.

0.5

1.0

1.5

2.0

2.5

3.0

m

1 shot Cifar100

kl=0.0

kl=1.0

kl=10.0

kl=100.0

Figure 8: Impact of

0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 ECE Ablation study. We carry out ablation with 1 -shot Cifar100 dataset to study the impact of the base rate transformation order m (Section 3.2) for different strengths of incorrect evidence regularization on the calibration performance. As we increase m , the probability gap between classes improves, leading to more confident predictions. However, the model starts to become overconfident for large m values (see Figure 8).Moreover, with an increase in incorrect evidence regularization strength, the optimal m value decreases to a smaller value ( e.g., optimal m = 2.0 for λ = 0 . 1 , and optimal m = 1 . 5 for λ = 10). Choosing a proper m leads to the best-calibrated evidential model. We present additional results studying the impact of m in Appendix F.1.

m

Limited by space, we provide results comparing meta-learning methods on standard few-shot tasks in Appendix F.4, discuss impact of various components (number of classes, data size, and unfrozen parameters) in Appendix F.5, and include additional experiments and comparisons, including OOD settings, in Appendix F.6. Moreover, we carry out additional ablation experiments to study the impact of incorrect evidence regularization strength (Appendix F.2), and the impact of ensemble components (Appendix F.3). We further carry out experiments and discuss applying PEFT to foundation models pre-trained in a self-supervised fashion (Appendix G). We discuss the societal impact and limitations of the work in Appendices H and I, respectively.

5 Conclusion

In this work, we focus on transformer-based large vision foundation models and investigate different parameter-efficient fine-tuning techniques for effective few-shot adaptation. We observe that the existing models are severely under-confident, especially in challenging datasets and settings. Moreover, existing models lack fine-grained uncertainty quantification capabilities. We extend the models to uncertainty-aware evidential models, and resort to the evidential framework to develop a novel Bayesian parameter efficient fine-tuning ( B-PEFT ) framework that integrates evidence-based base rate adjustment to addresses the under-confidence and a diversity inducing evidential ensemble technique to further improve the reliability in model prediction and uncertainty quantification. The B-PEFT framework possesses theoretically sound properties to ensure its superior generalization capability and robust calibration behavior. We carry out intensive experiments across different benchmark datasets and diverse few-shot settings that demonstrate the outstanding performance of B-PEFT .

Acknowledgments

This research was supported in part by an NSF IIS award IIS-1814450. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agency. We would like to thank the anonymous reviewers for their constructive comments.

References

Krizhevsky Alex. Learning multiple layers of features from tiny images. https://www. cs. toronto. edu/kriz/learning-features-2009-TR. pdf , 2009.

Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 , 2020.

Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems , 33:14927-14937, 2020.

Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06470 , 2020.

Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1334913358, 2021.

Eugene Berta, Francis Bach, and Michael Jordan. Classifier calibration with roc-regularized isotonic regression. In International Conference on Artificial Intelligence and Statistics , pages 1972-1980. PMLR, 2024.

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning , pages 1613-1622. PMLR, 2015.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision , 2014.

Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems , 33:1128511297, 2020.

Bertrand Charpentier, Oliver Borchert, Daniel Zügner, Simon Geisler, and Stephan Günnemann. Natural posterior network: Deep bayesian predictive uncertainty for exponential family distributions. In International Conference on Learning Representations , 2022.

Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts. Advances in Neural Information Processing Systems , 33:1356-1367, 2020.

Francesco D'Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems , 34:3451-3465, 2021.

Francesco D'Angelo, Vincent Fortuin, and Florian Wenzel. On stein variational neural network ensembles. arXiv preprint arXiv:2106.10760 , 2021.

Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems , 34:20089-20103, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1126-1135. JMLR. org, 2017.

Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems , pages 9516-9527, 2018.

Gianni Franchi, Olivier Laurent, Maxence Leguéry, Andrei Bursuc, Andrea Pilzer, and Angela Yao. Make me a bnn: A simple strategy for estimating bayesian uncertainty from pre-trained models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12194-12204, 2024.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

Mudasir A Ganaie, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthurai N Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence , 115:105151, 2022.

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision , 132(2):581-595, 2024.

Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4367-4375, 2018.

Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta-learning probabilistic inference for prediction. arXiv preprint arXiv:1805.09921 , 2018.

Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. In International Conference on Learning Representations , 2018.

Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332 , 2021.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. Eˆ 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint arXiv:2307.13770 , 2023.

Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang, Siyuan Qi, and Dongfang Liu. Facing the elephant in the room: Visual prompt tuning or full finetuning? arXiv preprint arXiv:2401.12902 , 2024.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get m for free. arXiv preprint arXiv:1704.00109 , 2017.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision , pages 709-727. Springer, 2022.

Audun Jøsang. Subjective logic , volume 3. Springer, 2016.

Audun Josang, Jin-Hee Cho, and Feng Chen. Uncertainty characteristics of subjective opinions. In 2018 21st International Conference on Information Fusion (FUSION) , pages 1998-2005. IEEE, 2018.

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision , 128(7):1956-1981, 2020.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Hae Beom Lee, Hayeon Lee, Donghyun Na, Saehoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-ofdistribution tasks. In Eighth International Conference on Learning Representations, ICLR 2020 . ICLR, 2020.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 , 2021.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems , 31, 2018.

Aryan Mobiny, Pengyu Yuan, Supratik K Moulik, Naveen Garg, Carol C Wu, and Hien Van Nguyen. Dropconnect is effective in modeling uncertainty of bayesian deep networks. Scientific reports , 11(1):5458, 2021.

Azadeh Sadat Mozafari, Hugo Siqueira Gomes, Wilson Leão, Steeven Janny, and Christian Gagné. Attended temperature scaling: a practical approach for calibrating deep neural networks. arXiv preprint arXiv:1810.11586 , 2018.

Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems , 32, 2019.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing , pages 722-729. IEEE, 2008.

Deep Shankar Pandey and Qi Yu. Multidimensional belief quantification for label-efficient meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14391-14400, June 2022.

Deep Shankar Pandey and Qi Yu. Evidential conditional neural processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 9389-9397, 2023.

Deep Shankar Pandey and Qi Yu. Learn to accumulate evidence from all training samples: Theory and practice. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 26963-26989. PMLR, 23-29 Jul 2023.

Tim Pearce, Felix Leibfried, and Alexandra Brintrup. Uncertainty in neural networks: Approximately bayesian ensembling. In International conference on artificial intelligence and statistics , pages 234-244. PMLR, 2020.

Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548 , 2017.

Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vuli´ c, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779 , 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Rahul Rahaman et al. Uncertainty quantification and deep ensembles. Advances in Neural Information Processing Systems , 34:20063-20075, 2021.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems , 30, 2017.

Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972 , 2021.

Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems , 31, 2018.

Weishi Shi, Xujiang Zhao, Feng Chen, and Qi Yu. Multifaceted uncertainty estimation for label-efficient deep learning. Advances in neural information processing systems , 33, 2020.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Zhao Song, Ke Yang, Naiyang Guan, Junjie Zhu, Peng Qiao, and Qingyong Hu. Vppt: Visual pre-trained prompt tuning framework for few-shot image classification. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, 2023.

Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Christian Tomani, Daniel Cremers, and Florian Buettner. Parameterized temperature scaling for boosting the expressive power in post-hoc uncertainty calibration. In In European Conference on Computer Vision (ECCV) , 2022.

Dennis Ulmer. A survey on evidential deep learning for single-pass uncertainty estimation. arXiv preprint arXiv:2110.03051 , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems , 29, 2016.

Cheng Wang. Calibration in deep learning: A survey of the state-of-the-art. arXiv preprint arXiv:2308.01222 , 2023.

Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research , 20(4):447-482, 2023.

Jingyuan Wen, Yutian Luo, Nanyi Fei, Guoxing Yang, Zhiwu Lu, Hao Jiang, Jie Jiang, and Zhao Cao. Visual prompt tuning for few-shot text classification. In Proceedings of the 29th International Conference on Computational Linguistics , pages 5560-5570, 2022.

Liqi Yan, Cheng Han, Zenglin Xu, Dongfang Liu, and Qifan Wang. Prompt learns prompt: exploring knowledge-aware generative prompt collaboration for video captioning. In Proceedings of international joint conference on artificial intelligence (IJCAI) , pages 1622-1630, 2023.

Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems , pages 7332-7342, 2018.

Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen, Prashanth G Shivakumar, Yile Gu, Sungho Ryu Roger Ren, Qi Luo, Aditya Gourav, I-Fan Chen, Yi-Chieh Liu, et al. Low-rank adaptation of large language model rescoring for parameter-efficient speech recognition. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , pages 1-8. IEEE, 2023.

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199 , 2021.

Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 698-714. Springer, 2020.

Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference on machine learning , pages 11117-11128. PMLR, 2020.

Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. In International Conference on Machine Learning , pages 26135-26160. PMLR, 2022.

Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems , 33:12827-12836, 2020.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16816-16825, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Supplementary Material

Appendix

Table of Contents

A Organization of the Appendix

In Section B, we present the table with summary of the key symbols used in this work.

In Section C, we present related works in uncertainty-aware deep learning, describe evidential deep learning for classification in details, and discuss some key advantages of using evidential models to address the under-confidence issue over standard Bayesian models.

In Section D, we present proof of all our theoretical claims.

In Section E, we present the details of hyperparameters and additional experiment details.

In Section F, we present additional experiment results including the impact of m for calibration performance, comparison results with meta-learning methods, the impact of the number of classes, data size, and unfrozen parameters, comparison in OOD settings, the impact of incorrect evidence regularization strength, the results of additional PEFT methods, and the impact of ensemble components.

In Section G, we discuss the calibration and accuracy behavior for PEFT of large foundation models pre-trained in a self-supervised fashion.

In Section H, we discuss societal impact of our work.

In Section I, we discuss the limitations of our work and present potential future direction.

The source code for the experiments carried out in this work is attached in the supplementary materials and is available at the link: https://github.com/ritmininglab/B-PEFT

B Summary of the Symbols

C Further Discussion on Uncertainty-Aware Deep Learning

C.1 Existing Uncertainty Quantification Methods in Deep Learning

Accurate quantification of predictive uncertainty is essential for the development of trustworthy Deep Learning (DL) models. To this end, DL models have been augmented to become uncertaintyaware using a variety of approaches such as ensemble-based approaches [36, 49], bayesian neural networks based approaches [42, 19, 7], and deterministic neural network based approaches [56, 11, 3]. Deep ensemble techniques [36, 49] construct an ensemble of neural networks, and the agreement/disagreement across the ensemble components is used to quantify different uncertainties. Alternatively, Bayesian neural networks [19][7][42] have been developed that consider a Bayesian formalism ( e.g., bayes-by-backprop [7], dropout during test [19]) to quantify different uncertainties.

ABNN [18] introduces Bayesian normalization layers after training of deep learning models, and requires additional training of these layers in a post-hoc manner. Deterministic neural network-based approaches [57, 41, 10] extend the existing neural network to become uncertainty-aware and enable the networks to quantify fine-grained uncertainties with a single forward pass through the network. Evidential deep learning models [56, 5, 75, 10, 10, 62], an instance of deterministic approaches, introduce a conjugate higher-order evidential prior for the likelihood distribution to enable the model to express the fine-grained uncertainties in both classification[56, 11] and regression problems [3, 47]. Towards classification, evidential models [56, 5, 75] introduce higher-order evidential Dirichlet prior to the multinomial likelihood that enables the deterministic neural network model to capture different uncertainty characteristics. In what follows, we first provide some additional details on using evidential learning model to perform classification. We then highlight some important advantage of using evidential models over standard Bayesian models in uncertainty quantification.

C.2 Evidential Deep Learning Models for Classification

Evidential Deep Learning models, based on Subjective Logic theory [33], aim to train the model such that for any new input sample, the model can make predictions, as well as output fine-grained uncertainty information (via vacuity [56] and dissonance [34]). Towards capturing fine-grained uncertainty for classification problems, EDL models assume that the label for each sample is obtained from a generative process with a Dirichlet prior and a multinomial likelihood. The parameters for the Dirichlet prior express the vacuity and belief masses for uncertainty estimation. The conjugacy between the Dirichlet prior and the multinomial likelihood is explored, and different evidential losses are introduced for model training and inference [48]. In this work, we consider Type-II Maximum Likelihood-based evidential loss L Log ( x , y ) [56] with incorrect evidence regularization L inc reg ( x , y ) given by [56]

Wereplace the softmax layer in the head of the VPT model with exp activation function. To avoid zero evidence regions , we also include the correct evidence regularization L cor ( x , y ) = -λ cor log( α gt -1) ( λ cor is the magnitude of the model's vacuity) in model training objective. The evidential model outputs evidence vector e = ( e 1 , e 2 , ...e N ) for a given input x and corresponding ground truth label of y . Based on the evidence, Dirichlet parameters are obtained as α i = e i +1 . The Type-II Maximum likelihood-based evidential loss is given by

The incorrect evidence regularization guides the model to minimize the evidence for all classes other than the ground truth class and can take one of the following forms

KL-based incorrect evidence regularization term as in EDL [56]

̸

Where ˜ α = y + ( 1 -y ) ⊙ α = (˜ α 1 , ˜ α 2 , ... ˜ α N ) parameterize a dirichlet distribution, ˜ α i = gt = 1 , ˜ α i = α i ∀ i = gt , and ⊙ represents element-wise product. Here, the KL regularization term encourages the Dirichlet distribution based on the incorrect evidence i.e., Dir ( p | ˜ α ) to be flat which is possible when there is no incorrect evidence.

Incorrect evidence sum based regularization as in ADL [57]

Incorrect belief-sum based regularization as in Units-ML [46]

All three regularizations guide the model to minimize the incorrect evidence(ideally close to zero). In our experiments, we consider KL-based incorrect evidence regularization.

C.3 Evidential models vs. Standard Bayesian Models

As compared with the Bayesian-inspired models, evidential learning offers two key properties that allow us to formulate a principled solution to address the unique under-confident behavior of the PEFT methods. First, thanks to its evidence-based fine-grained uncertainty decomposition capability, we can separate two distinct sources of second-order uncertainty, including vacuity and dissonance. Different from the commonly used first-order uncertainty (e.g., entropy), these two second-order uncertainty serve as a key tool to understand why PEFT methods are both accurate (with a low dissonance) while being under-confident (with a high vacuity). This key insight suggests that these methods systematically under-estimate the contribution from the prior knowledge to the downstream task. While the classical Bayes' theorem offers a principal idea to address the issue, which is to strengthen the prior belief, there is a lack of practical way to achieve this. As the second key property, evidential learning allows us to leverage the base rate, which is rooted in the subjective logic theory as an effective vehicle to adjust the prior belief gained through pre-training. To this end, we propose a transformation function in Eq. (6) to adjust the base rate that leads to the increase of the model confidence while maintaining the predictive accuracy of the model as guaranteed by our theoretical results in Lemma 2 and Theorem 3. Furthermore, we develop belief-based diversity for ensemble of evidential models leading to the the B-PEFT model. In theory, evidential deep learning model could be augmented with the Bayesian normalization layers [18] or Bayesian neural networks [7] as an alternative to belief-based diversity of B-PEFT . We leave exploration of different techniques for diversity for Bayesian evidential model as a potential future work.

D Proofs of Theoretical Results

In this section, we provide the proofs of the major theoretical results presented in the main paper.

D.1 Proof of Lemma 2

Proof. Consider an input sample x for which the model outputs the evidence ( e 1 , e 2 , ..., e N ) ⊤ . Let e max = max( e 1 , e 2 , ..., e N ) , and e min = min( e 1 , e 2 , ..., e N ) . Here, e max ≥ e i ≥ e min ∀ i ∈ [1 , N ] . For the evidential model with fixed base rate of a i = 1 N ∀ i ∈ [1 , N ] , the model's predicted class is given by c pred = arg max( e 1 + a 1 × W,e 2 + a 2 × W,..., e N + a N × W ) = arg max( e 1 +1 , e 2 + 1 , ..., e N +1) = Index ( e max ) . For the calibrated model with learnable χ = ( χ 1 , χ 2 , ..., χ N ) ⊤ , the model's predicted class is given by c new pred = arg max( α 1 , α 2 , ..., α N ) = arg max( e 1 + χ 1 × W,e 2 + χ 2 × W,..., e N + χ N × W ) = arg max( e 1 + N ( e 1 e min ) -N,e 2 + N ( e 2 e min ) -N,..., e N + N ( e N e min ) -N ) . Since e max ≥ e i ≥ e min ∀ i ∈ [1 , N ] , α max ≥ α i ≥ α min ∀ i ∈ [1 , N ] , and c new pred = c pred

D.2 Proof of Theorem 3

Proof. Consider an input sample x for which the model outputs the evidence ( e 1 , e 2 , ..., e N ) ⊤ . Let e max = max( e 1 , e 2 , ...e N ) , e min = min( e 1 , e 2 , ...e N ) , and e max ≥ e 2nd ≥ , ..., ≥ e min . For the evidential model with a fixed base rate of a i = 1 N ∀ i ∈ [1 , N ] , the difference between the Dirichlet parameters for class with maximum evidence and class with the second maximum evidence is given by α max -α 2nd = e max + a max W -e 2nd -a 2nd W = e max -e 2nd as a i = 1 N ∀ i ∈ [1 , N ] .

For the calibrated model with learnable χ = ( χ 1 , χ 2 , ..., χ N ) , the difference between the Dirichlet parameters for class with maximum evidence and class with the second maximum evidence is given by α max -α 2nd = e max + χ max W -e 2nd -χ 2nd W = ( e max -e 2nd ) + ( χ max -χ 2nd ) W . Now,

Since e i e min ≥ 1 ∀ i ∈ [1 , N ] , and e max ≥ e 2nd ≥ , ..., ≥ e min , ( χ max -χ 2nd ) ≥ 0 ∀ m > 0 . For e max > e 2nd , & m> 0 , χ max -χ 2nd > 0 . Thus, with the proposed learnable base rate, the gap between

the two largest Dirichlet parameters is maintained whenever m = 0 and/or e max = e 2nd . Moreover, whenever m ≥ 1 and e max > e 2nd , the Dirichlet parameter gap between the two classes is increased by a factor of ( e max e min -1 ) m -( e 2nd e min -1 ) m .

D.3 Connection with SVGD-based Bayesian Ensembling and Proof of Lemma 4

We first carry out an analysis of the update in Stein Variational Gradient Descent (SVGD) based ensembling [12, 13] that reveals the repulsive force acting among the ensemble components that pushes the particles away and introduces diversity. We then consider ensemble components with different strengths of incorrect evidence regularization L inc reg and analyze the update to the evidential model in the evidence space that reveals a repulsive diversity-enforcing force acting identical to the SVGD based ensemble.

SVGD update involves randomly initializing the particles and iteratively updating the particles to match the target distribution, which is summarized below.

Algorithm 1 SVGD Update

Input: { x 0 i } N i =1 : A set of initial parameters, and target distribution density function p ( x ) For L iterations , iteratively update the particles as

Here, ϵ l is the step size at iteration l , and k ( · , · ) is the kernel function that measures similarity. Output: { x 0 i } N i =1 : A set of initial parameters, and target distribution density function p ( x )

For given incorrect evidence regularization L inc reg , and P ensemble components with regularization strengths λ p , p ∈ [1 , P ] , the ensemble components in the evidence space are implicitly pushed away from each other by a force λ p ∇L inc reg that acts identical to the repulsive force in Stein Variational Gradient Descent (SVGD) based ensembles.

Proof. For simplicity, consider RBF kernel for k ( · , · ) i.e., k ( a, b ) = exp ( -1 h ( a -b ) 2 ) , two particles x 1 , x 2 , and analyze their updates in SVGD based ensembling. At iteration l , the update to particle x 2 is given by

In the above update, - → Q = k ( x l 2 , x l 1 ) ∇ x l 1 log p ( x l 1 ) + k ( x l 2 , x l 2 ) ∇ x l 2 log p ( x l 2 ) aims to guide the particles in the direction that maximizes the likelihood, and the update direction - → R = ∇ x l 1 k ( x l 1 , x l 2 )+ ∇ x l 2 k ( x l 1 , x l 2 ) acts as the repulsive force. Considering the repulsive force

As can be seen, the repulsive force - → R pushes the particle x l 2 in the direction away from particle x l 1 that introduces diversity. With more particles, each particle is updated in the direction that maximizes the likelihood, and the particle is pushed away from all other particles (by force R ).

Next, consider ensemble components with different strengths of incorrect evidence regularization L inc reg to analyze the update to the evidential model in the evidence space. For simplicity, we consider the incorrect evidence sum-based regularization similar to ADL without correct evidence regularization (The analysis is valid for all incorrect evidence regularization and for models with correct evidence regularization). For a model with incorrect evidence regularization, the overall evidential loss is given by:

The gradient of the loss with respect to logits (the output head layer, where e k = exp( o k ) ) is given by

Consider K class classification problem. The gradient update to the logit layer for the evidential model is given by

Here, y k ∈ [0 , 1] , y k = 1 if k = gt , and y k = 0 otherwise. Moreover, the λ value is varied, and different λ values lead to different evidential models.

The update force - → A pushes the evidential model in the direction that maximizes the likelihood (similar to - → Q in SVGD-based update), and the force - → B implicitly pushes the ensemble components away from each other (similar to the repulsive force - → R in SVGD-based update). Each component moves in - → B direction with a different force determined by the incorrect evidence regularization strength λ that ensures that the ensemble components are diverse. Due to the different strengths of incorrect evidence regularization, each ensemble component places a different priority level for minimization of incorrect evidence over acquiring correct evidence, which ensures that the ensemble components remain diverse.

E Dataset and Implementation Details

We consider ViT model backbone [15] that is pre-trained in a supervised fashion, and 4 benchmark datasets of Cifar10 [1], Cifar100 [1], Food101 [8], and Flowers102 [45]. We consider few-shot adaptation with K -shot classification problem (We experiment with K values of 1 , 2 , 5 , 10 , and 20 ). The few-shot training set is constructed by randomly selecting K samples per class from the training set of the benchmark datasets. We consider 2 -shot validation set for all datasets and settings. We train the model on the few-shot training set, use the 2 -shot validation set for hyperparameter tuning, and evaluate all models on the benchmark test set with all the test set samples. We augment the few-shot training set and the few-shot validation set with resize, random horizontal flip, and cropping. The dataset details are also presented in Table 5. We train all the models for 50 epochs on the few-shot training dataset with a batch size of 64 samples at each iteration and evaluate the model on the benchmark test set. The evidence is in the range [0, infinity], and some stability issues could potentially arise in extreme cases when the logit output is extremely low (i.e. close to negative infinity). In our experiments, we did not observe the stability issue. Still, the issue can arise in some extreme cases for which a small delta in the denominator could be introduced or the network's logits could be bounded to be greater than a small negative value. For the calibration baseline model of Parameterized Temperature Scaling (PTS) [61], we consider a 2-layer neural network with 128 nodes in the hidden layer (we carry out hyperparameter tuning with 1-layer, 2-layer, and 3-layer networks and select the best model), and train with a learning rate of 0.00001. For Temperature Scaling [26], we optimize for the temperature hyperparameter using Adam optimizer, and a learning rate of 0.01 (We also experiment with SGD optimizer, and learning rates of 0.1, 1.0, 0.01, and 0.0001 to select the best performing model). The evidential models use incorrect evidence regularization strength of ( 0 , 0 . 1 , 1 . 0 , 10 . 0 , 100 . 0 , and 1000 . 0 ). For Isotonic Regression [6], we consider multi-class setting and sklearn package. We use VPT [32] as the representative PEFT where not specified due to its superior performance. The key model performance results are averaged across 5 different runs to present the mean and the standard deviation. The experiments use Pytorch and are carried out on a workstation with NVIDIA RTX A6000 GPU.

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 49.155, ECE: 0.272

(a) 1 Shot, KL = 0.1, m = 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 49.155, ECE: 0.103

(b) 1 Shot, KL = 0.1, m = 1.5

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 49.155, ECE: 0.070

(c) 1 Shot, KL = 0.1, m = 2.0

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 50.905, ECE: 0.214

(d) 1 Shot, KL = 100.0, m =

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 50.905, ECE: 0.083

(e) 1 Shot, KL = 100.0, m =

1.5

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 50.905, ECE: 0.124

(f) 1 Shot, KL = 100.0, m =

2.0

F Additional Experiments

F.1 Impact of m on Expected Calibration Error

The developed B-PEFT model introduces the post-hoc calibration technique that adjusts the base rate in the evidential model with one additional hyperparameter m (see Section 3.2 for details). We carry out a grid search using the few-shot validation dataset and select the optimal m for the evidential models. In this section, we carry out detailed ablation to study the impact of m with few-shot Cifar100 datasets. We consider m values of (0 . 5 , 1 . 0 , 1 . 5 , 2 . 0 , 2 . 5 , 3 . 0) , incorrect evidence regularization strengths of (0 . 0 , 0 . 1 , 1 . 0 , 10 . 0 , 100 . 0 , 1000 . 0) , and few-shot values of (1 , 2 , 5 , 10 , 20) (see Table 6 and Figure 9, Figure 10). Across all the experiments, we observe that as we increase m , the model transforms the evidence to maximize the evidence gap making the model more confident on its knowledge. With increased confidence, the model's calibration performance improves. However, a large increase in m values starts to make the model overconfident leading to increased ECE and poor calibration. The strength of incorrect evidence regularization also impacts the optimal value of m . For large values of incorrect evidence regularization, a smaller m value suffices to make the model well-calibrated. The trend is seen across all few-shot classification settings.

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 71.497, ECE: 0.346

(a) 5 Shot, KL = 0.1, m = 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 71.497, ECE: 0.108

(b) 5 Shot, KL = 0.1, m = 1.5

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 71.497, ECE: 0.041

(c) 5 Shot, KL = 0.1, m = 2.0

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 74.147, ECE: 0.357

(d) 5 Shot, KL = 100.0, m =

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 74.147, ECE: 0.116

(e) 5 Shot, KL = 100.0, m =

1.5

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 74.147, ECE: 0.041

(f) 1 Shot, KL = 100.0, m =

2.0

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 49.845, ECE: 0.406

(a) Standard CE model

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 50.965, ECE: 0.494

(b) Evidential Model

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 50.965, ECE: 0.079

(c) Calibrated Evidential

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 52.825, ECE: 0.068

(d)

B-PEFT

(Ours)

F.2 Impact of Incorrect Evidence Regularization Strength ( λ )

Evidential deep learning models introduce incorrect evidence regularization to minimize the evidence of classes other than the ground truth class. In this work, we use KL divergence-based incorrect evidence regularization (see Section C.2), and introduce a hyperparameter λ that controls the priority the model places on minimizing the incorrect class evidence over maximizing the correct class evidence. In this section, we study the impact of the hyperparameter λ on the model performance with K -shot Cifar100 and Flowers102 experiments (We experiment with K values of (1 , 2 , 5 , 10 , 20) ). We observe that the model's calibration performance (the ECE) is optimal when no incorrect evidence regularization is used i.e., λ = 0 (see Table 7 and Table 8). However, no incorrect evidence regularization hurts the model's generalization performance. With the increase in incorrect evidence regularization, the model's generalization performance (indicated by accuracy) improves albeit with ECE tradeoff. However, very large incorrect evidence regularization misguides the model to only focus on minimizing incorrect class evidence hurting generalization. The optimal λ value leads to the best generalization performance while hurting the ECE performance. Moreover, with a larger number of shots in training, the optimal λ value is generally smaller (For instance, in 1 -shot Cifar100, optimal λ = 1000 . 0 , in 2 -shot Cifar100, optimal λ = 10 . 0 , and in 5 -shot Cifar100, optimal λ = 0 . 1 ). Once the optimal λ value is determined, we can resort to our evidential base-rate adjustment that leads to a calibrated evidential model with good generalization performance.

F.3 Impact of Ensemble Components

We present the experiment to study the impact of ensemble components in Figure 12. The experiment is performed on 1 one-shot Cifar100 dataset using prompt-based adaption for vision transformer. There is a performance gain of both accuracy and ECE with the use of all ensemble components. We also note that as we increase the number of components from 1 to 3 , both the generalization and calibration performance increase significantly. For instance, the ECE with a single ensemble component is 0.088 which improves to 0.077 with 3 ensemble components while the accuracy improves by almost 3%.. However, with a further increase in the number of ensemble components, the generalization/calibration performance

49

50

51

52

0.078

0.080

0.082

0.084

0.086

0.088

ECE

Ensemble Components

1 components

2 components

3 components

4 components

5 components

improvement is not significant. Thus, for our B-PEFT model, we carry out an ensemble of 3 evidential models that is a good balance between the number of ensemble components and the gain in generalization/calibration performance.

F.4 Few Shot Learning Results

In this set of experiments, we apply our model to the 5 -way 1 -shot mini-ImageNet testing tasks and compare it with representative meta-learning models. The results are summarized in Table 9. Benefiting from the knowledge acquired during pre-training, the proposed PEFT-based model outperforms the episodic meta-learning models by a large margin, demonstrating the potential of large vision foundation models for effective few-shot learning. However, the VPT model is underconfident as shown in Table 10 and Figure 13. Our B-PEFT model improves on the VPT model's generalization and calibration leading to promising few-shot adaptation results.

F.5 Impact of Different Components

In this section, we study the under-confidence behavior of PEFT methods w.r.t. data size, number of classes, and number of unfrozen parameters.

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Data size. We observe that the model's accuracy increases with more training samples (see Table 1 where we vary shots from 1 to 20). The under-confidence remains, even with further increase in training samples. To this end, we conduct additional experiments on Cifar100 by increasing the training samples per class to 500 and observe the under-confidence issue despite the increase in the accuracy. The trend is summarized in Figure 15 (a-b). We see an increase in accuracy and a decrease in ECE. However, even with 500 samples per class, the under-confidence issue remains. Further, we report the accuracy and ECE of the fully fine-tuned model (fine-tuning of all the parameters) for 1 shot cifar100 in Table 11 where we observe a decrease in accuracy while the calibration issue remains. We observe that full fine-tuning leads to overconfidence behavior, hurting the generalization performance, as seen in Table 11 and reliability plot as presented in Figure 14.

0

100

200

300

400

500

Number of Shots

50

55

60

65

70

75

80

85

(a) Data size trend

0

100

200

300

400

500

Number of Shots

0.40

0.42

0.44

0.46

0.48

0.50

0.52

ECE

ECE Trend

(b) Data size trend

2

4

6

8

10

Number of Prompt

40

42

44

46

48

1 shot Cifar100

Shallow Prompt

(c) Prompt size trend

2

4

6

8

10

Number of Prompt

0.34

0.36

0.38

0.40

0.42

ECE

1 shot Cifar100

Shallow Prompt

(d) Prompt size trend

Number of classes: To study the impact of number of classes, we formulate 5-way 1 shot, 10-way 1 shot, and 100-way 1 shot tasks using Cifar100. The results are presented in Table 12. As we decrease the number of shots from 100 to 5, we see an increase in accuracy and a decrease in ECE. We observe that the model is more accurate as tasks become easier (indicated by fewer classes i.e., lower N value in Table 12). However, the under-confidence issue remains.

Number of unfrozen parameters: We conduct additional experiments on Cifar100 100-way 1-shot tasks by varying the number of prompts for 1) shallow prompt: prompt added to the input only and 2) deep prompt: prompt added to all Transformer encoder layers' input as well. The accuracy and ECE trends are presented in Figure 15 (c-d). As can be seen, with the increase in the number of prompts for both shallow and deep prompts, there are fluctuations in accuracy and ECE performance. However, the under-confidence issue persists for all the cases.

F.6 Additional Experiments and Comparison

In this section, we carry out additional experiments to study the OOD performance of our model, and compare our model with additional methods present in literature.

OOD Performance: The current work, being an instance of fine-grained uncertainty quantification works, could potentially help in OOD detection. To this end, we present the OOD results of Cifar10 as in-distribution dataset and Cifar100 as out-of-distribution dataset with AUROC, FPR95, AUPR metrics for our model on 100-way 1-shot and 100-way 5-shot Cifar100 tasks in Table 13. As seen, B-PEFT performs better than PEFT, and with more training data, the model's OOD detection capabilities improve. Even with only 5 samples/class (i.e. 100-way 5-shot Cifar100 task), the model can achieve an AUROC of 92.58.

Model Comparison: We first carry out experiments with cosine classifier [22] without training for the 100-way 1-shot task on Cifar100. The cosine classifier has comparable generalization performance (accuracy) in comparison to VPT based model (see Table 14). However, looking at the ECE, the miscalibration issue is even higher than VPT based model. Hence, the simple solution (cosine classifier) does not ensure calibrated predictions. We also carry out experiments with test time augmentation [4] and VPT fine tuning with LoRA [70], on 100-way 1-shot Cifar100 dataset. Towards comparison with Bayesian inspired methods [14], we use Laplace approximation on last layer of the model using Kronecker Product and Diagonalization represented by KronLaplace and DiagLaplace in Table 14. As can be seen, these methods also suffer from the under-confidence issue when straightforwardly extended to the VPT. B-PEFT achieves much better generalization and calibration performance than these baselines.

G Calibration Behavior of Self-Supervised Model

In this work, we focus on vision foundation models that are pre-trained in a supervised learning paradigm. These models have shown remarkable effectiveness in a wide range of areas including image classification, video understanding, and visual recognition among others. Alternatively, foundation models have been developed that pre-train in a self-supervised fashion ( e.g., CLIP [52]). These models demonstrate good zero-shot performance on various datasets. As a representative of the self-supervised models, we use CLIP in our experiments. Parameter-efficient methods [77, 76, 21] have been proposed to adapt the CLIP model for downstream tasks. We use the popular methods: adapter and prompt for few-shot adaptation to study the calibration behavior. The reliability plot of different shot adaptations (1,2 and 5

0 1 2

5

10

20

Shots

20

25

30

35

40

45

50

55

60

Num of Shots vs Accuracy

zero-shot

prompt

adapter

shots per class) for cifar100 along with accuracy and ECE is presented in Figure 17. The accuracy behavior as we increase the number of shots from 1 to 20 is shown in Figure 16. In both methods, we observe for few-shot adaptations, the accuracy is either lower or comparable to zero-shot performance. More specifically, for adapter-based adaptation, even with 5 shots per class, the accuracy does not reach zero-shot accuracy. On the contrary, the parameter-efficient fine-tuning of supervised foundation models shows consistent improvement in accuracy as we increase the number of shots.

Towards calibration performance, we observe that prompt-based adaptation, along with zero-shot generally show good calibration performance with some degree of overfitting (see Figure 17). In contrast, prompt adaptation for supervised models that are severe, and prompt adaptation for CLIP models have no such issue. However, the adapter-based adaptation is mostly over-confident and hence has a higher ECE value than our proposed model. Considering these results for parameter-efficient few-shot adaption of pre-trained self-supervised models, the calibration and uncertainty behavior of such pre-trained models pose an interesting direction for further investigation.

H Societal Impact

We study the parameter-efficient fine-tuning techniques for large pre-trained vision foundation models, where we identify two key issues: the under-confidence of the fine-tuned models in their

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 41.960, ECE: 0.091

(a) 1 Shot using prompt

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 44.370, ECE: 0.076

(b) 2 Shot using prompt

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 49.050, ECE: 0.056

(c) 5 Shot using prompt

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 21.370, ECE: 0.225

(d) 1 Shot using adapter

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 29.150, ECE: 0.212

(e) 2 Shot using adapter

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

0.0

0.2

0.4

0.6

0.8

1.0

Accuracy: 43.340, ECE: 0.152

(f) 5 Shot using adapter

predictions, and lack of fine-grained uncertainty quantification capabilities. We develop a novel Bayesian Evidential model: B-PEFT that addresses the weaknesses of existing PEFT for pre-trained foundation models. Being an instance of the PEFT, our developed model enables the large foundation models to be adapted to challenging few-shot problems in a parameter-efficient and computationally efficient manner with limited memory requirements and energy footprint. Moreover, the developed model improves the generalization performance and the model's predictions are calibrated ensuring trustworthiness. Finally, the model has fine-grained uncertainty quantification capabilities which are highly desirable when applying these models in real-world safety-critical scenarios. Overall, our developed B-PEFT is expected to have a strong positive societal impact.

I Limitations and Future Work

In this work, we investigate the calibration of transformer-based foundation models under few-shot adaptation using various parameter-efficient fine-tuning methods. We focus on fine-tuning supervised pre-trained models for few-shot learning. We note that there are other self-supervised pre-trained models that show promising results for various benchmark datasets. We investigate the calibration of CLIP, a representative method, under few-shot adaptation using the two most popular parameterefficient fine-tuning methods: prompt and adapter. Our preliminary results demonstrate that the few-shot performance does not consistently increase in comparison to zero-shot. Similarly, promptbased fine-tuning has relatively better calibration than adapter-based fine-tuning. As an extension of this work, we will investigate the calibration performance of self-supervised foundation models. Additionally, it could be interesting to study the calibration performance of PEFT for tasks beyond image classification, e.g., to other modalities such as audio and language foundation models. For instance, the ideas developed in this work could potentially be used in situations where the PEFT leads to mis-calibrated models and the developed model requires trustworthy fine-grained uncertainty quantification capabilities. If these data modalities are also modeled using transformers and follow the parameter-efficient fine-tuning paradigm in performing downstream tasks, we expect the proposed approach can benefit them in a similar way.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction present the paper's contribution and scope, and match the theoretical and empirical results in the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the proposed work have been discussed in Section I.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Complete proof of all the theoretical claims is presented.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details for reproducibility along with link to the code is provided.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Only benchmark datasets and publicly available models are used for training and evaluation.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental setting and details are provided.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All the key results present the mean and standard deviation of five trials.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details of all compute resources used in experiments is provided

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The work confirms to the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The societal impact is discussed in Section H.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The work poses no obvious high risk for misuse.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: References and citations are provided as required.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The link to the source code and resources is provided.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing and research with human subjects is involved in this research work.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing and research with human subjects is involved in this research work.

Peer reviews:
comments: SUMMARY:
This paper focuses on the parameter efficient fine-tuning (PEFT) for foundation models in the few-shot learning settings. The authors reveal that the adapted models lack accurate fine-grained uncertainty quantification capabilities. Specifically, the few-shot tuned models perform remarkable accuracy but with large expected calibration errors. To address this problem, they propose a lightweight Bayesian PEFT framework by designing two Bayesian components to improve models' confidence. Theoretical analysis and experiments on 4 visual benchmarks prove the effectiveness of the proposed method.

STRENGTHS:
1.The paper is well-written and the storyline is clear.
2.The proposed Bayesian-PEFT is simple-and-effective, and theoretical analysis in the paper can reveal the underlying mechanism of the method.
3.Extensive experiments on ViT in several few-shot settings show the effectiveness of Bayesian-PEFT.

WEAKNESSES:
1.The title may be a bit inappropriate. If the Bayesian-PEFT is claimed to be used for foundation models, experiments on more types of foundation models are required. For example, I am curious whether it still works well on large language models (LLMs), e.g., LLaMA3. Also, it is unclear whether the under-confidence issue observed in the ViT also exists in LLMs.
2.Although it may be beyond the scope of the paper's research, I am still curious whether the under-confidence issue studied in the paper disappears when there is sufficient training data.
3.It would be better if a schematic diagram of the proposed Bayesian-PEFT could be provided.

QUESTIONS:
Does the proposed Bayesian-PEFT work well in LLMs, e.g., LLaMA3?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This submission presented a lightweight Bayesian Parameter Efficient Fine-Tuning (Bayesian-PEFT) framework for large transformer-based foundation models. Experiments across diverse datasets demonstrated the improved calibration performance by Bayesian-PEFT on multiple other PEFT techniques.

STRENGTHS:
* The paper is well written and the experiments are extensive.

* The proposed transformation function $A_m$ is well motivated and the experiments supports authors' claim.

WEAKNESSES:
* The dependence on Evidential Deep Learning is quite significant. To some extent, the proposed method extends EDL, rather than being designed specifically for PEFT of foundation models.

* Some editorial errors (e.g., "we we") exists, the submission may need further proofreading.

QUESTIONS:
* In Eq. (6), would there be a numerical stability problem when dividing e_{min}, as this value can be very small?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

comments: SUMMARY:
This paper studies the problem of parameter efficient fine-tuning. The authors pointed out mis-calibration issues caused by parameter efficient fine-tuning, which is, more specifically, under-confident estimation when fine-tuning data is limited. To solve this issue, the author proposed Bayesian PEFT, a Bayesian parameter efficient fine-tuning method that introduces two components, i.e., a base rate adjustment to strengthen the prior belief of pre-trained knowledge, and a evidential ensemble with diverse ensemble components. The authors provided both theoretical justification as well as extensive experiments across diverse datasets.

STRENGTHS:
1. This paper is well written and easy to follow.

2. The paper discusses an interesting observation on low confidence in few-shot setup for PEFT.

3. The proposed method seems to be reasonable and solve the identified problem.

WEAKNESSES:
1. It would be interesting to see more in depth discussion and empirical analysis on the observation of uncertainty of PEFT, particularly, related to data size and number of unfrozen parameters. 

2. More discussion related to OOD generalization might be needed. Particularly, how does the proposed method help OOD generalization in terms of prediction accuracy.

QUESTIONS:
1. Is fully fine-tuned model also have low uncertainty compared to PEFT model? 

2. Will the proposed method improve robustness such as OOD generalization in terms of model accuracy? 

3. How does the proposed approach work on other modalities, such as audio and language?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper advances the Bayesian Parameter Efficient Fine-Tuning (Bayesian-PEFT) approach: a strategy to fine-tune large pre-trained foundation models with well calibrated classifier and ability to gracefully deal with out-of-distribution (OOD) data thanks to uncertainty quantification from ensembling.\nFirst the authors study the behavior of a fine-tuned foundation model with several PEFT methods (bias fine-tuning, adapters, side-tuning, visual prompt-tuning) in the few-shot learning regime (defined here as reducing the number of training samples per class, but keeping the original number of classes) and show that even though they achieve good accuracy, they all display under-confident predictions. The authors attempt to trace back the source of this behavior with evidential learning tools and conclude that the models are accurate thanks to relatively greater evidence of the correct class compared to other classes, and that models are under-confident because all classes are being assigned very low evidence, including the correct ones.\nInspired by this finding, the authors devise an evidential learning based PEFT strategy that adjusts the base rates of the evidential head such that prior belief from pre-training knowledge is strengthened while maintaining accuracy. Secondly the authors propose an evidential ensemble with a technique to induce diversity with different incorrect belief regularization strengths that penalize the model differently when assigning high beliefs to non-correct classes.\nThe proposed Bayesian-PEFT is evaluated on 4 datasets  (CIFAR10, CIFAR100, Food101, Flowers102) under different few-shot regimes, showing boosted accuracy and calibration.\n\nSTRENGTHS:\n### Significance\n- this work deals with an important problem in the current landscape of machine learning: how to leverage existing foundation models to downstream tasks with few labels while ensuring their reliability\n \n\n### Originality\n- I find the use of evidential learning for this setting to be original. The idea of preserving prior knowledge and increasing the evidence for all classes with evidential learning for this use-case is novel\n\n- the strategy of inducing diversity of ensembles with different strengths for incorrect evidence regularization is novel for the evidential learning literature and has the benefit of not needing communication between ensemble particles during training -> improved computational efficiency\n\n### Clarity\n- the paper is overall clearly written conveying a well argued story about the behavior observed for different PEFT methods in the few-shot regime, then studying it with evidential learning lens and then proposing a few solutions to mitigate the encountered non-desirable behaviors\n\n- the code comes to support information potentially missing from the paper\n\n### Quality\n\n- the authors conduct numerous experiments under several few-shot learning regimes, showing improved performances. They also conduct several ablations and sensitivity studies towards better understanding the impact of the building blocks of this method and the choices made for them\n\n- code is provided in the supplementary for reproducibility of results\n\n- the authors study their work on several PEFT methods from computer vision literature, mostly VPT in the main paper and the others in supplementary\n\nWEAKNESSES:\n### Framing of few-shot learning task\n- the few-shot learning setting considered here is quite different from the one considered in most of the few-shot learning/meta-learning literature with many small sets of support images and classes and query images (sampled from miniImageNet, Omniglot, CIFAR, etc.), sometimes called _\"episodes\"_. In contrast the proposed few-shot learning setting proposed here keeps the original number of classes and reduces the number of samples to 1-20 samples per class. This ends up as a different setting as the model is finetuned on 200 to 4000 images for CIFAR100 for instance from the 1-shot to the 20-shot setting, which is not a big dataset but still not that few images to train on.\n\n- I'm wondering whether the proposed setup of having many classes, but few samples, may be the source of the under-confidence behavior of PEFT methods for the few-shot learning setup that is proposed here. In such cases, simple solutions from the few-shot learning literature that do not require training: e.g., cosine classifier [b], [c]  (i.e., averaging features from the same class, do dot-product with query features, then softmax on similarities)\n\n- the authors could refer to this setting that they propose by a different name to avoid confusion with few-shot learning where models can be finetuned with very few samples and there are several Bayesian methods there already. A potential alternative could be: low-shot regime.\n\n\n### Motivation for evidential learning approach and few baselines\n- the formalism based on evidential learning is interesting, but is not always trivial to implement and previous works [d],[e] reported difficulties in scaling evidential methods and the Dirichlet predictor to larger number of classes and encoders.\n\n- while the approach itself is clearly described, the motivation for using such a line of approach instead of others that are more simple, generic and with a wider adoption, e.g., last-layer methods like Laplace-Redux [f], Bayesian last layer [g]. Since the endeavor of this work is to look at more practical settings with few labeled samples and reliability constraints, it may make sense to compare this work with some simple baselines to show the benefits of the approach and its trade-offs.\nOther potentially relevant baselines would be the cosine classifier from few-shot learning, LoRA ensembles [h], linear probing with test-time augmentation [i]\n\n- It would be also interesting to expand the list of PEFT methods studied with more recent and potentially more parameter efficient ones like LoRA [j] or VeRA [k] that have been shown useful to large transformer networks and could expand the scope of this work beyond image classification.\n\n\n### Missing related works\n- as mentioned above, the field of few-shot learning/meta-learning has seen several Bayesian methods with uncertainty quantification. Often these meta-learning methods fine-tune the network at test-time with just a few labeled samples (from classes not seen during pre-training). Here is a non-exhaustive list of relevant works from this area: PLATIPUS [m], VERSA [n], LLAMA [o], Bayesian MAML [p], Bayesian TAML [q], etc.\n\n- a discussion on how the current B-PEFT is different from them and what type of problems it addressed and with which benefits would be useful here.\n\n- the idea of converting a pre-trained network into a Bayesian Neural Network and training over a short period of a few epochs (not a low-shot setting though) has been recently proposed in ABNN [l]. The diversity-inducing mechanism with different incorrect belief regularization strengths can also be related with the random prior mechanism from ABNN [l] where an ensemble of networks was obtained from a pre-trained network and the diversity was induced by introducing a different class-level prior for each ensemble member. ABNN is pretty recent so it can be considered as concurrent work. \n\n\n### Misc.\n- the OOD experiment could benefit from the inclusion of typical metrics used in literature (e.g., AUROC, FPR95, AUPR, etc.) to allow easier comparison and putting the performance in context of other works.\n \n\n### Minor - Novelty\n- the evidential learning choice seems to be inspired to some extent by the multidimensional belief quantification work [37] that used uncertainty for vanilla few-shot learning / meta-learning. The authors however expand the idea further with belief strenghtening and incorrect belief regularization for ensembling.\n\n\n\n\n**References:**\n\n[a] Vinyals et al., Matching networks for one shot learning, NeurIPS 2016\n\n[b] Gidaris et al., Dynamic Few-Shot Visual Learning without Forgetting, CVPR 2018\n\n[c] Qi et al., Low-Shot Learning with Imprinted Weights, CVPR 2018\n\n[d] Joo et al., Being Bayesian about Categorical Probability, ICML 2020\n\n[e] Franchi et al., Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification, arXiv 2020\n\n[f] Daxberger et al., Laplace Redux -- Effortless Bayesian Deep Learning, NeurIPS 2021\n\n[g] Kristiadi et al., Being bayesian, even just a bit, fixes overconfidence in relu networks, ICML 2020\n\n[h] Balabanov et al., Uncertainty quantification in fine-tuned LLMs using LoRA ensembles, arXiv 2024\n\n[i] Ashukha et al. Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning, ICLR 2020\n\n[j] Hu et al., Low-rank adaptation of large language model, ICLR 2022\n\n[k] Kopiczko et al., VeRA: Vector-based Random Matrix Adaptation, ICLR 2024\n\n[l] Franchi et al., Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models, CVPR 2024\n\n[m] Finn et al., Probabilistic Model-Agnostic Meta-Learning, NeurIPS 2018\n\n[n] Gordon et al., Meta-Learning Probabilistic Inference For Prediction, ICLR 2019\n\n[o] Grant et al., Recasting gradient-based meta-learning as hierarchical bayes, ICLR 2018\n\n[p] Kim et al., Bayesian Model-Agnostic Meta-Learning, NeurIPS 2018\n\n[q] Lee et al., Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks, ICLR 2020\n\nQUESTIONS:\nThis paper takes an interesting direction of study: ensuring reliability of PEFT foundation models in the low-shot learning regime.\nI find the endeavor of the authors nice, with a good story and several PEFT methods. However I do have several concerns regarding the chosen few-shot learning setup (different from the common practices in the literature) and not necessarily realistic, the limited motivation brought for the use of evidential learning as well as the limited baseline. In addition, the relatively rich literature of Bayesian methods for few-shot learning is ignored.\n\nMy current rating is leaning towards reject at this time (a bit on the fence), but I'm looking forward for the rebuttal.\n\nHere are a few questions and suggestions that could be potentially addressed in the rebuttal or in future versions of this work (please note that suggested experiments are not necessarily expected to be conducted for the rebuttal):\n\n1. Please argue why using evidential learning should be used for this setting rather than last-layer or other Bayesian inspired methods\n\n2. What is the performance of a cosine classifier in this setting? What about Laplace-Redux or LoRA ensembles?\n\n3. How is this method working with LoRA-like methods, e.g., VeRA?\n\n4. Why considering the proposed few-shot learning setup instead of the one from legacy few-shot learning / meta-learning literature\n\n5. Can the authors discuss the difference and benefits of Bayesian-PEFT w.r.t. Bayesian few-shot learning methods?\n\n6. Inclusion of OOD specific metrics in the OOD experiment.\n\nLIMITATIONS:\nThe authors addressed some limitations in the supplementary.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}