Title: Bayesian-Informed Diverse Sampling for Calibration of Fine-Tuned Foundation Models with Evidential Ensembles
Primary area: foundation or frontier models, including LLMs
Keywords: Model calibration, uncertainty evaluation

Abstract: As large foundation models become increasingly crucial in critical domains, concerns regarding the handling of incorrect content from these models continue to grow. Fine-tuning these models for downstream tasks exacerbates the risks of generating poor-quality outputs, as potential domain shifts and variability in downstream task data size can significantly impact performance. Recent efforts have focused on improving model reliability by quantifying uncertainty in generated content. State-of-the-art solutions generate diverse answer paths through probabilistic decoding methods like top-k, nucleus sampling, or beam search, using the inconsistencies among these answers as a measure of uncertainty. From a Bayesian perspective, aggregating these diverse paths effectively marginalizes over the approximate model parameter distribution, while a single output based on maximum conditional probabilities serves as a point estimate of that distribution. This further justifies the superiority of multi-output methods over single-output approaches. However, fine-tuning on limited downstream data can result in a posterior distribution that is more complex, flatter, and potentially multimodal, rendering current methods insufficient for accurate uncertainty quantification. To better approximate this complex posterior, we propose a novel diversity-inducing ensemble approach guided by Distributionally Robust Optimization (DRO) and evidential theories. Our method is theoretically guaranteed to improve calibration performance. Empirical results across benchmark datasets and large visual language models demonstrate the effectiveness of our approach in estimating content quality and enhancing model reliability.