Variational Visual Question Answering for Uncertainty-Aware Selective Prediction
Authors: Tobias Jan Wieczorek
Nathalie Daun
Mohammad Emtiyaz Khan
Marcus Rohrbach
Venue: Accepted by TMLR ()

Abstract: Despite remarkable progress in recent years, Vision Language Models (VLMs) remain prone to overconfidence and hallucinations on tasks such as Visual Question Answering (VQA) and Visual Reasoning. Bayesian methods can potentially improve reliability by helping models predict selectively, that is, models respond only when they are sufficiently confident. Unfortunately, such approaches can be costly and ineffective for large models, and there exists little evidence to show otherwise for multimodal applications. Here, we show for the first time the effectiveness and competitive edge of variational Bayes for selective prediction in VQA. We build on recent advances in variational methods for deep learning and propose an extension called "Variational VQA". This method improves calibration and yields significant gains for selective prediction on VQA and Visual Reasoning, particularly when the error tolerance is low (≤ 1%). Often, just one posterior sample yields more reliable answers than those given by models trained with AdamW. In addition, we propose a new risk-averse selector that outperforms standard sample averaging by considering the variance of predictions. Overall, we present compelling evidence that variational learning is a viable option to make large VLMs safer and more trustworthy.

Published in Transactions on Machine Learning Research (04/2026)
Variational Visual Question Answering for Uncertainty- Aware Selective Prediction
Tobias Jan Wicczorck
tobios. www.ssorckkhhadddrrsttaall
TU Darmstadt & Al, Germany
Nathalie Daun
TU Darmsladl & hessian.AA,, AI, Germany
Mohammad Emtiyaz Khan
RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
Marcus Rohrbach
TU Darmstadt Aessian.Al, Germany
Reviewed on OpenReview: https: // openreview. net/ forum? jd-jtaMIdJIso
Abstract
Despite remarkable progress in recent years, Vision Language Models (VLMs) remain prone to overconfidence and hallucinations on tasks such as Visual Question Answering (VQA) and Visual Reasoning. Bayesian methods can potentially improve reliability by helping models predict selectively, that is, models respond only when they are sufficiently confident. Unfortunately, such approaches can be costly and ineffective for large models, and there exists little evidence show otherwise for multimodal applications. Here, we show for the first time the effectiveness and competitive edge of variational Bayes for selective prediction in VQA. We build on recent. advances in variational methods for deep learning and pro- pose extension called " "Variational VQA", This method improves calibration and yields significant gains for selective prediction on VQA and Visual Reasoning, particularly when the error tolerance is low (< 1%). Often, just one posterior sample yields more reliable answers than those given by models trained with AdamW. addition, we propose a new risk-averse selector that outperforms standard sample averaging by considering the variance of predictions. Overall, we present compelling evidence that variational learning is viable option to make large VLMs safer and more trustworthy.
1 Introduction
Advances in VLMs (Wang et al., 2023; 2024; Li al., 2024) have led substantial gains on classical VQA benchmarks (Antol et al., 2015; Goyal et al., 2017), with performance now approaching surpassing strong VQA models miscalibrated, prone to hallucinations, and often confidently guess answers instead of expressing uncortainty (ef. Fig. 1). In short, these models do not have good notion of confidence about their own knowledge. This shortcoming hinders their deployment in safety-critical domains such as medical diagnosis or assistance for the visually impaired. When a model is confronted with adversarial (Sheng al., 2021) or unanswerable (Bigham et al., 2010) inputs, which are common in the real world, these issucs bocome even more pronounced.
human levels. However,
One way improve reliability to allow model to abstain when it is unsure of its response. The selective prediction framework formalizes such abstentions (Chow, 1957; El-Yaniv and Wiener, 2010), where the main

---

Published in Transactions on Machine Learning Research (04/2026)
Question: Does the pedestrian light say walk?
Correct answer: "No"
AdamW:
VarVQA: " "Not sure"
"Yes"
Threshold for abstention
P(Yes)
.5
Figure 1: Despite recent performance gains, VLMs trained with popular optimizers like AdamW do not know when they are wrong, Our VarVQA approach uses posterior variances to help the model decide when to abstain. The result shown on the right is for BE:T-3 (Wang et al., 2023); which achieves near-human accuracy on VQAv2 (Goyal al., 2017).
challenge is to find a good confidence estimator that separates correct answers from incorrect ones. Given such an estimator, the incorrect answers can be replaced with an output that indicates that the model does not know the answer such 'Not sure' or 'I do not know', which replaces potentially costly errors with absten- tions. Although recent work has connected selective prediction to hallucinations, (Kalai et al., 2025), the literature multimodal models remains relatively sparse. Previous approaches in the multimodal domain have proposed to incorporate additional model components to improve confidence estimates: Whitehead et al. (2022) train a lightweight head on top of the frozen VLM backbone, while Srinivasan et al. (2024) use external vision tools and an additional language model to quantify uncertainty. Both works do not attempt to improve the reliability of the underlying model. Instead, their solutions introduce additional overhead the prediction pipeline while also adding new failure points for uncertainty estimation.
Variational Bayesian (VB) methods (Graves, 2011) can potentially address the unreliability of VLMs without requiring additional components tools. In particular, the uncertainty in the learned posterior distribution over model parameters can be used to help the model make prediction only when it is sufficiently confident. This theory remains untested though, as for a long time, VB approaches have been ineffective for large transformer-based architectures. However, the recently developed Improved Variational Online Newton (IVON) optimizer (Shen el al., 2024) has enabled elfective variational training of models such as GPT-2 (Radford et al., 2019) with no significant overhead compared to AdamW (Loshchilov and Hutter, 2019). So far, IVON has been limited to unimodal domains, and in this work, multinodal applications, particularly with regards to selective prediction. Our contributions are as follows:
the first to test its effectiveness for
We demonstrate that variational training is effective for large multinodal transformer- based archi- tectures and introduce the Variational VQA (VarVQA) framework for selective VQA abstentions.
2. We demonstrate improved uncertainty estimation across multiple dimensions: better calibration, and enhanced selective prediction with particularly large gains at low error tolerances, well increased robustness under distribution shift.
3. We establish superior sample efficiency compared to Monte-Carlo (MC) Dropout, showing that Variational VQA provides better reliability given an equal compute budget.
4. We propose a new risk-averse selector function that leverages output variance, yielding consistent improvements in high-stakes selective prediction where errors are partieularly costly.
2

---

Published in Transactions on Machine Learning Research (04/2026)
2 Background and Related Work
2.1 Uncertainty and Reliability in Visual Question Answering
Visual Question Answering (VQA) is popular multimodal task that requires model to understand two modalities and their interaction to predict answers. While multimodal models (Ti et al., 2023; Wang et al., 2023; 2024) have recently achieved human-level performance on standard benchmarks like VQAv2 (Goyal et al., 2017), they still perform poorly at selective prediction on the same benchmarks (Dancette et al., 2023). Our work here the first to address the selective prediction task using variational Bayesian methods.
Selective Prediction. In the selective prediction framework (Chow, 1957; El- Yaniv and Wiener, 2010), a "selector" assigns a confidence score to the answer given by a model and subsequently decides whether the prediction is accepted or the model is forced to abstain instead (that is, it says "I do not know"). This decision is made by comparing the assigned confidence against a given abstention threshold. In VQA in particular, the model learns function f : Q + A to predict an answer E A, given a multimodal input (i, a) consisting of an image i E I and question q E Q. In selective prediction notation, the model output space is augmented by an abstain output D. This transforms the predictive model f into selective model m, incorporating both ʃ and a selector g. The answer (1x) is accepted if g(x)) is above the abstention threshold y E I., and rejected otherwise. We follow the notation of Whitehead et al. (2022):
m(*) = (J=))()) = ((x) if if)
-(((e)
2 7 if g( n.
A high threshold y corresponds to a conservative case, in which the model answers only the questions which it is most confident. Lowering reduces abstentions (higher coverage), but increases the error rate among accepted answers (higher risk). The tradcoff between risk and coverage is unavoidable, but better confidence estimator yields a lower risk at any given coverage level. In practice, the cost of error or the risk level is specified in advance, and y is set accordingly, see Section 4.2, Typically, the answer likelihood (Geifman and El- Yaniv, 2017) or the predictive entropy are used as selection functions.
Most of the prior work on seleetive prediction can be classified into either external approaches or integrated approaches. In external setups, & selector is built on top of the frozen predictive model, for example in the form of trainable model head (Whitehead et al., 2022; Mielke et al., 2022; Mushtaq al., 2025), LoRA parameters (Chen et al., 2023) or vision tools (Srinivasan et al., 2024). In integrated setups, predictor and selector have at least one combined training phase. Integrated selectors take different forms as well, such as a model head (Geifman and El-Yaniv, 2019) or a dedicated abstention class (Ziyin et al., 2019). However, if model and selector are trained together, instabilities often ensue, which require special treatment (Geifman and EI- Yaniv, 2019). Bayesian approaches have not been considered for selective prediction so far, with the exception of concurrent work by (Daheim et al., 2025), which has explored IVON for generative language modeling, although they do not consider multimodal tasks. In contrast to prior work selective prediction in VQA, our objective is to directly improve the reliability of model confidence estimates without additional parameters, training phases, or tools. In other words, we train VLMs where reliability is "baked-in" by design, not added as an afterthought.
Calibration. Calibration represents different angle uncertainty estimation, namely the alignment of model's predictive confidence with its accuracy: When the model expresses 4% confidence in answer, it should be correct x% of the time. The difference between calibration and selective prediction becomes clear when considering a model that is correct on y% of examples and is also always exactly y% confident. (for a fixed satisfying 0 < y S 100). Although this model is perfectly calibrated, it cannot distinguish its correct and incorrect outputs and thus fails at the task of deciding when to abstain. Prior work has found that large neural networks often exhibit overconfidence, particularly in Out- Ont-OO-Domain (OOD) settings (Snock et al., 2019). In unimodal classification tasks, temperature (Guo et al., 2017) and Platt (vector) Platt et al. (1999)
3

---

Published in Transactions on Machine Learning Research (04/2026)
scaling are effective at improving calibration. Ensembling (Lakshminarayanan et al., 2017) typically yields even better results, but requires prohibitive resources to train N models. New ideas, such prompting the model to express a verbalized confidence have been mostly ineffective for VLMs (Xuan et al., 2025). We show that Variational VQA yiclds well-calibrated VLMs, achieving a lower Expected Calibration Error (ECE) than vector scaling, while matching other sampling methods like Monte-Carlo Dropout (Gal and Ghahramani, 2016). In general, we argue that for a VLM to be reliable, should be calibrated and also know when to abstain. Both these are improved with Variational VQA in comparison to models trained with the AdamW optimizer.
2.2 Variational Bayes for Deep Learning
Variational Bayesian Learning provides a principled approach to estimate uncertainty by learning probability distributions (often Gaussians) over the weights of a neural nelw While conventional deep learning methods estimate network weights 0 by minimizing empirical risk E(O) 인 Σi (,()), variational methods estimate distribution q(0) over network weights by minimizing the variational objective
[(q(O)) = XEE(o [((0]] + DK (g(0) lI p(0).
Here, N is the size of the training set, (,))) the loss for example i, (xl(y) denotes the Kullback-Leibler divergence, is a scaling parameter, and p(0) the prior weights distribution. To keep computational costs manageable, q()) often chosen to be a diagonal-covariance Gaussian, that is, q(0) = ( | m, diag(v)),
where and y are the paramcter mean and variance vectors. The objective L(q(0)) can be reparametrized in terms of and v, and L(m, u) is typically approximated through MC sampling of the weights.
IVON. In the early 2010s, variational methods that directly optimize parameter means and variances through standard deep learning techniques such as SGD achieved promising results (Graves, 2011; Blundell et al., 2015). However, in subsequent years, these approaches could not keep up with the growth in scale of network architectures (Trippe and Turner, 2018; Foong et al., 2020; Coker et al., 2022). Recently, natural gradient methods that build Hessian estimates through an Adam-like update (Khan el al., 2018; Osawa al., 2019) have addressed this issue. The IVON optimizer (Shen et al., 2024) further develops those and obtains comparable accuracy and better uncertainty estimates than AdamW at nearly identical training cost. In particular, IVON uses an Adam-like (Kingma, 2014) update for the parameter means m and variances v. Similarly to adaptive scaling in Adam, IVON updates m by using gradients scaled with h- an online estimate of the diagonal Hessian. In an update step, we first sample ~ q, then compute a minibatch gradient I. The following four lines are used to compute the minibatch Hessian estimate h, update the moving average of the Hessian h and then update the means m and variances
h g(0 m)
(3)
(1 (1- B2 - Ri2 2(h + 8)
h 82h (1 -22
(4)
g + 5m
m - m
(5)
h 1 5
8 1(h + 5)
(6)
Here, is the learning rate, 5 the weight decay and ğ moving average of the gradient. IVON also uses Adam-like momentum for the gradients and the Hessian. A notable difference to Adam, however, is the absence of the square root over the scaling vector + in line Eq. (5). Notably, the initialization of the
4

---

Published in Transactions on Machine Learning Research (04/2026)
Hessian estimate ho is crucial hyperparameter to set carefully. For more details, we refer to the original paper by Shen et al. (2024).
We use IVON because it offers several advantages compared to other Bayesian baselines. Unlike the Laplace approximation (MacKay, 1992; Daxberger et al., 2021), it does not require an additional pass through the data to compute the Hessian. Neither does it require additional training like Stochastic Weight Averaging (SWA) (Izmailov et al., 2018). Compared to MC Dropout (Gal and Ghahramani, 2016), the advantage is the availability of fixed posterior form that can be more casily used for downstream tasks. For instance, al.; 2017), which can further improve
the method easily amenable to ensembling (Lakshminarayanan performance (Daheim et al., 2025). We offer new insights compared to previous TVON works (Shen et al., 2024; Cong al., 2025; Daheim al., 2025), by showing its effectiveness in training multimodal models and for selective prediction. We further propose a new selection function that uses the output variance, which was never utilized in prior work.
3 Variational VQA
Our Variational VQA approach uses the IVON optimizer to train large VLMs and evaluates the reliability of its output confidences in comparison to baselines like AdamW and MC Droput. In Section 3.1, we describe how model confidencos are obtained, in Section 3.2 describe the baseline selectors, and in Section 3.3 we present our new risk-averse selector.
3.1 Inference and Model Confidence
At inference, variational methods typically use the learned posterior through MC sampling. However, if computing efficiency imperative, one can ignore the variances (set v = 0) and use the mean parameters m for inference (Shen et al., 2024), which requires only one forward pass. We refer to this approach as 'VarVQA mcan'. For an input x, denote the model's output likelihood vector by [a; 0), where the k-th entry fa (x; 0) contains the likelihood of class k (out of K classes). VarVQA, on the other hand, performs sampling, that is, draw model parameters g(s) 9 to get S e N likelihood vectors f(:((())). These are aggregated to obtain mean likelihood vector To and mean likelihood variance vector 52:
((x)
1 5
Σ ((x;
((x))
1 S (
Σ
3.2 Baseline selector functions
We start with the baseline selector for deterministic methods (AdamW, VarVQA mean). We employ the widely used MaxProb selector (Geifman and EI- Yaniv, 2017), which uses the answer likelihood. In a classifi- cation task, the MaxProb seloctor is defined as (x)(x)) max; f& (a;; #). We use MaxProb, because we find it to consistently outperform predictive entropy and related functions. In the case of multiple samples, reason- able baseline is predictive averaging (Gal and Ghahramani, 2016). Here the selector is ( max, (x(x), where Pu (x) is the k-th entry of the vector p(x), cf. Eq. (7).
3.3
A new risk-averse selector
In this work, particularly for the context of selective prediction, we propose to go Beyond Predictive Averaging (BPA) by also employing the empirical output variances (ef. Eq. (7)). This is done in a risk-averse (Pratt, 1978) manner, by penalizing high-variance predictions. While Pratt (1978) subtracts the variance (with a prefactor), we found the standard deviation to work best:
5

---

Published in Transactions on Machine Learning Research (04/2026)
Optimizer
Prediction
Selector
Determnistic
standard
9MP(#) max (x( 0)
AdamW
VarVQA mean
Base
VLM
Sampling-basse
IVON
MC Dropout
MMP (2) max PA (x)
VarVQA
9BPA = (a(())) G (z)
Figure Overview of the methods we experiment with and their selectors. PMP for calibration and 9BPA for selective prediction.
Variational VQA employs
(BPA(I) = px(())- -fx())
(8)
Herc, & = arg max; (x). Thus, the risk-averse selector docs not change the prediction, only the confidence.
Another perspective motivating ΦвPa comes from Bayesian credible intervals. If we approximate the distri- bution of the highest-likelihodd class as Gaussian - that is, /N (M,, [ then the term f4, - -xx corresponds to approximately the 16th percentile of this distribution. In other words, there roughly an 84% postcrior probability that the true class probability exceeds J.. - ox. This provides conservative lower confidence bound that is particularly valuable in high-stakes selective prediction where overconfident predictions carry significant costs. We find that the lo choice empirically balances conservatism against practical utility.
All our selective prediction results with VarVQA use ЛBPA- In Section 4.5, we provide ablation against predictive averaging. When it comes to calibration, VarVQA uses predictive averaging, as the subtraction of a leads to systematic underconfidence" When using MC Dropout with AdamW, we found no systematic benefits of ДВPA- We speculate that this is bocause the postcrior was not actively learned. Thus, we use only AhP for Dropout. The selectors used for each method are visually summarized in Figure 2.
4 Experiments
We describe our experimental setup, models and datasets in Scction 4.1 and the evaluation metrics in Sec- tion 4.2. Our results show that Variational VQA is effective for multimodal models, more sample-efficient than MC Dropout (Sec. 4.3), and more robust to OOD data than AdamW-trained models (Sec. 4.4). More- over, our novel selector 9BPA outperforms posterior predictive averaging on high-stakes selective prediction (cf. Sec. 4.2) across multiple models and tasks (Soc. 4.5).
4.1 Experimental Setup
We explore the effectiveness of Variational VQA on two large VLMs: ViLT (Kim et al., 2021) and BETT-3 (Wang et al., 2023). BETT-3 is ncar-SOTA? on VQAv2, but still small enough for full fino-tuning. Both ViLT and BETT-3 treat VQA as a classification task to 3129 answers, which is standard practice (Anderson et al., 2018). In terms multimodal tasks, we explore VQA (fine-tuning on VQAv2 (Goyal et al., 2017), evalustion on VQAv2 and AdVQA (Sheng et al., 2021)) and Visual Reasoning (fine-tuning and evaluation on NLVR2 (Suhr al., 2019)). The publicly available VQAv2 test splits do not include labels, which are
In selective prediction, only relative confidences matter, there no negative impact.
2 As of 10/2025, see the VQAv2 Challenge on FvalAT

---

Published in Transactions on Machine Learning Research (04/2026)
required to evaluate calibration and selective prediction (cf. Sec. 4.2). Therefore, follow previous work (Whitehead et al., 2022) and divide the VQAv2 validation set into over three training runs with different seeds. Error bars and shaded regions indicate standard error.
All results are averaged
Hyperparameters. We the optimal hyperparameters reported in (Kim et. al., 2021; Wang el al., 2023) for AdamW. For IVON, most defaults, of Shen et al. (2024), can be used, but the learning rate and Hessian initialization need to be adjusted. However, we find that due to strong correlation between the two, the dimensionality of the search space is effectively one. A full account is provided in Appendix A.
Number of Samples. By default, Variational VQA uses S 64 MC samples. We did not find significant improvements beyond this number. For early stopping,
eight MC samples to save compute.
Tempcrature and Vector Scaling. Previous work (Whitchoad et al., 2022) has shown that calibrating models with widespread methods like Temperature Scaling (Guo et al., 2017) and Vector Scaling (Platt al., 1999) has only a small effect on their selective prediction performance. We confirm these findings and show that the effect is consistently positive, and can be applied on top of any method (e.g. AdamW or VarVQA) to receive small additional gains. Full results are in Appendix C.
4.2 Evaluation Metrics
Accuracy. We work with the standard VQA accuracy (Antol al., 2015), which can also take non-integer values (0.3, 0.6, 0.9), besides and 1, if fewer than out of 10 annotators agree. NLVR2 accuracy is binary.
Calibration. We evaluate calibration using the Expected Calibration Error (ECE) (Pakdaman Naeini et al., 2015; Guo et al., 2017), as is standard practice. The ECE is computed by partitioning the model's answer confidences on dataset into m bins Dm, and then summing the bin-wise deviations of confidence from accuracy. We 15 in our experiments.
M
D Dm. - g(Dm) D
ECE = Σ
(9)
m=1
Coverage at Risk. For the selective prediction metries, we follow prior work (Geifman and El- El-ianiv, 2017; Whitehead et al., 2022; Dancelte et al., 2023). The standard selective prediction metric is Coverage Risk (C@R), which measures the percentage of questions the model is able to answer (í.e. where does not abstain), while keeping the error tolerance r below given risk level R:
C@R = max C(y) s.t. r(7) R, where we define
(10)
1
((>) = Σ [((()) 2 7), and
(11)
D
ZED
1
ELED (1- (((((
r(>)
D
(12)
C(z)
(13)
A larger CâR is better, as a model that abstains on (almost) all inputs is useful. We also compute the under the Risk-Coverage curve (AUC) (Kamath et al., 2020). A weakness of C R is that the
7

---

Published in Transactions on Machine Learning Research (04/2026)
threshold y is determined using the test set. This is necessary otherwise, comparison of results would be challenging: For a given risk R, one would have to judge both threshold generalization (i.e. whether the test risk matches the bound R), and the achieved test coverage.
Effective Reliability. Whitehead et al. (2022) suggested Effective Reliability &e that avoids test set threshold selection. It differs from accuracy by negative cost assigned to wrong answers:
Acc(x)
if g 2 and Acc(2) > 0, g(x) 2 y and Acc(x) = 0, if ((x) %
(x())
(14)
The total effective reliability is 2. EZED Pa(a), and the abstention threshold y is determined by optimizing & validation data. We report accuracy (Acc), CaR and & in percent, while keeping the ECE in [0, 1] to be consistent with Whitehead et al. (2022).
High-Stakes metrics. Both selective prediction metrics (C@R and Φ2) feature a parameter that controls the severily of mistakes. Our findings match previous work (ef. Tabs. 1,2 in (Whitehead et. al., 2022)): Models disproportionately struggle with settings in which errors are very costly (low-R, high-c) We collectively refer to these metrics high-stakes. For practical applications, it is arguably more important that models perform well in high-stakes metries than in low-stakes metrics, since large amounts of errors (even as low as 5%) are not acceptable in many real-world scenarios. Moreover, for ID experiments we observe saturation" in low-stakes metrics and thus focus our reported results on high-stakes.
It should be noted that, stakes are set too high (i.e. cost c too high or risk R too small), results can become noisy, as the impact of individual overconfident samples rises. This issue increases with smaller and less well-curated datasots (label noise can have an impact). In our experiments, we observe that the results were stable only up to c 100 and down to R Rs 4%, which is why we stop reporting there.
4.3 In-Distribution Experiments
We show ID results after fine-tuning VQAv2 in Table 1 and on NLVR2 (Visual Reasoning) in Table 2. Figure 3 visualizes the VQAv2 results. Variational VQA matches the accuracy of the conventional AdamW optimizer (Fig. 3a), while achieving better calibration in terms of lower ECE (Fig. 3b), and better (high- stakes) selective prediction in terms of higher C@1% (Fig. 3c) and 1100 (Fig. 3d). Additionally, 'VarVQA mean' (ef. Sec. 3.1), is frequently more reliable than AdamW (lower ECE, higher C&R, &o), while needing the same inference compute. Finally, the VarVQA sampling strategy consistently outperforms MC Dropout, which uses the same amount of samples at inference, in terms of selective prediction, while achieving low ECE of S 0.03 throughout and < 0.02 VQAv2 with all three tested models. Regarding selective prediction, the benefits of VarVQA over AdamW are largest for the high-stakes metrics. When only one mistake per 200 samples is allowed (C@4%), VarVQA on different VLMs improves 7% - 9% on VQAv2 and 14% on NLVR2 vs. AdamW. When Deep Ensembles (Lakshninarayanan et al., 2017) are applied on top of an existing method, the reliability improves (ef. Table 3). However, on VQAv2, even vanilla VarVQA is often better than the AdamW Ensemble and the VarVQA Ensemble stays consistently ahead. More in-depth results in Appendix Appendix F. We note that Deep Ensembles cause significant overhead through the necessity of training N models instead of onc. (We use N 3 due to computational constraints.)
3Tee achieved C&R and in these settings much further helow the theoretical optimum than for high R/low c. 4or example, BEIT-3 large an VQAv2 achieves 0510% 81% and 0820% 98%.
8

---

Published in Transactions on Machine Learning Research (04/2026)
0.08
25
Accurac (%)
0.07
20
76
0.06
@
0.05
Coverage
24
8
15
0.04
10
72
18
0.03
0.02
ViLT
BEIT-3 base
BEiT-3 large
ViLT
BEIT-3
BEİT-3 large
VILT
BEIT-3 base
BEİT-3 large
VILT
BEÎT-3
BETT-3
base
base
large
Accuracy
(b) Calibration (4)
(c) Selective Prediction
(d) Selective Prediction
+ VarVQA
T
VarVQA mean
T AdamW
Accuracy, calibration and selective prediction results for different models after fine-tuning on VQAv2. Error bars indicate standard crror across three sceds.
Figure 3:
Table 1:
Reliability evaluation VQAv2 for fine-tuned models. The variable N denotes the number of forward passes. Best results per model are bold.
Selective Prediction
Sel. Prediction low-stakes
Calibration
Model
Method
N
Acc.
high-stakes
ECE (4)
Ca 7%
C@1%
450
1100
C'05%
210
AdamW
1
69.30
0.061
5.03
10.58
8.41
2.89
36.24
24.05
VarVQA mean
1
69.63
0.071
6.77
13.32
9.74
5.45
37.93
25.08
ViLT
AdamW Dropout
64
69.66
0.019
10.44
16.63
12.51
8.44
38.49
26.18
VarVQA
64
69.71
0.019
13.81
19.68
12.93
10.88
39.53
27.15
AdamW
1
73.60
0.041
10.35
18.55
15.59
8.65
47.93
33.40
BEIT-3 VarVQA mean
73.84
0.039
14.08
21.98
16.72
11.36
49.57
34.80
base
AdamW Dropout
64
73.46
0.019
13.07
20.11
16.61
9.44
47.49
33.36
VarVQA
64
73.79
0.018
18.10
24.66
19.26
13.90
49.76
35.22
AdamW
78.59
0.039
21.63
32.15
26.31
17.80
63.19
45.83
BETT-3 VarVQA mean
78.96
0.035
25.32
35.35
28.31
21.25
64.83
47.43
large
AdamW Dropout
64
78.11
0.018
25.28
34.52
27.99
20.65
63.00
16.23
VarVQA
64
78.89
0.018
28.13
37.05
29.56
23.21
64.68
48.06
4.4 Mixed ID/OOD Experiments
Following (Dancette et al., 2023), we use VQAv2 (Goyal et al., 2017) and AdVQA (Sheng et al., 2021) ID and OOD datasets, respectively. Both datasets use COCO images (Lin et al., 2014), but AdVQA has a different multimodal distribution (more challenging questions). We use the splits from (Dancette et al., 2023), which draw testing data from Pmix; where
P Pmix aPOOD + (1 PID.
(15)
using PID VQAv2 and PooD AdVQA. Different mixtures are obtained by varying a E [0, 1]. Figure 5 shows the results for BETT-3 large. Although the accuracy drops equally fast for all methods, Variational VQA remains better calibrated (Fig. 5b). The decline in C@1% is equal in absolute numbers (Fig. 5c), but this implies that the relative performance of VarVQA vs. AdamW is increasing higher OOD fractions.
9

---

Published in Transactions on Machine Learning Research (04/2026)
Table 2: Reliability evaluation on NLVR2 for fine-tuned models, The variable denotes the number of forward passes. Best results per model are bold.
Selective Prediction
Sel. Prediction lou-stakes
Calibration
Model
Method
N
Acc.
high-stakes
ECE (L)
Ca
Ca1%
450
100
C95%
410
AdamW
83.45
0.059
6.42
11.61
4.58
2.24
54.79
26.18
BEIT-3
VarVQA mean
1
83.28
0.058
5.15
15.58
6.44
1.41
55.66
27.30
base
AdamW Dropout
64
83.18
0.016
9.98
15.99
6.95
2.95
55.43
27.63
Var
64
83.11
0.031
15.42
23.36
11.20
5.00
57.16
29.23
AdanW
1
88.34
0.041
16.53
41.14
18.08
9.45
78.53
45.64
BETT-3
VarVQA mean
88.83
0.062
17.15
31.07
15.27
3.57
80.17
45.02
large
AdamW Dropout
64
88.11
0.017
33.21
44.69
23.43
14.71
76.99
46.55
VarVQA
64
89.26
0.029
32.89
49.24
25.56
14.85
82.11
49.51
Table 3: Comparison of VarVQA to Deep Ensembles (Lakshminarayanan al., 2017), applied to AdamW- trained models and top of VarVQA; both VQAv2. The ensembles use three models each.
Selective Prediction
Sel. Prediction low-stakes
Calibration
Model
Method
Acc.
high-stakes
ECE (4)
C 2 %
C@1%
P
2 100
C'05%
1
AdamW Ensemble
69.69
0.049
6.97
12.30
9.97
3.67
37.86
25.14
ViLT
VarVQA
69.71
0.019
13.81
19.68
12.93
10.88
39.53
27.15
VarVQA Ensemble
70.08
0.018
14.39
20.09
14.88
10.65
40.51
27.57
AdamW Ensemble
74.70
0.018
15.84
23.66
19.19
11.82
51.33
36.65
BETT-3
VarVQA
73.79
0.018
18.10
24.66
19.26
13.90
49.76
35.22
base
VarVQA Ensemble
74.18
0.015
18.34
25.70
19.27
10.98
51.16
36.01
AdamW Ensemble
79.45
0.020
26.50
36.95
30.51
19.78
66.25
48.84
BETT-3
VarVQA
78.89
0.018
28.13
37.05
29.56
23.21
64.68
48.06
large
VarVQA Ensemble
79.14
0.015
28.68
37.97
30.40
23.91
65.56
48.52
Thus, there is reason to believe that Variational VQA may be fundamentally more robust to OOD data than AdamW-trained models. The results for the other models and metrics are in Appendix D.
4.5 Beyond Predictive Averaging
We compare the performance of our novel selector ЛвPA (cf. Soc. 3.3) to the baseline die (ef. Soc. 3.2). The full results are shown Tables and 5. For the high-stakes selective prediction metrics, ΦвPA consistently outperforms the sample averaging of p achieving e.g. 5% higher C@1% % on NLVR2 for BETT-3 base. For the mostly saturated low-stakes selective prediction metrics (grayed), there is no clear winner. When using MC Dropout, did not find any systematic improvement of ДвPA over D
4.6 Qualitative Results
We show qualitative examples that highlight the difference in uncertainty estimates between AdamW and Variational VQA in Figures and 7. Further qualitative examples for VQAv2, AdVQA and NLVR2, including failure cases, can be found Appendix E. As the accuracy of the AdamW- and IVON-trained models is
10

---

Published in Transactions on Machine Learning Research (04/2026)
37.5
22.5
48
24
35.0
332.5
8
Coverage
40
820
Coverag
17.5
30.0
32
16
15.0
27.5
24
12
1
16
64
16
64
16
64
1
16
64
MC samples
MC samples
MC samples
MC samples
(a) BETT-3 large, VQAv2
(b) BETT-3 large, NLVR2
24
14
Coverage @ 1%
25
22
20
9
Coverage coverage
20
15
6
18
10
3
8
0
1
MC samples
MC samples
MC samples
MC samples
(c) BEIT-3 base, VQAv2
(d) BEIT-3 base, NLVR2
VarVQA
VarVQA mean
AdamW Dropout
AdamW
Figure Comparison of Variational VQA MC Dropout, which uses the same inference compute, on high-stakes selective prediction. Error bars and shaded regions indicate standard error across three seeds.
80
19
Accuracy (96)
75
0.12
Coverage @
70
ECE
0.09
24
100
0.06
16
-20
0.03
-40
0.1
0.33 0.5
0.67
0.33
0.67
0.67
0.1
0.5 0.67
AdVQA fraction
AdVQA fraction
AdVQA fraction
AdVQA fraction
(a) Accuracy
(b) Calibration (L)
(c) Selective Prediction (d) Selective Prediction
+ VarVQA
T VarrQA mean
AdamW
Figure Accuracy, calibration and selective prediction results for different VQAv2/AdVQA mixtures for BEIT-3 large. Error bars indicate standard error across three seeds. In (d), every model in the gray area is worse than a model that abstains every input.
similar, we focus on cases where they prodict the same answer, as this reflects the typical behavior. The key improvement of VarVQA lies not in better accuracy, but rather in improved uncertainty estimates. A further study that investigates the behavior on the different. question categories of VQ,A22 and AdVQA (Binary, Number, and Other), can also found Appendix E.
5 Discussion
In this work, explore Variational VQA, í.e. the application of Variational Learning for multimodal tasks. Our implementation replaces the standard AdamW optimizer with the IVON method and uses multiple
11

---

Published in Transactions on Machine Learning Research (04/2026)
Table Comparison of our risk-averse selection function ДвPA (Eq. (8) against KMP on VQAv2 with VarVQA (N 64 samples as always). Best results per model are bold.
high-stakes
low-stakes
Dataset
Model
Selector
1 C@
2
%
C@01%
P50
P100
C@5%
P10
SMP
13.35
19.24
13.04
10.05
39.52
26.64
ViLT
9BPA
13.81
19.68
12.93
10.88
39.53
27.15
VQAv2
17.15
23.87
18.64
12.23
49.91
35.17
BETT-3 base
JBPA
18.10
24.66
19.26
13.90
49.76
35.22
KMP
27.09
36.00
28.82
22.14
64.82
47.58
BEiT-3 large
9BPA
28.13
37.05
29.56
23.21
64.68
48.06
Table 5: Comparison of our risk-averse selection function 9BPA (Eq. (8)) against IMP NLVR2 with VarVQA (N 64 samples always). Best results per model are bold.
high-stakes
low-stakes
Dataset
Model
Selector
Ca 1 %
C@1%
$
1000
C@5%
410
2
GMP
10.64
22.20
9.75
3.95
57.18
29.28
BEiT-3 base
NLVR2
9BPA
15.42
23.36
11.20
5.00
57.16
29.23
27.61
48.16
24.26
13.59
82.16
49.51
BETT-3 large
9BPA
32.89
49.24
25.56
14.85
82.11
49.51
ples from the learned posterior at inference. In addition, new selector goes beyond the standard predictive averaging by incorporating the output's variance into the abstention decision. Our findings demonstrate that Variational VQ. has two possible applications: when inference costs should be minimal, parameter means can be used at inference to at least match the accuracy of AdamW and decently increase reliability. When higher inference costs are acceptable, multiple MC samples from the posterior can be used. Better reliability is demonstrated by better calibration well as better selective prediction, both in distribution for multiple tasks, and in the challenging mixed ID/ ID/ODD setting. The novel selector further improves selective prediction in high-stakes settings with almost no computational overhead.
Variational VQA also has some limitations, particularly involving hyperparameter tuning with IVON. While we observe correlations between the critical hyperparameters (discussed in the Appendix), which can be exploited to reduce the search space, tuning still remains more involved than with AdamW. Additionally, while VarVQA makes large gains in high-stakes selective prediction vs. AdamW, overconfidence still remains an issue, and Coverages remain well below the theoretical optimum (xe Ace. for low risks). Thus, more work is needed to make models truly 'know what they do not know!
An exciting avenue for future work is to avoid the computational burden of sampling for VarVQA by variance propagation in one forward pass. Recently, Li al. (2025) proposed new method in this domain that has shown promising results for unimodal tasks with IVON. Such 'streamlining' is only possible if learned parameter variances are available, which is not the case for e.g. MC Dropout. While Variational VQA intrinsically improves reliability, the incorporation of previous methods through e.g. training (variational) selector on top of the (variational) model, could also further enhance reliability. Improvements could also obtained by using more expressive posterior that uses full covariance, however at the time of writing, there is no practical alternative to IVON for large models that uses full covariance.
12

---

Published in Transactions on Machine Learning Research (04/2026)
How many slices cut?
Is the man gloveless? (Ground Truth: yes)
anyone touching this umbrella?
(Ground Truth: 9)
(Ground Truth: yes)
AdamW
7 0.986
(0.987) wrong
(0.989) > wrong
(0.991) wrong
VarVQA JMP
= 0.976
(0.593) a abstain
no (0.679) > abstain
no (0.708) abstain
VarVQA JAPA
0.957
(0.486) > abstain
no (0.568) > abstain
no (0.544) + abstain
Figure 6: Qualitative examples on VQAv2 with BETT-3 large where AdamW wrong while VarVQA abstains. The abstention thresholds y were determined by optimizing P100 on VQAv2 validation data. Model answers are displayed in bold, the corresponding answer confidences provided in brackets.
In least one image there is green bookshelf with 7 shelves full of books (Ground Truth: False)
At least one television hangs on a wall near some simple paintings (Ground Truth: False)
AdamW 7 0.994
True (0.995) > wrong
True (0.996) wrong
VarVQA tvr 7 0.994
True (0.692) > abstain
True (0.782) abstain
VerVQA JDPA 7 = 0.992
True (0.326) > abstain
True (0.496) abstain
Figure 7: Qualitative examples on NLVR2 with BETT-3 large where AdamW wrong while VarVQA abstains. The abstention thresholds 7 were determined by optimizing 1000 on NLVR2 validation data. Model answers are displayed in bold, the corresponding answer confidences are provided in brackets.
Acknowledgements. This research was partially funded by an Alexander von Humboldt Professorship in Multimodal Reliable AI, sponsored by Germany's Federal Ministry for Research, Technology and Space and by a OOEWE-Spiteen-Professt
Mohammad Emtiyaz Khan was supported by the Bayes duality project, JST CREST Grant Number JPMJCR2112. The work has benefited from the Excellence Cluster "Reasonable A1" by the Deutsche Forschungsgemeinsshaatt (DFG, German Rescarch Foundation) under Germany's Excellence Stratogy EXC-3057. For compute, we grate- fully ackuowledge support from the hessian.Al Service Center (funded by the Federal Ministry of Research, Technology and Space, BMFTR, grant no. 161S22091) and the hessian.Al Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S-DIW04/0013/000
LLOEWE/4a/5511 /519//99000000 (555000002(0010)/900
References
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom- Up and Top-Down Attention for Image Captioning and Visual Question Answering. In IEEE Conference Compuler Vision and Pollern Recognition (CVPR), 2018.
13

---

Published in Transactions on Machine Learning Research (04/2026)
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015.
Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samuel White, and Tom Yeh. VizWiz: Nearly Real-Time Answers to Visual Questions. In UIST, 2010.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight Uncertainty in Neural Network. In International Conference on Mochine Learning (TCML), 2015.
Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan Ô. Arik, Tomas Pfister, and Somesh Jha. Adaptation with Self-Fxaluation to Improve Selective Prediction in LLMss In EMNLP (Findings), 2023.
Chi-Keung Chow. An Optimum Character Recognition System Using Decision Functions. IRE Transactions on Electronic Computers, 4(BC-6))247-254, 1957.
Beau Coker, Wessel P. Bruinsma, David R. Burt, Weiwei Pan, and Finale Delei-Velez. Wide Mean-Field Bayesian Neural Networks Ignore the Data. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
Bai Cong, Nico Daheim, Yuesong Shen, Rio Yokota, Mohammad Emtiyaz Khan, and Thomas Möllenhoff. Improving LoRA with Variational Learning. arXvv preprint ar Xiv:2500 14280, 2025.
Nico Daheim, Clara Meister, Thomas Möllenhoff, and Iryua Gurevych. Uncurtainy--ware Decoding with Minimum Bayes Risk. International Conference on Learning Representations (TCLR), 2025.
Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, and Marcus Rohrbach. Improving Selective Visual Question Answering by Learning from Your Peers. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Erik A. Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig, Laplace Redux- Effortless Bayesian Deep Learning. In Advances in Newral Information Processing Systems (NeurIPS), 2021.
Ran Ylnia and Yair Wiener. On the Foundations of Noise-Free Selective Classification. J. Mach. Learn. Res. (JMLR), 11:1605 1641, 2010.
Andrew Y. K. Foong, David R. Burt, Yingzhen Li, and Richard Turner. On the Expressiveness of Approximate Inference in Bayesian Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In International Conference on Machine Learning (ICML), 2016.
Yonatan Geifman and Ran El-Yaniv. Selective Classification for Deep Neural Networks. Advonces in Newral Information Processing Systems (NeurIPS), 2017.
Yonatan Geifman and Ran EI- Yamiv. SelectiveNet: A Deep Neural Network with an Integrated Reject Option. In International Conference on Machine Learning (ICML), 2019.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Alex Graves. Practical Variational Inference for Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On Calibration of Modern Neural Networks. In International Conference on Machine Learning (ICML), 2017.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging Weights Leads to Wider Optima and Better Generalization. In Conference on Uncertainty in Artificial Intelligence (UAT), 2018.
Adam Tauman Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang, Why Language Models Hallucinate. 2025.
14

---

Published in Transactions on Machine Learning Research (04/2026)
Amita Kamath, Robin Jia, and Percy Liang. Selective Question Answering under Domain Shift. In Annual Meeling of the Association for Computational Linguislics (ACL), 2020.
Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and Scalable Bayesian Deep Learning by Weight- Pertur-Pettu in Adam. In International Conference on Machine Learning (ICML), 2018.
Wonjae Kim, Bokyung Son, and Ildoo Kim. VILT: Vision-and-Languuae Transformer Without Convolution or Region Supervision. International Conference on Machine Learning (ICML), 2021.
Diederik P Kingma. Adam: A Method for Stochastic Optimization. ar preprint arXiv: urXiv:1412.6988/ 2014.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In Advances Neural Information Processing Systems (NeurIPS), 2017.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. LLa VA-Nex- Interleave: Tackling Mulli-Image, Video, and 3D in Large Multimodal Models. arXiv preprinl arXio:2407 07895, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: Bootstrapping Language-Imagee Pre-training
with Frozen Image Encoders and Large Language Models. In International Conference Machine Learving (ICML), 2023.
Rui Li, Marcus Klasson, Arno Solin, and Martin Trapp. Streamlining Prediction in Bayesian Deep Learning. In International Conference on Learning Representations (ICLR), 2025.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In European Conference Computer Vision (ECCV), 2014.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Leorning Representations (ICLR), 2019.
David J. C. MacKay. Information- Based Objective Functions for Active Data Selection. Newral Comput., 4(4): 800-604,11 1992.
Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing Conversational Agents' Overconfidence Through Linguistic Calibration. Annual Meeting of the Association for Computational Linguisties (ACL), 10:857- 872, 2022.
Erum Mushtaq, Zalan Fabian, Yavuz Faruk Bakman, Anil Ramakrishna, Mahdi Soltanolkotabi, and Salman Aves- timehr. HARMONY: Hidden Activation Representations and Model Oulput-Aware Uncertainty Estimation for Vision-Language Models. In IEEE Conference on Compuler Vision and Pallern Recogriition (CVPR), 2025.
Kazuki Osawa, Siddharth Swaroop, Mohammad Emtiyaz Khan, Anirudh Jain, Runa Eschenhagen, Richard E. Turner, and Rio Yokota. Practical Deep Learning with Bayesian Principles. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht, Obtaining Well Calibrated Probabilities Using Bayesian Binning, In AAAI Conference Artificial Intelligence (AAAI), 2015.
John Platt al. Probabilistic Outputs for Support Vector Machines and Comparisons Regularized Likelihood Methods. Advances in Large Margin Classifters, 10(3):61-74, 1999.
John W Pratt. Risk Aversion in the Small and in the Large. In Uncertainly in Economics, pages 59-79 Elsevier, 1978.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. OpenAl I blog, 1(8);9, 2019.
Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Cian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas Möllenhoff. Variational Learning is Effeective for Large Deep Networks. In International Conference Machine Learning (ICML), 2024.
15

---

Published in Transactions on Machine Learning Research (04/2026)
Sasha Sheng, Amaupreet Singh, Vedanuj Goswami,, Jose Alberto Lopez Magana, Trislan Thrush, Wojciech Galuba, Devi Parikh, and Douwe Kiela. Processing Systems (NeurIPS), 2021.
Visual Question Answering. In Advances in Neural Information
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, Sculley, Joshua V. Dillon, Jie Ren, and Zachary Nado. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. In Advonces Neural Information Processing Systems (NeurIPS), 2019.
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, and Khyathi Raghavi Chandu. Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning. In ACL (Findings), 2024.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A Corpus for Reasoning about Natural Language Grounded in Photographs. In Annual Meeting of the Association for Compulational Lánguistics (ACL), 2019.
Brian Trippe and Richard Turner. Overpruning in Variational Bayesian Neural Networks. arXiv preprint arXi::801.0620 2018
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-Vi VL: Enhancing Vision-Language Model's Perception of the World Any Resolution. arXiv preprint arXiv:2409 12191, 2024.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Lin, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly. Tn Europeon Conference on Computer Vision (ECCV), 2022.
Weihao Xuan, Qingcheng Zeng, Heli Qi, Junjue Wang, and Naoto Yokoya. Sveing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models. oi preprint arXiv:2505. 20236. 2025.
Liu Ziyin, Zhikang Wang, Paul Pu Liang, Ruslan Salakhutdinov,, Louis-Philippe Morency, and Masahito Ueda. Deup Gamblers: Learning to Abstain with Portfolio Theory. Advances in Newral Information Processing Systems (NeurIPS), 2019.
16


Reviews:
[{"review_id": "xWR2WFE9Ne", "reviewer": "Reviewer_X9RQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Variational VQA (VarVQA), which introduces variational Bayesian learning into vision-language models by leveraging the IVON optimizer to learn a posterior distribution over model parameters. This enables intrinsic uncertainty modeling without sacrificing accuracy or requiring additional model components. Through extensive experiments on visual question answering and visual reasoning tasks, the authors show that VarVQA substantially improves calibration and selective prediction performance, with particularly strong gains in high-stakes settings where error tolerance is low. Moreover, the proposed approach is more sample-efficient than MC Dropout and demonstrates increased robustness under distribution shift. Overall, this is a novel and promising work that leverages variational inference to advance multimodal language models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. The paper provides comprehensive experiments across multiple VLM architectures, datasets, and evaluation metrics, showing consistent improvements in calibration, selective prediction and sample efficiency over baselines such as AdamW and MC Dropout.", "audience_yn": "Yes", "audience_explanation": "Yes. The proposed method improves the reliability of multimodal large language models for VQA, a very interesting and relevant topic in ML domain.", "requested_changes": "- Although the BPA selector $\\mu - \\sigma\\$ is empirically effective, can the author provide a more intuitive explanation of why it works?\n- ViLT and BEiT-3 are not the most commonly used VLMs today; why the author choose these model, and do you considering more modern VLM models such as LLaVA-1.6, or Qwen2.5-VL, etc?\n- Section 5.3 seems did not include the discussion of Figures 3(b), 3(c), and 3(d).", "broader_impact_concerns": "The paper aims to improve the reliability and trustworthiness of multimodal large language models through better uncertainty estimation and selective prediction. The proposed approach is intended to reduce overconfident errors rather than enable new capabilities, and no significant negative societal or ethical impacts are foreseen.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a9OmDCkLjF", "reviewer": "Reviewer_GBnG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Positive Aspects\n\n\t-- New Application of Bayesian Methods: This paper successfully applies a variational Bayesian approach (the IVON optimizer of Shen et al., 2024) to large Vision-Language Models (VLMs) for Visual Question Answering. The Variational VQA (VarVQA) framework learns a distribution (mean and variance) over model weights, enabling intrinsic uncertainty estimation without additional model components. The authors show that variational training can be applied to a state-of-the-art VQA transformer (BEiT-3) with no loss in accuracy or significant training overhead. Further, the central strength of the work lies in significantly improved selective prediction under strict error tolerances.VarVQA also consistently outperforms Monte Carlo dropout when both use the same number of inference samples. VarVQA produces far more trustworthy VQA systems, especially in scenarios where mistakes are very costly. Additionally, the paper shows that their BPA selector outperforms conventional max-prob on the variational model, especially in high-stakes regimes.\n\n\t-- Improved Calibration and Reliability: Models trained with VarVQA are markedly better calibrated than those with standard training. The paper shows VarVQA achieves a lower Expected Calibration Error (ECE) than even post-hoc calibration techniques like temperature or vector scaling. For example, VarVQA yields a smaller ECE on VQAv2 compared to a vector-scaled baseline, while matching the calibration performance of Monte Carlo dropout. This is impressive, as calibration typically requires extra tuning, whereas VarVQA provides well-calibrated probabilities out-of-the-box. Empirically, just using the mean weights of a VarVQA model (no sampling) already produces lower ECE and higher selective accuracy than a conventional model with the same inference budget.\n\n\t-- Risk-Averse Selection Function: Another positive aspect is the introduction of a simple yet effective risk-averse selection strategy (g_BPA  that leverages the output variance. Instead of using only the mean predicted probability for confidence (as in standard max-prob or predictive averaging), the authors subtract the model’s uncertainty (standard deviation) for the top answer which penalizes answers with high posterior variance. The paper shows that this BPA selector outperforms conventional max-prob on the variational model, especially in high-stakes regimes.\n\n\t-- The study is mostly empirically rigorous except for limited baselines. The authors evaluate across multiple datasets – VQAv2 for in-domain VQA, NLVR2 for visual reasoning, and a mixture of VQAv2 with AdVQA for a distribution shift scenario. They test multiple different model scales (from a smaller ViLT model to BEiT-3 base and large), demonstrating that VarVQA’s benefits hold across architectures and sizes, on key standard metrics. \n\nNegative Aspects\n\n\t-- Baseline Comparisons could be expanded: While the paper compares against Monte Carlo dropout and standard training, it omits some other competitive uncertainty estimation approaches. In particular, it would strengthen the work to compare against deep ensembles, which are a known standard for uncertainty (albeit they are costly but for uncertainty calibration this might be somewhat less relevant — nonetheless the tradeoff would be interesting to the community). The authors also confess that ensembling can yield   excellent calibration but is resource-intensive; however, a small-scale ensemble experiment (even with fewer models or smaller backbones) could quantify how VarVQA measures up to that sort of like an “upper bound” — I think this tradeoff would add a lot of value to the paper. Similarly, other prior VQA-specific selective prediction methods (e.g. the head-based confidence model of Whitehead et al., 2022, or the LLM+vision-tool approach of Srinivasan et al., 2024) are discussed in related work but not directly compared in experiments. This raises the question if the paper’s improvements are mainly due to the threshold selection mechanism  ... rather than just the Bayesian training itself – a direct empirical contrast with those methods (or a combination thereof) would bee illuminating.", "claims_and_evidence_yn": "Yes", "claims_explanation": "i would say mostly yes, but the set of baselines could have been expanded (see strengths/weaknesses above)", "audience_yn": "Yes", "audience_explanation": "the paper makes a noteable contribution of using bayesian methods for multimodal calibration which i feel several tlmr audiences will be interested in.", "requested_changes": "-- More baselines (see weaknesses above)\n\nQuestions:\n-- The authors cite past work on the shortcomings of variational Bayes for large networks (e.g. over-pruning effects), yet the paper does not deeply discuss whether such issues arose in their experiments. Could the authors clarify ? Did the authors notice any cases of the variational model becoming overconfident or underconfident due to the limitations of the approximation? A discussion of potential failure modes (e.g. if the variances collapse prematurely e.g. see Coker et al. 2022 on wide BNNs) would help understanding of the method’s robustness.\n\n-- Behavior on Unanswerable or Out-of-Distribution Questions: How does VarVQA handle cases where the question cannot be answered from the image? Intuitively, a well-calibrated model should abstain ? Did the authors notice something like this ?\n\n-- The method uses a diagonal Gaussian posterior for practicality. Do you think a more expressive posterior (like a full-rank covariance or a mixture) would significantly improve results, or is mean-field sufficient here? In other words, are the current bottlenecks more due to, say, data/label noise and model capacity, or due to the approximate inference? If you have insights or evidence (perhaps from the Foong et al. 2020 expressiveness perspective), it would be great to discuss whether pursuing richer posteriors might yield marginal gains or major improvements in selective performance.\n\n-- Why did you particularly choose this selective technique instead of something like confidence head? \n\nMinor: there are a couple of typos:\n\nCalibraion– Missing  “t” \n\n“BEiT-3 s near-SOTA” \n\n“framework Chow (1957)” —> “framework by Chow (1957)” or “framework (Chow, 1957)” or may be the authors wanted to use \\citep instead of \\citet\n\nReference: “Weight uncertainty in neural network.” —> “neural networks”", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s5e9OYV7uv", "reviewer": "Reviewer_ZqNN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Variational VQA (VarVQA), a framework for uncertainty estimation in multimodal vision–language models by training with the IVON variational optimizer, so that the model learns parameter variances that can be leveraged for uncertainty-aware selective prediction. The authors claim to be the first to apply IVON to VLMs, and they provide empirical evidence of improved uncertainty behavior.\n\nA second contribution is a simple risk-averse selection rule that adjusts confidence using predictive dispersion—specifically, subtracting the (top-class) standard deviation from the mean confidence—which improves high-stakes selective prediction in their experiments.\n\nStrengths. (1) The paper targets a clear and timely problem (overconfidence/hallucination and selective prediction in VQA). (2) Experiments across multiple backbones show comparable accuracy while improving high-stakes selective prediction, and often improving calibration. (3) The compute-controlled comparison against MC Dropout is a useful and practically relevant baseline.\n\nConcerns / weaknesses. (1) The novelty may feel primarily empirical/system-level: IVON itself is not new, and the proposed selector is simple and lacks a strong theoretical connection to the selective-prediction objective beyond intuition. (2) IVON appears to introduce additional hyperparameters and tuning complexity relative to AdamW, raising questions about ease of adoption and robustness across settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, most claims are supported by experimental evidence on standard datasets using established metrics for calibration and selective prediction.", "audience_yn": "Yes", "audience_explanation": "The topic is of broad interest to the TMLR audience, and the results suggest that variational training can transfer effectively to multimodal settings and potentially improve reliability. That said, the current algorithmic design and analysis remain somewhat intuitive, and the paper could do more to provide deeper (theoretical/mathematical) insight into why the approach improves uncertainty estimates and under what conditions it may fail.", "requested_changes": "Clarify novelty and positioning: Be explicit that the primary novelty is applying IVON to VLMs plus the selective-prediction selector, and more clearly distinguish what is inherited from IVON versus what is new in this work.\n\nQuantify training overhead: Report wall-clock training time and memory comparisons for AdamW vs. IVON.\n\nEmphasize results that choose the selection threshold $\\gamma$ on a validation set, and/or report how well a validation-chosen  $\\gamma$ transfers to test risk/coverage, since test-time threshold selection can overstate deployability.\n\nFinally, presentation and claims should remain appropriately conservative regarding real-world reliability improvements.", "broader_impact_concerns": "Abstention improves safety but may introduce disparate coverage across groups and shift risk/workload to downstream humans. If trained on user-provided images/questions, the system also raises privacy concerns around sensitive visual data handling and potential memorization, and multi-sample inference increases compute/energy costs.", "additional_comments": "The topic is relevant, but the contribution currently feels very incremental: IVON is existing work and the proposed selector is a simple heuristic. The experiments are suggestive but lack deeper analysis and deployable evaluation protocols.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)