Knowledge Exchange with Confidence: Cost-Effective LLM Integration for Reliable and Efficient Visual Question Answering
Authors: Mahsa Mozaffari
Hitesh Sapkota
Xumin Liu
Qi Yu
Venue: ICLR 2026 Poster (2026)
Keywords: visual question answering
model calibration
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent advances in large language models (LLMs) have improved the accuracy of visual question answering (VQA) systems. However, directly applying LLMs to VQA still presents several challenges: (a) suboptimal performance when handling questions from specialized domains, (b) higher computational costs and slower inference speed due to large model sizes, and (c) the absence of a systematic approach to precisely quantify the uncertainty of LLM responses, raising concerns about their reliability in high-stakes tasks. To address these issues, we propose an UNcertainty-aware LLM-Integrated VQA model ($\texttt{Uni-VQA}$). This model facilitates knowledge exchange between the LLM and a calibrated task-specific model (\ie \texttt{TS-VQA}), guided by reliable confidence scores, resulting in improved VQA accuracy, reliability and inference speed. Our framework strategically leverages these confidence scores to manage the interaction between the LLM and $\texttt{TS-VQA}$: the specialized questions are answered by the $\texttt{TS-VQA}$ model, while general knowledge questions are handled by the LLM. For questions requiring both specialized and general knowledge, the $\texttt{TS-VQA}$ provides candidate answers, which the LLM then combines with its internal knowledge to generate a more accurate response. Extensive experiments on VQA datasets demonstrate the theoretically justified advantages of $\texttt{Uni-VQA}$ over using the LLM or $\texttt{TS-VQA}$ alone.

Published as a conference paper at ICLR 2026
KNOWLEDGE EXCHANGE WITH CONFIDENCE: COST- EFFECTIVE LLM INTEGRATION FOR RELIABLE AND EFFICIENT VISUAL QUESTION ANSWERING
Mahsa Mozaffari I, Hitesh Sapkota 2; Xumin Liu 1 Qi Yu 1.2+
Rochester Institute of Technology
mmozaffari prrzafffrr@@mall @mail rit.riiteee. edu, {xumin. qi.yujerit.edu
2 Amazon
sapkoh@ amazon. com
ABSTRACT
Recent advances in large language models (LLMs) have improved the accuracy of visual question answering (VQA) systems. However, directly applying LLMs VQA still presents several challenges: (a) suboptimal performance when handling questions from specialized domains, (b) higher computational costs and slower inference speed due to large model sizes, and (c) the absence of systematic ap- proach to precisely quantify the uncertainty of LLM responses, raising concerns about their reliability in high-stakes tasks. To address these issues, propose UNcertainty-avee aware L.M-IIntegrated VQA model (Uni-VQA). This model facili- tates knowledge exchange between the LLM and calibrated task-specific model (i.e., TS-VQA), guided by reliable confidence scores, resulting in improved VQA accuracy, reliability, and inference speed. Our framework strategically leverages these confidence scores to manage the interaction between the LLM and TS TS-VQA: the specialized questions are answered by the TS-VOA model, while general knowledge questions are handled by the LLM. For questions requiring both spe- cialized and general knowledge, the TS-VQA provides candidate answers, which the L.LM then combines with its internal knowledge to generate a more accurate response. Extensive experiments on VQA datasets demonstrate the theoretically justified advantages of Uni -VQA over using the LLM or TS -QQ alone.
INTRODUCTION
Recent advances in Large Language Models (LLMs) have opened new opportunities to enhance Visual Question Answering (VQA) performance by leveraging the rich general knowledge these models acquire through large-scale pre-training. LLMs consistently achieve higher accuracy on VQA tasks compared to traditional task-specific VQA models (TS-VOA), which are smaller models trained specifically for visual question answering. However, fully relying on LLMs for VQA faces critical practical challenges that limit their real-world deployment.
The primary challenge is computational efficiency. LLMs typically require billions of parame- ters, resulting in prohibitive computational overhead, high financial costs, and significant inference latency. These limitations become critical in time-sensitive applications (Ding al.) (025) and resource-constrained environments. Moreover, recent studies show that multi-purpose LLMs can be orders of magnitude more expensive to operate than task-specific models during inference. Environ- mental concerns add another layer of complexity, as large-scale models contribute substantially to carbon emissions and energy consumption (Strubell et al. (2200) Bommasani al.) 2021 Weidinger et al. 2022 Wu et al., 2022). Additionally, relying on third-party LLMMs introduces recurring costs, and potential data privacy risks.
However, this computational burden may be unnecessary for many questions. Not all visual ques- tions require the full power of massive language models smaller TS-VQA models can effectively handle simpler queries while consuming significantly less computational resources. Furthermore,
Work related to Amazon.

---

Published as a conference paper at ICLR 2026
Figure I: Comparison of TS-VOA (VisualBERT), LLM (Mistral-7B), and our hybrid Uni-VQA (73% delegation) in average latency, carbon emission ac- curacy, and ECE (Expected Calibration Error) on
T5-VA4 (ViuarBENT)
una-rgA
LLM (Mistrak-78)
AZT
COCO-QA. ("ECE measures the gap between confi-
dence and accuracy the model; lower ECE better.)
LA 002 Embueens 16)
AUAMAY C11
ESR CU1
(63)
TS-VQA models trained domain-specific data can sometimes provide more accurate answers than LLMs in specialized areas where the LLMs lack sufficient knowledge. Most importantly, our empirical analysis reveals that LLMs and TS-VOA models possess complementary strengths: even when TS-VOA models are uncertain about their final answers, they often generate valuable candi- date answers that, when shared with LLMs, substantially improve LLM performance (as shown in Fig. [b) This suggests an opportunity for a collaborative approach rather than simply choosing between the two model types.
Building on this observation, we propose hybrid framework that enables strategic collaboration between TS TS-VQA models and LLMs. The key insight is to use the TS-VQA model's confidence scores to determine not only when to consult the LLM, but also when and how to transfer specialized knowledge from the TS-VOA model to enhance LLM reasoning. To realize this vision, we introduce Uni-VQA (UNcertainty-awan LLM integrated VQA), novel hybrid framework that intelligently combines TS-VQA models with L.MSs through cofiidnce-basee collaboration. However, standard VQA models trained with cross-entropy loss tend to be overconfident and poorly calibrated, as illus- trated in Fig. pp. They often produce incorrect answers with high confidence scores, meaning these scores cannot reliably indicate when the model is actually correct rendering them untrustworthy for decision-man makiig.. To address this critical issue, we develop a calibration technique that ensures confidence scores accurately reflect the likelihood of correctness, shown in Fig. 22. enabling reliable confidence-based integration within our hybrid framework.
With properly calibrated confidence scores, Uni framework operates through three-tiered knowledge exchange mechanism based on confidence levels. For highly specialized questions where the TS-VQA model exhibits high confidence (meaning answer is very likely be correct), the sys- tem provides answers directly without consulting the LLM, leveraging the model's domain expertise efficiently. For questions requiring broad general knowledge, where the calibrated TVVVQA model shows low confidence, the framework delegates entirely to the LLM. Most importantly, for questions requiring both specialized and general knowledge where the TS-VOA model has partial knowl- edge but remains uncertain our framework enables a novel form of collaboration. The TS-VQa model transfers its specialized knowledge by providing dynamically selected candidate answers to the LLM, which the LLM then incorporates with its general knowledge produce more accurate responses. This collaborative approach leverages the complementary strengths of both model types.
The overall framework during the inference is illustrated in Fig. Zt. By selectively delegating ques- tions that require general knowledge to the LLM, our framework significantly reduces overall com- putation and inference costs, while achieving higher accuracy, compared to using the LLM alone, as shown in Fig. [ Experimental results across multiple VQA datasets show that Uni-VOA out- performs both the L.LM and VS-VQA used in isolation, while dramatically reducing computational overhead. Our contributions are summarized as follows:
We develop a calibration technique that enables TS-VQA models to provide reliable confidence estimates essential for effective confidence-based hybrid integration. We introduce UNcertainty-awan LLM integrated VQA model (Uni-VQA) that enables cost- effective knowledge exchange between LLMs and calibrated TS-VOA models, improving both accuracy and efficiency through strategic collaboration.
We provide theoretical analysis to justify the calibration benefits of our diverse ensemble, and demonstrate substantial improvements in both accuracy and efficiency across multiple TS-VOA models and VQA datasets.
RELATED WORK
Visual Question Answering. To tackle the complex VQA problem, various methodologies have
been developed (Schwenk et al. 2022 Lin et al.) 2022 Gao et al.) 2022 Qian et al. 2022) To
Carbon emissions of models are estimated using https: http//gtttuuutt wtw.tthwb...omm.mo cwmmmlmmmm.mmmmmmmm..mmmm.... co2 cot.coocccccccooo////oo
2

---

Published as a conference paper at ICLR 2026
Q: what is animal?
Low
Raject VON'S snower
LLM Accurssyy
72
1.00
1.50
Ask leacher
68
70-
Viuu
Patl
00.5
30.50
Acc 20.
VQA
Moderate
Verity the aEswe:
66
0.25
0.25
2 K/Numm, Answer Candidatesl
1.4
High
Accept BEswer
Confidencce
Confitience
(a)
(b)
(c)
(d)
Figure 2: (a) Effectiveness of incorporating candidate answers of four TS-VDA models on the performance of Mistral-7B, demonstrating improved accuracy as more knowledge is shared with the I.I.M. (b) and (e) present reliability diagrams
baseline and calibrated "SSVQAS (VisualBERT), respectively, showing how model confidence aligns with actual accuracy (orange bars represent a perfect calibration). (d) General workflow of Uni-VCA during inference, illustrating how confidence levels determine whether to use the VQA model directly, consult an LLM, or rely entirely on the LLM.
enhance the understanding of the context present in the VQA text, attention-basee mechanisms have
been used (Gao et al. 2019; Lu al. 2018, Yu et al. 2019). Pre-training has also been leveraged,
where models are first pre-trained using un unlabeled data and then fine-tuned in the downstream VQA
tasks (Shen et al.] 2021 Alayrac et al. 2022 Zeng et al. 2023 Bao al. 2022 Wang et al. 2023
Large Language Model-based VQA. Due to pre-training and reasoning capabilities of L.LMs, these models provide an implicit knowledge source for the VQA tasks. Yang et al. (2022) use image captions to provide visual context to GPT-3 as an implicit knowledge base for knowledge-based VQA task. Yu et al. (2023) propose framework that prompts LLMs with complementary answer candidates and answer-an aware examples to enhance OK-VQA performance. However, these LLM- based VQA models are inadequate for building reliable, efficient, and cost-effective VQA due to their total reliance on LLMs to address all the questions. The knowledge exchange between the LLM and the TS-VQA is not properly guided, which may lead to sub-optimal performance.
Calibration in VQA. Whitehead et al. (2022) introduced the concept of reliability in VQA, treat- ing it as a selective prediction task. They propose incorporating an additional selection mechanism to determine whether the model should provide an answer or abstain, based on an estimated con- fidence score. Training the selector component requires an additional held-out labeled dataset. To avoid this. Dancette et al. (2023) propose training strategy, which enables training both the VQA model and selector on the same dataset, by obtaining pseudo-labels for training the selector in a distributed manner. While these methods enhance the model prediction reliability by abstaining answers with low confidence, they do not address the issue of poor-calibration and overconfidence phenomenon stemming from memorization effect. Also, abstaining when confidence low limits their use real-world applications where always expect answer.
Retrieval-Auummented Generation. Our framework shares a high-level principle with Retrieval- Augmented Generation (RAG) methods (Guu et al.) 2020 Lewis et al. 2020 Hu et al. 2023 Izac- ard & Grave 202T): both augment LLMs with external modules. However, they target different bottlenecks and complementary. RAG retrieves textual evidence from external corpora (e.g., the web, knowledge bases) to expand LLM knowledge coverage, typically invoking the LLM on every query. In contrast, Uni employs calibrated TS VS-VQA model that provides candidate answers and confidence scores, enabling selective L.MM invocation only for low- and mid-confidence cases. The two frameworks address orthogonal concerns: RAG controls what textual evidence the LLM sees; Uni-VCA controls when and how the LLM is used. Crucially, our diverse ensemble calibra- tion naturally pushes out-of-distribution and knowledge-intent intennsve questions (where TS TS-VQA lacks expertise), toward the lowest-confidence region (further discussion in Appendix G. This creates a natural integration point: questions routed to the LLM without TS TS-VQA candidates are those that would benefit most from RAG augmentation. Thus, Uni -VQA could directly wrap a RAG-enhanced LLM for low-confidence queries without modifying the calibration or routing logic.
3 METHODOLOGY
Assume D = {(v.. qn a > is a dataset consisting of N instances, where each instance com- prises image Vn, question qr and an answer a,, We establish X' = V Q as the input space,
3

---

Published as a conference paper at ICLR 2026
What job does this man have?
Q:
reflection of?
an office?
VQA
Answers X
Answer: bird
Anewer: yes of Conf: (fiict
Conf: (Low)
0.32 (Moderate) bind,
Please answerr the queation
be provided with contect, a quetion about the context, and multple-chllce answeess Your task to select the currect answer based on the contest descriptons.
according to the contest
Bramples:
Accept VQA answer
(eammpley exompity -:]
Examples:: (examppoo la cmmplea....
Contant: (image cuptiins)
Contest:: (image copptonn)
Question:
Question: if hot the reffection of?
Wrut alocr this man hazv?
Answer candidates: (Aini, senynal soueer dunkk oky..")
Answer:
Answer:
answer yes v
LLM as
Frozen LLM
LLM Consultant
Compurational Cost: 44449
45444
unsweriuua barber
GRSWET LAMS sky
Figure 3: General overview of Uni-VOA framework at inference time. LLM serves different roles depending on the VQA's confidence. (a) If the VQA model is least confident, the LLM serves as a teacher and provides the answer the question, (b) If VQA model is confused among multiple candidate answers, the LLM serves as consultant and helps to select answer from candidate answers, (c) if VQA model is highly confident, Uni-VQA directly answers without LLM involvement.
with X,, (v', qn. representing an input data point. Additional concepts utilized in the paper elaborated in the Appendix.
3.1
OVERVIEW OF THE FRAMEWORK
Fig. illustrates the overview of the proposed UniiVQA framework. During the training phase, we first train a well-calibrated TS-VOA model employing a diverse ensemble based approach. This calibration step crucial because reliable confidence scores enable effective integration be- tween the TS-VOA and LLM components during inference. The inference phase operates through confidence-guided process: Initially, the calibrated TS-VQA model generates an initial answer along with its associated confidence score c. Based on this confidence score, the framework routes the query to one of three distinct scenarios defined by confidence thresholds and u. 1) when the TS-VOX exhibits high confidence (c 2 u), the TS-VOA answer is accepted directly without LLM's involvement, leveraging the model's domain-specific expertise efficiently. 2) When confidence low (c < 1, typically for questions requiring broad general knowledge beyond TS-VQA's 's special- ization), the question is fully delegated to the LLM without answer candidates, which we refer to as the LLM as Teacher scenario. 3) For moderate confidence levels (1 < u), where TS-VQA has partial knowledge but remains uncertain, answer candidates of TS-VDV VQA are dynamically selected and provided to the LLM in what call the LLM Consultant scenario, enabling collaborative reasoning where the LLM integrates these specialized insights with its own general knowledge.
This cetiidence-guider delegation mechanism strategically leverages the complementary strengths of both model types. utilizes the TS-VQA's domain-specificc expertise with low computational cost for high-confidence questions, harnesses the LLM''s broad reasoning capabilities for challenging general knowledge queries, and facilitates knowledge exchange through candidate answers when both specialized and general knowledge are needed improve predictive performance.
3.2 RELIABLE VQA VIA MODEL CALIBR ATION
In our Uni-VQA framework, the integration of LLM and TS-VQA models depends critically on the TS-VQA: model's confidence estimates. For optimal integration, these confidence estimates must re- liably indicate answer correctness, i.e., low confidence should signal incorrect answers, while high confidence should signal correct ones. This requires well-calibrated TS-VO2 models where con- fidence estimates accurately align with actual accuracies. However, standard VQA models trained with cross-entropy loss suffer from overconfidence (Fig. 2p). consistently expressing higher confi- dence than their actual accuracies.
4

---

Published as a conference paper at ICLR 2026
Diverse Ensemble Strategy. To address the calibration problem, propose Diverse Ensemble (DE) strategy that creates multiple complementary TS-VOA models, each specializing on different aspects of the data distribution. Deep ensembles
shown to effectively improve model accuracy and calibration (Wilson & Izmailov 2020; I Lakshminarayanan et al. [017] Wood et al. 2023 Sap- kota et al. 2024). partieularly when diversity is enforced among base learners. Our approach lever- ages Distributionally Robust Optimization (DRO) (Duchi & Namkoong) 2019) to train an ensemble of E diverse TS VQ-VQA models that naturally complement each other.
Given training samples { and per-sample loss 1(In, θ) (cross-entropy), we view calibra- tion as learning under an adversarial reweighting of the empirical distribution. For each ensemble member, we minimize a DRO-style weighted loss:
(D)O Σ w,l(xn, +)
where determines the emphasis on each training instance X,. The weight vector is dynami- cally computed at every training step based on the current model's losses. Concretely, we adopt the regularized DRO formulation with KL-divergence, which yields the closed-form softmax weighting (see Appendix C for derivation):
" (A) exp( 0)/2)
ΣE= ())(((
where A > controls how far w deviate from uniform weights, and thus how strongly the model focuses on high-loss (difficult) samples.
By varying the hyperparameter À across ensemble members, we obtain models that specialize on different difficulty regimes. When À is small, the weighting scheme emphasizes challenging sam- ples, producing model that tends be cautious (lower confidence) since it has learned to handle difficult cases. When À is large, the weighting approaches uniform distribution, creating model that captures general patterns and tends be more confident typical samples. In all experiments, use an ensemble of E = 3 models with small, medium, and large À values, creating complementary expertise across the difficulty spectrum (See Appendix C and Appendix G.13 for details).
At inference time, we average the logits of the ensemble members, fDE(X) EE (x(). (where fx( represents the logits from the e-th ensemble member), and obtain the confidence score from the resulting softmax distribution. This combination naturally produces well-calibrated confi- dence scores because the cautious models (trained on hard samples) temper the overeonfidence of models trained on easier samples, while confident predictions from samyls experts are vali- dated across the ensemble.
our diverse ensemble significantly improves calibration by assigning appropriately lower confidence to incorrect answers while main- taining high confidence for correct responses, making the confidence scores a reliable indicator for delegation mechanism.
As demonstrated in Figs. [[]add 12 in the appendix section G.3
3.3
CONFIDENCE GUIDED KNOWLEDGE EXCHANGE
Calibrated VV-VQA confidence scores are critical in selectively delegating challenging questions along with answer candidates to the LLM, not only improving the overall predictive performance but also enabling efficient inference of easier questions by the TS-VQA model. Additionally, the effectiveness of these candidate answers varies significantly across different confidence intervals.
Motivated by this observation, we hypothesize that across different confidence levels, the number of answer candidates the LLM benefits from varies by confidence level. Specifically, for an effec- tive combination of L.LM and VQA by annder-candidate augmentation, fewer answer-candidates are needed at high confidences of T S-VQA, while more candidates become beneficial as the confidences decrease. At lowest confidences, providing a large number of answer candidates is impractical, mak- ing it more effective for the LLM to answer the questions without relying any answer candidates. To validate this hypothesis we compare the LLM's predictive accuracy within each confidence in- terval of the TS-VOA for varying number of answer candidates in top-0, top-1, top-2. and top-10 along with LLM's performance without answer candidates. As Fig. 4p suggests, in higher confi- dence intervals, I.L''s performance higher when fewer answer candidates are presented. As the
5

---

Published as a conference paper at ICLR 2026
K=0
K=1
K=
K= 10
[0.0,
(0.3, 0.4)
(0.5, 0.6)
(0.9,
Besik Leamned function lower bound O7
LLM Accuracy
0.93
0.60-
0.40
0.90
0.10
0.55-
0.88
0.00 0.25 0.50 Confidence
0.75
(a) Dynamic top-k
(b) L.LM accuracy confidence
Figure 4: (a) Learned mapping: confidence to k. (b) Accuracy for & E 1, 10} across confidence bins.
confidence interval decreases, LLM's performance is enhanced when more answer candidates are included. In the lowest confidence intervals, the LLM's performance with answer candidates drops as compared to when no answer candidate is presented.
To that end, we propose a dynamic approach for effective answer candidate selection, informed by the TS-VOA's ' answer confidence. Considering c; max (x, the confidence of the predicted answer ā, We define / as threshold for delegating to LLM with no answer candidates, and u as thresholds for using TS-VQA for question answering. Specifically, if c; 2 then the answer predicted by the TS TS-VOA model i.e. å, is accepted, and if c < I, answering is delegated to LLM, without any answer candidates included in the prompt. For 2 2 answering delegated to LLM provided with K ((G) 2 answer-candidate where K is determined by:
((c:) s [Me-W (e-WW(
where 0 < l u S 1, and learnable parameters M, W are determined based on validation set and [x]] ] is the rounding operation that converts the fractional value into the closest integer. Fig. presents the learned top-k answer candidate selection for Ca1 salibrated CLIP-ViL ViL.
3.4 ACCELERATING INFERENCE WITH KNOWLEDGE DISTILLATION
To further reduce the inference cost, propose leverage knowledge distillation transfer the predictive accuracy and calibration of the diverse ensemble (DE) into a single TS-VQA model with the same architecture as the individual ensemble components. Instead of learning from target labels using cross-entrope loss, the distilled model minimizes the Kullback-Leible divergence between its output distribution and the diverse ensemble's output logits distribution. This approach effec- tively preserves both accuracy and calibration with theoretical guarantees (Allen-Zhu & Lii 2023) Hebbalaguppe et al., 2024) while eliminating the additional computational burden of ensembling- Our experiments show that the distilled model maintains comparable ECE and accuracy (within 0.4%) while reducing latency by up to 60%. Further details and numerical results are provided in Appendix G.9
THEORETICAL ANALYSIS
In this section, we theoretically demonstrate that the proposed Uni Uni-VQA technique effectively del- egates a greater number of incorrect predictions, that would otherwise be confidently wrong. We show this in two steps. First, demonstrate how the diverse ensemble technique improves cali- bration. In the second step, we show that with better calibration, more incorrect samples are shifted into the low-confidence regions, allowing them to be effectively delegated to the LLM for correction. Complete proofs for the theoretical results are provided in Appendix D
Diverse ensemble improves the ECE. In this section, we showcase the lemma demonstrating how diverse ensemble techniques improve the model calibration (i.e., ECE) compared to Expected Risk Minimization (ERM)-based model. Specifically, in the following lemma, we show that the DE loss will be upper bound on the regularized cross-entropy loss where the regularizer is the negative entropy of the predictive distribution p = fx(
Lemma 4.1 Consider ( being the diverse ensemble loss and (0) being the cross entropy
loss for the subnetwork and fr fō (x) being the prediction distribution of the base subnetwork Then we have:
(DE 2
(0) A.H
(3)
em1

---

Published as a conference paper at ICLR 2026
where |E| is the total number of subnetworks used in ensemble, C is the normalization stant of DRO weights, Re is the DRO hyperparameter controlling the balance between CE loss and predictive entropy, and (' being the entropy of the p.
Remark. The Lemma indicates that minimizing the DE loss leads to: (a) minimization of the cross-entropy loss, and (b) an increase in the entropy of the predictive distribution p. Increasing the entropy of the predictive distribution can avoid overconfident predictions produced by the neural network, thereby improving the calibration. As a result our approach will reduce the likelihood of emrors in the high confidence region, ensuring that the incorrec predictions remain in the low confidence regions. These low- cww-oofidence questions are then delegated to the LLM, that provides the final answer with the support of the dynamically selected candidate answers from the TS-VQA.
Diverse ensemble maximizes incorrect sample delegation. Because of the improved calibration achieved through the diverse ensemble technique, our approach shifts more incorrect samples into the low confidence region compared to the ERM-based approach. This is because, ERM tends to produce overconfident prediction for most the samples, causing many wrongly answered samples to fall in the high-come cogfidenfeeeeee region (as shown empirically in Fig. TTp. In contrast, diverse ensemble lowers confidence levels, leading to a higher number of samples in the low-confidence region.
ERM
Diverse Ensemble
Pythia
VILBERT
VisualBERT
CLIP-VL
N
Confidence Threshold (T)
Figure 5: Empirical evidence illustrating Nin. Ni across four VQA architectures.
Theorem 4.2 Let Nog and NT. Esx be the total number of samples belonging the low confidence region R: {p E T]. with T being the threshold defining the low-confidence region. Then, for the region R. the following holds true
N
(4)
where N New are the number of incorrect samples from DRO and ERM, respectively in region R.
Remark. By leveraging the DE-framework, we ensure that the incorrect samples are more likely to be the low-confidence region, as empirically illustrated in Fig. 5 It maximizes the LLM's ability to correct these incorrect answers. In contrast, the ERM-bbased approach frequently assigns high confidence scores to incorrect samples due to overfitting. As a result, the delegation threshold must be set very high to pass these samples to the LLM for correction. This leads to either suboptimal accuracy if threshold is not high enough, or sub-optimal efficiency the threshold is set too high, requiring more frequent delegation to the LLM.
EXPERIMENTS
We evaluated the performance of our Uni Uni-VQA framework on multiple existing VQA architectures and report comparative quantitative results on the VQA-v2 (Antol al.) 2015) and COCO-QA (Ren cet al. 2015) test splits, and conduct extensive ablation studies to justify the effectiveness of various proposed components. This includes effectiveness of (i) diverse sieemble-based VQA, (ii) answer- candidate augmented LLM prompting, and (iii) dynamic answer-candidate selection approach. Due to limited space, have included more experimental results in the section G.9
Baselines. We have considered five baselines. This includes (a) Pretrained-LLMM, (b) TS TS-VQA, (c) VectorScale-based post-hoc calibrated VQA, referred to as VectorScale (Guo et al. 2017), (d) hybrid LLM-VQA confidence threshold based delegation, referred to LLM-VQA, and (e) VQA models with our novel calibration technique denoted as Calibrated. LLM- VectorScale refers to integration of LLM with the VectorScale calibrated VQA. Specifically, in terms of VQA models, consider five TS-VOA models: Pythia (Jiang et al., 2018 CLIP-ViL (Shen et al. 202I). ViLBERT
(Lu et al. 2019) VisualBERT (Li et al. 2019), and BETT3 (Wang al. 2023). Pythia (Jiang al.,
7

---

Published as a conference paper at ICLR 2026
Pythia-LUM
VLSERT-LLM
VisuatBERT-iLMM
CLIVIL-LLM
VOA Accuracy
70-
VVGA (Oum)
68-
022
0.4
3.5
0.5
222
0.4
0.5
0.5
0
0.2
0.4
0.5
0.5
0.2
1.4
0.6
3.5
Delegation thrasholdd
Figure 6: Performance comparison of the proposed method against LLM-VQA, and LLM- VectorScale (thresh- old), with respect the delegation threshold. Accuracy zero delegation is accuracy of TS-VQA model.
PytharLLM
VILBERT-LUM
CLPVIL-LUM
VOA Accuracy
70
LLM-OR torrabals)
60
60
07
LLM
40
55
55
100
22
40
60
80
100
20
40
67
170
0
40
20
100
Delegation percentage
Figure 7: Performance comparison of the proposed method against LLQ. and LLM- VectorScale (thresh- old), with fixed top-10 10 answer candidates, with respect to the delegation percentage.
2018 is a bottom-up top-down model, which leverages the up-down attention mechanism (Anderson) et al. 2018), and combines the representations of question and image by element-wise multiplica- tion. CLIP-ViL (Shen et al., 2021) uses the Movie-MCAN architecture (Nguyen et al., 2020) with the visual encoder of the CLIP (Radford et al.) al.] (222]) pre- pre-training model. ViLBERT (Lu al. 2019)
and VisualBERT (Li et al.) 2019) are pre-training-bar based transformer architectures with attention mechanisms. BEiT-3 is state-of-the-art general-purpose vision-language model trained through masked-data modeling. For LLM-based models, we have employed frozen Mistral-7B (Tiang et al., 2023), and VA-155 13B (Liu et al. 2023) VLLM.
Dataset and evaluation metries. We use VQA-v2 (Antol al. 2015) and COCO-QA (Ren al. 2013) data sets. See section F. for more details. We utilize four metrics for evaluation: (I) VQA accuracy (ACC) to illustrate predictive performance, (2) Expected Calibration Error (ECE) which measures the difference between model confidence and actual accuracy (lower is better, with 0 in- dicating perfect calibration), and is used to assess the reliability, (3) the proportion of questions assigned to LLM (LLM-De Diie--eeeggg leg %) as proxy for computational expense and inference time, re- flecting the extra cost incurred by LLM, and (4) Average latency of inference (Latency) measured in seconds. For complete implementation details refer to Appendix F33
5.1 COMPARISON RESULTS
Fig. 6 compares our approach with the baselines for various delegation thresholds. With the del- egation threshold of 0, none of the questions is delegated to the LLM whereas, as the threshold increases, more questions with lower confidence scores are delegated to the LLM. LLM-only in- dicates the baseline result when we directly answer all questions using LLM. There are two key observations that can be inferred from Fig. 6 First, delegating low-confidence samples to the LLM improves performance across all baselines. including Unii VQA. This improvement can be at- tributed to the LLM's ' ability to handle challenging questions that the TS-VQA models struggle with. Second, due to its superior calibration, coupled with the uncertainty-sennitti dynamic delegation technique, our Un -VQA delegates more incorrect samples to the I.M,, achieving better overall per- formance compared other baselines. This highlights the importance of calibration enhancement and dynamic delegation in hybrid VQA models.
Fig. compares our Uni-VQA with baselines in terms of the VQA accuracy against the LLM- delegation percentage. First, our approach achieves the highest maximum accuracy among the base- lines. At any given fixed delegation percentage, also obtains a higher accuracy than the baselines. It's worth noting that, our model can match the accuracy of baselines with a lower delegation per- centage, which implies a lower inference-time and computational overhead. For example, in Visu- alBERT, Un -VQA achieves the same 68.3% VQA accuracy as LLM- VectorScale but with 1 1.67% lower LLM delegation. Table further demonstrates the effectiveness of our Uni-VQA with regard to different VQA models against the competitive baselines. The Table mainly demonstrates two key phenomena. First, our calibration technique, Calibrated (Ours), improves the calibration per- formance i.e., ECE without compromising the accuracy. Second, due the enhanced calibration, the presence of overconfident wrong predictions is effectively minimized in the highest confidence regions. As result, the uncertainty- dynamic delegation ensures that easier questions-thor
8

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of Uni-VQA with TS-VOA models and LLM across five architectures.
VQA-V2
COCO-DA
Model
ACCT
20E4
Dex-Deleg 1977 Latency
XC21
ECE4
Latency
LLM-unly (Mi)tral-BB)
0.534
72.03
0.27
0.531
Standard VQA
65.67
0.14
0.003
0.16
0.003
Pythia
VectorSccale
65.59
0.00
0.003
68.88
0.10
0.003
Calibrated (Ours)
66.15
0.06
0.009
68.64
0
0.009
Un1 -VQ (Ours)
71.00
0,05
78.77
0.426
74.78
0.06
64.84
0.333
Standard VQJA
09:35
0.18
70:38
B016
CLIP-VIL
VectonScale
69.81
0.15
0.031
70.11
0.11
0.017
Ca ibrated (Ours)
70.05
0.06
0.069
69.94
0
0.048
Uni VVA (Ours)
72.98
0.67
69.86
0.440
74.95
0.06
64,89
0.391
Standard VQA
VILBERT
VectonScall
66.87
0.14
0.011
69.04
0.17
0.007
Call lbbated (Ours)
66-90
0.05
0.027
70:59
0.02
0.012
Uni-VQA (Ours)
71.65
0.07
79.06
0.447
75.63
0.05
67.19
0.370
Stundard VQA
0,009
0.003
VisuaBBERT
VestonScale
64.33
0.010
0.18
0.003
Call aalibratdd (Ours)
65.26
0,03
0.027
67.38
0.01
0.009
Uná -VQA (Ours)
70.95
0.08
77.87
0.440
74.34
0.06
73.46
0.399
Standard VIJA
73.19
0:009
72.29
0.009
BEIT-3
VestorScale
73.62
0.14
72.16
0.009
Ca lilibraed (Ours)
73.25
0.04
0.027
71.94
0.02
0.027
Uni-VQA (Ours)
74,33
0.07
35,91
0.217
76.01
0.02
57.82
0.333
in the high-confidence bins of the calibrated TS-VOA model- confidently answered without further delegation to the LLM,, provided their confidence surpasses the dynamic threshold. Conse- quently, the hybrid model achieves higher accuracy with reduced reliance the LLM,, underscoring the importance of calibration enbancement and the Uni-VOA approach.
Table 2:
LLM--elegation percentage comparison proposed Uni-VOA against the LLM-VQA and LLM- VectorScale (threshold) baselines, to match the maximum accuracy achieved by baselines on VQA-v2.
LLM-Deleg (R))
LLS-Ueles eg (161)
Model
ACC
M-M-VQA
VVA
ACC
M. (thresh/ini VVCA
Pythla
XL.07
64.38
50.06 (-14.525))
N0.07
66.11
50.06 (-16.099))
CLIP-VIL
71.51
35.5
(-11 24.4
71.6
40.56
27.56 (9100))
ViLBERT
(997%)
VisuiBERT
69.75
64.01
47.51 (-16.9%)
69.88
66.79
49.18 (-17.618)
BETT-3
73.71
10.16
9.06 (-1.1%)
73.62
26.23
6,71 (19.528))
Our experiments reveal a trade-off between accuracy and the delegation percentage to the LLM. Adjusting the delegation threshold allows control over how often the LLM is used. Lowering the threshold reduces the reliance on the LLM and computational costs, but results in smaller gains in accuracy. This flexibility enables adaptation based on resource constraints and performance require- ments, making the hybrid approach versatile for practical applications.
5.2 ABLATION STUDIES
We analyze LLM/VLLM inference costs of our Uni-VQA in terms of the delegation percentage versus model accuracy, Additional ablation studies are provided in Appendix G
LLM Inference Cost Analysis. We study the effectiveness of our technique in terms of LLM inference cost. Table shows the fraction of samples delegated by our Uni-VQA model in order to obtain the same accuracy (i.c., 69.09% as that of the LLM-Only model where 100% samples have been answered by the LLM. As shown, with far fewer delegated samples to the LLM, we achieve competitive accuracy. For example, using ViLBERT VQA backbone, an accuracy of 69.09% (matching LLM-onll accuracy) with only 19.4% delegation to the LLM. This means, for most of the samples we can leverage the cheaper VSQVA model whereas, we can delegate only limited amount of low-confident samples to the LLM for the correction. Hence, we can maintain high predictive accuracy while being more efficient on LLM inference cost by significantly reducing the LLM reliance.
Uni-VQA achieves
While our Uni-VQA shows significant accuracy gains across all backbones in VQA-v2, the im- provement in BEiT-3 model is limited. This can be attributed to the fact that the BEÎT-3 model already demonstrates a strong performance compared to the LLM. In such cases where the TS-VOA is already highly competent, accuracy gain from LLM delegation is naturally smaller.
Table 2 shows the percentage of delegated questions of our technique baselined against the two competitive baselines: LLM-VQA and LLM- VectorStrSe (threshold). As shown, to achieve the given accuracy, proposed Uni-VOa delegates significantly lower fraction of questions to LLM and thereby being computationally more efficient.
9

---

Published as a conference paper at ICLR 2026
Table 3:
Delegation percentage for Uni-VQA
Table 4: LLaVA Delegation across models, match the
models to match LLM-Only accuracy.
LLaVA accuracy (78.35%).
L.Me-eeleg (%)
Model
MDN-Deleg (9)
LLM-VQA
LLM VectorScale
Uni Uni-VgA (Ours)
Target ACC
Pythia
ViLBERT
VisaaBBET
CLIP-ViL ViL.
80.3
65.4
ViLBERT
94.0
87.4
82.7
VQA-v2
69:09
38811
19.4
40.39
VisualBEK
93.2
84.6
COCOQA
72.03
17.14
8.06
12.13
Pythia
89.7
85.9
84.3
VLLM Inference Cost Analysis We also analyze the effectiveness of our Uni-VOA approach in reducing inference costs by using LLaVA, a large-scale vision language model (VLLM), as the LLM-based model. Our main objective is to demonstrate that Uni-VQA significantly lowers the reliance on LLaVA while maintaining comparable accuracy levels. Table 1 presents the LLM- delegation percentages for different VQA architectures, between our Uni VQA. LLM-VQA and LLM-VectorScale (threshold), in order to achieve the same accuracy as the LLaVA-only setup, indi- cating a substantial reduction in delegation when using Uni iVQA, For instance, with CLIP-ViL. as TS-VQA model, Uni-VQA -VAA achieves the same accuracy as LLaVA-only (78.35%) while requiring approximately 15% and 8% less delegation compared to LLM-VQA and LLLM-VcctorSS respec- tively.
By leveraging the calibrated confidences of our Calibrated TS-VQA models, Uni-VQA ef- fectively routes a fraction of questions to LLaVA only when necessary, avoiding redundant heavy computation on questions that can be reliably answered by the TS-VOA. Consequently, Uni-VQl not only reduces inference latency but also lowers the overall computational cost, making it cost- effective alternative to relying fully on large models.
Remark. As Tables 2 and 4 indicate, we observe that the reduction in LLM delegation is more pronounced for models with well-calibrated confidence scores. This further emphasizes the role of calibration of TS VQA models in enabling effective knowledge-exchange and uncertainty-awwar integration between the TS-VOA and LLM.
5.3 SENSITIVITY & ROBUSTNESS ANALYSIS
We conduct a cross-model hyperparameter transfer analysis, where we applied hyperparameters {1, K(c)) tuned on each model to other models, and measuring the impact their performance, to analyze generalizability of our hyperparameter selections. Table 9 (in Appendix G.7 shows that the maximum accuracy drop never exceeds 1.24% COCO-QA, confirming that pro- posed framework is not sensitive to careful tuning of the hyperparameters. Our analysis provides compelling evidence that careful threshold tuning is unnecessary, and thresholds show remarkable generalizability.
5.4 DISCUSSION
Reducing reliance on computationally intensive models is crucial in ensuring scalable and environ- mentally sustainable AI applications. as studies have highlighted sigmificant energy consumption and carbon emissions of large-scale language models (Strubell et al., 2020 Patterson et al. 2021 Our work addresses these concerns by minimizing frequent delegation high-cost models through strategic integration. Unlike pruning (Zhu et al.] [024]] Wan et al., 2023 Fu et al. (2024) and quanti- zation (Zhao et al.] 2024 Lin et al. (024)) techniques that reduce model size, our Uni VQA approach improves inference efficiency by dynamically determining when LLM delegation is necessary based on calibrated TS-V VVA confidence scores. This complementary approach can be combined with ex- isting model efficiency techniques to further reduce computational costs while maintaining accuracy.
CONCLUSION
In this paper, we introduce an uncertainty-aware LLM integrated VQA model, referred to as Uni iVQA, which facilitates knowledge exchange between the LLM and calibrated TS-VQA model based on reliable confidence scores. It cost-effectively improves VQA accuracy and inference speed. Our framework leverages well-calibrated confidence scores to guide the interaction between the LLM and TS VQA. We conducted extensive experiments across multiple datasets, which demon- strate the effectiveness of -VQA in terms of accuracy, computational efficiency, and reliability.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was sponsored by the Army Research Office and was accomplished under Grant Num- ber
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or im- plied, of the Army Research Office or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REPRODUCIBILITY STATEMENT
We provide: (1) Complete implementation details in Appendix [F3 with all hyperparameters in Ta- ble [8 (Appendix G (2) Reference public baseline implementations via MMF (Singh al., 2020) and UniLM (Wang et al., 2023) repositories; (3) Standard public datasets (VQA-v2, COCO- QA) with preprocessing documented in Appendix F.I. (4) LLM prompting methodology in Ap- pendix F:2, (5) Threshold learning procedure in Section [33 and F.4 and (6) code repository pro- vided in supplementary materials, with public release at publication. Computational environment
details are in Appendix G.12 (Tables 16 17
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In International Conference on Learning Representations (ICLR), 2023. URL https: /ttp:///eeeeeeeeeeeeemeomv. net/forum! oett:/Eoormm..cuuuu222233TCC
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6077-6086, 2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425 2425-2433, 2015.
Hangho Bao, Dong, Songhao Piao, and Furu Wei. BEIT: BERT pre-training of image trans- formers. In International Conference Learning Representations, 2022. URL https: //openeeview www./neuueideeeeeeemo
Bh2Sz5904
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.077 2021.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, and Mareus Rohrbach. Improving selective visual ques- tion answering by learning from your peers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24049-24055 2023.
Di Ding, Tianliang Yao, Rong Luo, and Xusen Sun. Visual question answering in robotic surgery: A comprehensive review. IEEE Access, 2025.
John Duchi and Hongseok Namkoong. Variance-basedd regularization with convex objectives. Jour- nal of Machine Learning Research, 20(68):1-55 2019.

---

Published as a conference paper at ICLR 2026
Shohei Enomoto and Takeharu Eda. Learning cascade: Confidence calibration for improving the accuracy and computational cost cascade inference systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7331-73333 2021.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning. vol. springer series in statistics. New York, 2001.
Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Na- jibi. Lazyllm: Dynamic token pruning for efficient long context IIm inference. arXiv preprint 1rXiv:2407.14057 2024.
Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natara- jan. Transform-retries retrreee--eneraee: Natural language-centen centric outside-knowlede visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 5067-5077, 2022.
Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 6639-6648, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks, International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10867-10877, 2023.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training, In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Ramya Hebbalaguppe, Mayank Baranwal, Kartik Anand, and Chetan Arora. Calibration transfer via knowledge distillation. In Proceedings of the Asian Conference on Computer Vision, pp. 513-530, 2024.
Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang. Kei-Wei Chang, Yizhou Sun, Cordelia Schmid, David Ross, and Alireza Fathi. Reveal: Retrieval- -aummeel-augmenned visual-language pre-training with multi-source multimodal knowledge memory. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 02369923379, 2023.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th conference of the european chapter of the 2021.
association for computational linguistics: main volume, pp.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Yu Jiang, Vivek Natarajan, Xinlei Chen, Mareus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia v0. 1: the winning entry to the vqa challenge 2018. arXiv preprint arXiv: /80700/907 2018.
Wittawat Jitkrittum, Neha Gupta, Aditya K Menon, Harikrishna Narasimhan, Ankit Rawat, and Sanjiv Kumar. When does confidence- -ased cascade deferral suffice? Advances in Neural Infor- mation Processing Systems, 36:9891-9990 -9966, 2023.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Zaid Khan, Vijay Kumar BG, Samuel Schulter, Manmohan Chandraker, and Yun Fu. Exploring question decomposition for zero-shot vqa. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2026
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and- -ad-language transformer without convo- lution or region supervision. In International conference on machine learning, pp. 5583-5594. PMLR. 2021.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented gener- ation for nhowledge-intensiyy nip tasks. Advances in neural information processing systems, 33: 9459-9474, 2020.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-ing -mage pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint vrXiv:/908.03355 2019.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. Revive: Regional visual representation matters in knowledge-bett based visual question answering. Advances in Neural Information Processing Systems, 35:10560- 15:10060-10571, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and- lannuuggg tasks. Advances in neural information processing systems, 32, 2019.
Pan Lu, Hongsheng Li. Wei Zhang, Jianyong Wang, and Xiaogang Wang. Co-attending free-form regions and detections with multi-modal multiplicative feature embedding for visual question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: Watts driving the cost of ai deployment? In The 2024 ACM Conference on Fairness, Accountability, and Trans- parency, pp. 85-99, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. Proceedings of the IEEEJeyf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr and Puneet Doka- nia. Calibrating deep neural networks using focal loss. Advances in Neural Information Process- ing Systems, 33:15288-155999 2020.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
Duy-Kien Nguyen. Vedanuj Goswami. and Xinlei Chen. Mo Revisiting modulated convolutions for visual counting and beyond. arXiv preprint arXiv:2004./ 1883, 2020.
13

---

Published as a conference paper at ICLR 2026
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mea- suring calibration in deep learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
David Patterson, Joseph Gonzalez, Quoe Le, Chen Liang, Lluis- Aluis-Miiuul Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint urXiv:2104. 10350, 2021.
Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Rephrase, augment, reason: Visual ground- ing of questions for vision-language models, arXiv preprint arXiv:2310.05861 2023.
Tianwen Qian, Jingjing Chen, Shaoxiang Chen, Bo Wu, and Yu-Gang Jiang. Scene graph refinement network for visual question answering. IEEE Transactions on Multimedia, 2022.
Stephan Rabanser, Nathalie Rauschmayr, Achin Kulshrestha, Petra Poklukar, Wittawat Jitkrittum, Sean Augenstein, Congchao Wang, and Federico Tombari. know what I don't know: Improving model cascades through confidence tuning. arXiv preprint arXiv:2502. 19335, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. Advances in neural information processing systems, 28, 2015.
Hitesh Sapkota and Qi Yu. Adaptive robust evidential optimization for open set detection from imbalanced data. In The Eleventh International Conference Learning Representations, 2023. URL htLLhttps:/ http:///ppppeeeeeeeeeeeee n forum id= hyddd3yy--BBBeee
net
Hitesh Sapkota, Dingrong Wang, Zhiqiang Tao, and Qi Yu. Distributionally robust ensemble of lottery tickets towards calibrated sparse network training. Advances Neural Information Pro- cessing Systems, 36, 2024.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54-63 2020.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pp. 146-162. Springer, 2022.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Wei Chang, Zhewei Yao, and Kurt Keutzer. How much clip benefit arXiv:2107.06383 2021.
tasks? arXiv preprint
Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Magana, Tristan Thrush, Wojciech Galuba, Devi Parikh, and Douwe Kiela. Human-adversarial visual question answering. Advances in Neural Information Processing Systems, 34:20346-2035,, 2021.
Amanpreet Singh, Vedanuj Goswami. Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Mmf: multimodal framework for vision and language research. MMF: A multimodal framework for vision and language research, 2020.
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, and Khyathi Raghavi Chandu. Selective "selective prediction":: Reducing unnecessary abstention in vision-language reasoning. arXiv preprint arXiv:2402.15611 Feb 2024.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13693-13696, 2020.
Anthony Meng Huat Tiong, Junnan Li. Boyang Li, Silvio Savarese, and Steven CH Hoi. Plug- and-play VQA: Zero-shot VQA by conjoining large pretrained models with zero training. arXiv preprint 1:Xxiv:2200008770 2022.
14

---

Published as a conference paper at ICLR 2026
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863, 2023.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEiT pretraining for vision and vision- language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E Gonzalez. Idk cascades: Fast deep learning by learning not to overthink, arXiv preprint arXiv: (.X.:::70600006000 2017.
David Warren and Mark Dras. Bi-directional model cascading with proxy confidence. arXiv preprint arXiv:2504. 19391, 2025.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transpurency, pp. 214-229, 2022.
Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than swer incorrectly. In European Conference on Computer Vision, pp. 148-166. Springer, 2022.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708, 2020.
Danny Wood, Tingting Mu, Andrew Webb, Henry WJ Reeve, Mikel Luján, and Gavin Brown. A unified theory of diversity in ensemble learning. Journal of machine learning research, 24(359): 1-49, 2023.
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, al. Sustainable ai: Environmental impli- cations, challenges and opportunities. Proceedings of Machine Learning and 5ystems,4:795-88 4:795-813, 2022.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang. Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-basee vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 3081 1081-330 -3089, 2022.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6281-6290, 2019.
Zhou Yu, Xuecheng Ouyang, Zhenwei Shao, Meng Wang, and Jun Yu. Prophet: Prompting large language models with complementary answer heuristics for knowledge-basee visual question an- swering. arXiv preprint arXiv:2303.01903 2023.
Yan Zeng, Xinsong Zhang, Hang Li, Jiawei Wang, Jipeng Zhang, and Wangchunshu Zhou. X 2-vim: All-in-one pre-trained model for vision-language tasks. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023.
Yilong Zhao, Chien-Yo Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate 11m serving. Proceedings of Machine Learning and Systems, 6: 196-200,, 2024.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 1556 1577, 2024.
15

---

Published as a conference paper at ICLR 2026
Supplementary Material
In this Appendix, we first present Table 5 summarizing the major notations used in our paper in Section A Next, we provide the important concepts required for the Methodology in Section B In Section C we present the detailed methodology for training our diverse ensemble approach for VQA calibration. In Section D we provide the detailed mathematical proofs for theoretical contributions. Section we provide additional related works. We provide additional experimental details along with the results in Section a detailed Ablation study in Section G and additional qualitative analysis in Section H Finally, we provide the broader impact statement and limitations associated with our work in Sections and respectively.
A
SUMMARY OF NOTATIONS
Table summarizes the major notations used our paper.
Table 5: Summary of Notation
Symbol Group
Notation
Description
A
Answer
V
Image set
Q
Question set
DATASET
V
Q
Input set
X,
(V qn))
Input image-question pair Total number of classes
C
Dy
[-divergence
[(x, θ)
Per-sample loss
DRO LOSS
DRO loss parameter
Output probability for the n-th sumple associated with class y
K
Number answer candidates from TS-VOA
HYBRID VQA
Confidence of predicted answer given input x;
K(c:)
Dynamically chosen candidate count based output confidence
B PRELIMINARIES
In this section, we provide the key concepts that are required to understand our approach.
VQA Accuracy: In the Visual Question Answering (VQA) task, each question is associated with multiple ground-truth answers provided by human annotators. Let a denote the set of ground-truth answers for given question, and let represent the answer predicted by VQA model. The VQA accuracy metric is defined as follows:
Acc( à a) min (1, # answers in matching
Expected Calibration Error (ECE): (Nainn et al. 2015 the calibration error between the estimated confidences and the actual accuracies. ECE is calculated by dividing the N predictions into M equal bins according to their confidence scores. Within each bin Bn, the average accuracy and confidence denoted by acc(Bm. and conf(Bm)). Then, ECE is calculated as (Guo et al. 2017):
metric commonly used to assess
Bmml Jace( Bm) conf Bma),
ECE Σ
m=1
where |Bm] is the number of samples in the m-th bin. In the context of VQA, where there is more than single ground- truth answer, ECE is measured with respect to the most frequent answer in the ground-truth annotations.
16

---

Published as a conference paper at ICLR 2026
Adaptive Calibration Error (ACE): NNxon al. 2019) an alternative metric to measure cal- ibration, which measures the difference between the confidences and accuracies across all classes, with adaptive binning rather than static and fixed-width binning as in ECE. In contrast, ACE divides the interval [0, 1] into bins with equal number of samples. ACE is defined as:
|A|
M
ACE
Σ
Σ
am y) - conf(m, y)
M
M indexes the bins, y E fI,....,ee |A[} indexes the classes, |A] is the total number of classes, acc(m, y) is the empirical accuracy in bin m for class y, and conf(m, y) is the mean predicted confidence for the same bin and class.
where m E [I....,M]
Brier Score: (Brier 1950) measures the squared error difference between the confidences and actual accuracies, without binning, and defined as:
Brier
Σ (p. v1)²
where pt and yi represent the confidence, and the prediction accuracy for the ith sample.
Negative Log Likelihood (NLL): defined as:
(Friedman et al. 2001 is also known as cross-entropy loss, and
NLL 1
logp(u;(z;).
where p(u)(ar;) are the predicted probabilities of the ground-truth to the true targets for the input.
C DETAILED METHODOLOGY: DIVERSE ENSEMBLE FOR VQA CALIBRATION
Our diverse ensemble approach builds upon Distributionally Robust Optimization (DRO) (Duchi & Namkoong 2019), which seeks to minimize the worst-case expected loss over an uncertainty set of distributions. The standard DRO formulation is:
N
( = max wEW
Σ n=1
w,, wnl(xnn 2)
(5)
where W the uncertainty set defined as:
W W WERN :
1,
w
(6)
N
Here, (y(w]| ) measures the f-divergence between the weight distribution and the uniform distribution N. and À controls the size of the uncertainty set.
To make the optimization tractable, we employ the regularized version with KL-divergence as the f-divergence measure. The closed-form solution for the optimal weights becomes:
exp(l(x,, 0)/2)
" (A)
ΣS exp(l(x,,
())11) 0)
This softmax-like weighting scheme has intuitive properties: (1) High Loss Emphasis: Samples with higher losses 1(xn, O) receive exponentially higher weights, (2) Temperature Control: The
17

---

Published as a conference paper at ICLR 2026
parameter À acts temperature parameter controlling the concentration of weights, (3) Normal- ization: The weights sum to I, maintaining a valid probability distribution.
Effect of Hyperparameter on Model Specialization The hyperparameter À fundamentally de- termines the focus of each ensemble member:
Case I: Small X (Hard Sample Expert)
When À + 0, the weight computation becomes:
i E H = arg max, [(x,, θ
lim (A)
0
otherwise
where H is the set of hardest samples. This creates model that focuses exclusively the most challenging examples.
Case 2: Large À (General Pattern Expert)
When X + oc, the weights approach uniform distribution:
lim (A) X
Vn
N
This is equivalent to standard Empirical Risk Minimization (ERM), producing a model that captures general data patterns, shows higher confidence on typical samples, achieves good average perfor- mance.
Case 3: Moderate À (Balanced Expert) Intermediate values of X create models that balance between hard and easy samples.
D MATHEMATICAL PROOFS
In this section, we provide the mathematical proof for Lemma 4.1 and Theorem 4.2
D.1 PROOF OF LEMMA
The DRO loss (Sapkota & Yu 2023 can be written as the following:
||| exp log(Av)
CDRO (0)
Σ
log(p,)
(10)
CDRO
Where P, is the predictive distribution, À is the DRO regularizer coefficient, DRO is the normal- ization constant and |A being total number of classes. We can write the following inequality
[A| Σ
(DRO 2
(1
Xpy) log P.
(1I)
CDRO
Where is the ground truth probability assigned to th class with Qy = 1 if = a(answer) and 4y otherwise.
Vy. log(P,) & O we can write the following:
A
|AI
1
(DRO(
Σ av log(lo))))))))).))))())))) -
Σ
qup log(p,)
DDRO
M
Σ
by log(ū,)
A max 4
My log (pg)
DDRO
18

---

Published as a conference paper at ICLR 2026
By Holder inequality Ilfglla - ||oll we further rewrite the above equation follows
|A|
LDRO (0) 2
Σ y
log(0,)
Σ py log(py)
CDRO
y=1
1 (CDnolCCE()) (C) XH[Ii]
Let A be the DRO specific parameters for the E ensemble models and CDRO be the respec- tive normalization constant then we can write the following:
E Σ
B Σ
CDOO
DRO (0)
1 [p]
(cr
(12)
DRo
e=1
e=1
Consider, C' E min {RO then we have the following
Lej
LDE(0) 2 0
(cr()) >x7(
(13)
This proves the Lemma.
Steps from Eq. 70 to 77
We can rewrite the following:
exp (- Io8(Eu) = (exp (log(d,))) Dy
X
Case 1: if 1:
In this case (1 Xpy) $ 0 and pyy
2 0 and therefore Py 2 (1 Apr)
Case 2: if if 1:
In this case as Py < and therefore py > whereas
Xpy) 1 and
therefore Py Xpy) As in both cases, Py -
2 (1 Apr)) and therefore Eq. leads from Eq.
10
D.2 PROOF OF THBBREM 4.2
Based on Lemma 4.1
minimizing our DE loss ensures increase in the entropy.
We first establish the relationship between entropy and confidence through general bounds that hold for any probability distribution, then show that increasing entropy lowers the admissible confidence bounds.
ENTROPY- CONFIEE-C BOUNDS
We first prove that entropy constrains confidence to lie within a bounded interval.
Lemma D.1 (Eatropy-Confidenc Bounds) Let p be a probability distribution over the classes and let max; Pi, denote the confidence. Define the binary entropy h(c) -clog c
<) log(1 c). Then the Shannon entropy H(p) satisfies:
- llg < H1(p) h(c) (1 (1-c) log(A -)
(14)
Lower bound: The lower bound follows from the relationship between Shannon entropy and min- entropy (Rényi entropy of order infinity). By definition:
H()) Σ
ps log 2
Σ
Ps log logg
19

---

Published as a conference paper at ICLR 2026
where the inequality holds because p. I c for all i, which implies - llg P; llog
Upper bound: Without loss generality, assume the maximum probability is p1 = c. Let 7 (r.....,TA. TA) be the distribution over the remaining A-1 classes, i.c., pt (1-c)r; for = .,...,A
with Σ 2i= T3 1. Then:
A
H(p) = - c clog c
Σ
Pi log
1=2
A Σ (1 c)ri (log(1 c) + r:)
-clloge
1=2
=-cloge (1 c) log -c) (1 c)
ri logr:
= h(c) + (1-c)
Since is distribution over A classes, we have H(r) I log(A 1) (with equality when is uniform). Thus:
H(p) & M()) + (1 - log(A 1)
CONFIDENCE UPPER BOUND A FUNCTION OF ENTROPY
Define Fa(c) = h(c) + (1 - log(A 1). We show that this function is strictly decreasing for c> ::
d --clog c (1- log((1 (1gc) (1 (Ag(A 1)]
Fc(c)
dc
log 1 1 log(1 c) + - log( A - 1)
1
log
log(A A 1) = log c(A - 1)
c
For c 1. we have c(A - 1 c, which implies a(A-1) < and thus F 0. Therefore,
FA is strictly decreasing on (1) 1] and hence invertible. Let 9A denote its inverse.
From the entropy bounds in Lemma D.I we obtain bounds on confidence function of entropy:
e H(p) < (H()))
(15)
Key Implication: As entropy Hp)) increases, both the lower and upper bounds confidence decrease. This means that high confidence values become incompatible with high entropy, regardless of how the non-maximum probabilities are distributed.
Example (VQA-v2 with = 3129): At (()) 1, the upper bound gives S 0.91; at H(p) = 4, we have & 0.59. Thus, moderately increased entropies already exclude very high confidences.
COMPLETING THE PROOF OF THEOREM
Minimizing DE loss ensures the increase in the entropy. which makes confidence á lower than that the ERM loss. We can state this fact in expectation: E|pDe] < E[pERM).
Considering the equal accuracy assumption between ERM and DE, write the following:
E[P(( + v)) DE E(P( # v) ENM
(16)
20

---

Published as a conference paper at ICLR 2026
Now let's break this into the high (> 7) and low confidence region (< 7). We can write the follow- ing:
E[P(O VIIDE + E[P(O VIDE = E[P( + V P * y)
(17)
Let us consider N ERM,in in be the number of incorrectly classified samples in the high-confidenee region in ERM and DE be the samples in DE. We make an assumption that the number of confidently wrong samples are higher in ERM. This has been observed in our empirical evaluation (Figure ( as well as found in the existing literature (e.g. see Fig. C.2 in Mukhoti et al. (2020))
Based on this expectation and invoking the fact that E[ĀDE] I E[PERM)) the incorrect samples using DE will be pushed more toward the low confidence region. This will lead to the following
EIP(O + v)IB S EE[P( + V)lRRM
(18)
Above equation immediately leads to the following:
E + D) 2 TPP D)ET
(19)
This proves our Theorem. Our empirical findings, shown in Figs. 11 and 12 support this, they demonstrate that our calibrated model has more samples in the less confident region compared to the uncalibrated Standard VQA. Fig. empirically validate that DE, 2 N hold. Additionally, Fig. 8 validate that NE 2 NERM-
Empirical Support for the Relationship Between Entropy and Confidence: We provide exten- sive empirical validation of our theoretical analysis: (a) Per-sample analysis: Among all samples with increased entropy under our method, 96.61 also exhibit decreased confidence (see Fig. 9b), providing strong empirical support for the inverse relationship. son correlation between entropy and confidence r = 0.9236, and the Spearman rank correlation is -0.9604 over the validation set. Both statistics confirm pronounced inverse relationship. (c) Bound verification: Fig. Da plots empirical (H max p(z)) pairs together with the theo- retical lower bound e H and upper bound ga (H), demonstrating that all predictions fall within the admissible region as predicted by Lemma D.I
(b) Correlation analysis: The Pear-
ERM
Diverse Ensemble
Confidence Threshold (T)
Figure 8: Empirical evidence illustrating No. 2 Ni across four VQA architectures.
E ADDITIONAL RELATED WORK
Our proposed framework relates to several research directions in the literature. While we discussed some related works in Section 2 approach also shares common goals with model cascades (Wang et al. 2017 Warren & Dras 2025 Jitkrittum et al., 2023 Enomoto & Eda 2021 Rabanser al. 2025 that aims at reducing the computational efficiency by strategically routing inputs through a cascade of models with progressively increasing capacity, complexity and computational costs, based on deferral mechanisms, hence enabling easy inputs be handled by cheaper and simpler models, while complex inputs being progressively cascaded to the more com- plex models. In this section, we elaborate on the connections between our approach and existing approaches in model cascades, highlighting their key distinctions setting our work apart.
Model Cascades:
Model cascades are often used improve inference efficiency by sequentially routing harder inputs to more sophisticated models, when earlier ones are uncertain, where deferring mechanism deter- mines whether to defer to large model, or accept the current model's output. Common deferring mechanisms rely confidence uncertainty estimates from smaller model. Sharing the same goal, our method different than existing method in the model cascades literature. Methods including IDK cascades (Wang al. 2017), rely solely raw confidence scores (typically the maximum softmax probabilities) without applying any explicit calibration. However, recent works (Jitkrittum et al. 2023 Enooooo & Eda 2021 Rabanser al. 2025) highlighted that uncalibrated confidences
21

---

Published as a conference paper at ICLR 2026
1.0
Admissible region
36651 (94.11%)
1244 (3.21%)
(F)(
AH Hous -Vannilaa
0.8
Gui Ours
Confidence
0.6
0.4
0.2
-2
94 (0.21%)
(2.51%)
0.0
0
2
4
6
-1.0
-0.5
0.0
0.5
Entropy H(p)
Ac COurs
OVanilla
(a) Confidence vs. entropy with theoretical bounds.
(b) Per-sample A H vs.
4
Standard TSAVAA
Standard TS-VQA
Ours (Diverse Ensemble)
Ours (Diverse Ensemble)
20
3
Density
Density
15
10
1
5
0
0.0
0.2
0.4
0.8
1.0
Entropy H(p)
Confidence c(p)
(c) Distribution of entropy (left) and confidence (right).
Figure 9: Entropy-confidence analysis. (a) Empirical predictions within the admissible entropy-contided region. (b) Quadrant analysis shows our method increases entropy while reducing confidence for the majority of samples. (c) Distribution shift: mcan entropy increases from 0.61 to 1.43; mean confidence decreases from 0.86 to 0.68.
can lead to suboptimal deferral decisions, especially when the downstream model behaves as spe- cialist. In contrast, other methods proposed explicit confidence calibration techniques to improve deferral, such as Saaming-to-cascadee (LiC) (Enomoto & Eda 202T) and gatekeeper-basee tuning (Rabanser al. (025), both of which improve routing decisions by improving the calibration of the smaller model's s confidence estimates. Nevertheless, these works consider simple routing as a defer- ring mechanism, i.e. if the calibrated confidence of the smaller model falls below a threshold, the input is routed to large model which predicts the final answer. contrast, our approach introduces adaptive integration by enabling the simple model to adaptively share knowledge with the larger model through candidate answers, which, as supported by our experiments, lead to more informed reasoning and significant improvements in our overall cascade accuracy.
ADDITIONAL EXPERIMENTAL DETAILS
In this section, we first provide detailed description of the datasets, followed by explanation of the LLM prompt construction and in-context example selection. Next, we provide the imple- mentation details of technique. After that we show the ECE plot of our technique along with other competitive baselines. Finally, we show the performance of the hybrid approaches where we integrate different baselines with I.L.ss.
22

---

Published as a conference paper at ICLR 2026
F.1 DATASET DESCRIPTION
We experiment on the VQA-v2 (Antol et al. 2015) and COCO-QA (Ren et al. 2015) datasets, which contain questions on the COCO image dataset (Lin et al. 2014). VQA-v2 dataset consists of 443, 757 questions in training split, and 10 ground-truth answers per each question. As the ground- truth answers of the test split of VQA-v2 are not publicly available. we use the validation and test splits as provided by (Whithhead et al. 2022), as evaluating the calibration error requires sample- level accuracies. The test split consists 106k, and the validation split consists of 86k questions.
COCO-QA dataset contains 78, 736 training, 38, 9488 testing questions generated from Microsoft COCO dataset (Lin et al., (2144), with a single ground-truth answer per question. experiments, we randomly sample a validation split of size of 12000 from the training set.
F.2 LLM-BASED INFERENCE FOR VQA
We describe the process of delegating question answering LLMs when the predicted confidence score of the TS VQA model falls below predefined threshold We outline the in-context learning based paradigm for prompting the LLM, and procedure for constructing effective prompts. For LLM-based prompting for VQA task, follow prior works (Yu et al. 2023) Yang et al. 2022). To leverage the LLM, we use the few-shot in-context learning approach, which is an effective approach to adapt the L.M to certain task, without the need for computationally intensive fine-tuning, by augmenting the prompt with input and output examples, enabling an efficient and training-free adaptation to the task.
F,2.1 PROMPT CONSTRUCTION
Creating structured input prompt for the LLM involves several components that help the LLM understand the question's 's context and generate accurate answers. The prompt structured as shown in the template below, where underlined text represents template keywords, and the rest are place- holders for the data samples.
Context:
\n Question: q \n Answer:
F.2.2 CONTEXTUAL INFORMATION
To help the LLM model comprehend the visual content referenced in the question, we use off-the- shelf image captioning models to generate descriptive of the image in textual format. Similar to prior works (Guo et al. 2023), we leverage the PNP-VQA model (Tiong et al., 2022 for image captioning, which generates captions relevant to the question, ensuring that the LLM has relevant contextual information to answer the question.
F.2.3 IN-CONTEXT EXAMPLES
In-context examples consist of an example prompt, along with the desired answers from the training data, formatted similarly the test prompt. These examples help the LLM generate the answer by following the pattern established in the prompt. For each test sample, multiple in-context examples selected based on their cosine similarity to the test image-question pairs. This involves extracting the image and text embeddings from the VQA data, using an off-the-shelf pretrained model. We specifically BLIP-2 model (Li al., 2023) for this purpose. The average cosine similarity between the embeddings of any two image- quesstin pairs (q:, V&) and (q/, Vy) in training and test splits is calculated. The top N examples with the highest similarity are then chosen as in-context examples.
F.2.4 ANSWER-CANDIDATE AUGMENTED PROMPTS
As demonstrated in Fig. 2 the predictive performance of the LLM can be enhanced when the prompt is augmented with some answer candidates. Assume that given input to the task- specifc VQA model, A& (a...... .g} the K candidate answers corresponding to the K answers with highest probabilities in descending order, and Cx = {p...... .px} are the corre- sponding probabilities. Given the of K answer candidates, we augment the prompt and present
23

---

Published as a conference paper at ICLR 2026
set of answer candidates as additional context to the question. The answer candidate augmented
prompt is constructed as bellow:
Context: \n Question: \n Candidates: C 'n
Answer: a
F.3 IMPLEMENTATION DETAILS
In this section, we provide additional implementation and experimental details of our proposed method and experiments. We conducted our experiments using PyTorch. Our experiments utilize publicly available implementations across all models. For all VQA architectures except for BEiT-3, use their implementations as provided by the MMF (Singh et al. 2020) repository 2 For BEiT- 3, we use its official implementation from Microsoft's UniLM project (Wang et al. 2023 To
train the standard VQA models, the training hyperparameterss of the networks given in MMF and UniLM repositories are used. For training Calibrated VQA models, we use the same training hyperparameters as the standard VQÁs. We adopt the VectorScale implementation from Whitehead et al. 2022
We trained BEÎT-3 TS-VQA on a single NVIDIA RTX A6000-48 GB GPU. Furthermore, the latencies and carbon emissions in Fig. and Table [ are reported based the models running a single A A100-40 GB GPU. For LLM inference with Mistral-7B, we run the model on a single A100-40 GB GPU.
single A100-40 GB GPU, and the rest of the TS-VOA models
VQA by the LLM Model Following (Yu al. 2023 Yang et 2022 we provide 9 captions as context for the question, and use PNP-VQA (Tiong et al. (022) for generating question-related captions, as the context about images in the prompt. For each test instance, 10 in-context examples from the training data are selected based the average of their image and question embedding cosine similarities, and included in the prompt. Specifically, BLIP-2 model is used to extract the image and question embeddings, used for in-context example selection. The LLM is queried 5 times to ensemble the answers as the final answer to the question. For answer-candidates -aadecdddettee--ugeeette VQA with LLM, restrict to using 10, 5, 2. and answer candidates. The LLM Maddd-aeddd-- inferences conducted once.
F.4 CONFIDENCE THRESHOOLD DETERMINATION AND DYNAMIC CANDIDATE SELECTION
In this section, we provide a detailed explanation of how the confidence thresholds / and u, as well the dynamic answer candidate selection function K (c;). determined using held-out valida- tion set. Our approach is fully data-driven and optimizes for accuracy while structurally ensuring efficiency through selective LLM delegation.
The threshold selection process is designed to maximize VQA accuracy on the validation set within each confidence region, while efficiency gains emerge naturally from the threshold structure itself. Specifically: (a) (Primary objective) We maximize validation accuracy within each confidence bin, and (b) (Efficiency mechanism) by setting threshold u based on where TS-VQA achieves best accuracy (or alternatively dynamically based on computational budget), we automatically avoid unnecessary LLM invocations in high-confidence regions.
Optimization Objective and Process:
The complete process consists of two main steps:
Step 1: Per-confidence-P Bin Policy Selection. We first partition the confidence range [0, 1] into B equal-width bins (we use B = 10 in our experiments). For each bin b, evaluate multiple inter- action modes on the validation set. (1) TS TS-VQA only, (2) LLM-only, without any answer candidates from TS-VQA, I.IM with top-K candidates (K {1, 7, 10)) from TS-VOA
24

---

Published as a conference paper at ICLR 2026
Standard
b) VectorScale
c) Calibrated (Ours)
1.00
ECE=0.1
1.00
ECE=0.09 Acc.m65.66 %
1.00
ECE-0.06
A60. %
Acc. 7 %
Accurac
Accurscyy
Pythia
0.50
0.25
0.25
0.25
0.00 50
0.00
0.25
0.50
0.75
1.00
0.00
00
0.25
0.50
0.75
1.00
0.00
0.00
0.25
0.50
0.75
1.00
Confidence
Conridence
Contidence
ECE- 1.
ECE-0 14
1.00
ECE=0.0
0.00 0.75
Accuracy
Acc. -0.1-
0.75
0.75
Acc.
ViLBERT
0.50
0.50
0.50
6.25
0.25
0.25
0.00
0.00
0.00
Confidence
Confidence
Confilinnce
1.00
1.00
1.00
ECE=0.14
ECE 0.14
ECE-D.D D.03
Accuract
0..75
Acc=64.9 %
Acc.=60,8 %
00.75
A6C %
VisualBERT
0.50
0.50
0.25
0.25
0.25
Confidence
1.0
0.00
0.25
Confidence
0.50
0.75
1.00
0.0
Confidence
0.3
1.0
FCF-
FCF-0
ECE-0.00 .08
Accuracy
0.75
.0.75
.0.75
Acc. 700.0
CLIP-VIL
0.50
0.50
ALCO
0.50
0.25
0.25
0.00
0.00
0.00
Conlidence
Confidence
Confidence
Figure 10: Effectiveness of DE-based VQA in improving the calibration of all VQA architectures. a) Standard, b) VectorScale-calirrated, and c) DE-based VQA models.
For each bin b and each mode m, we compute the VQA accuracy on validation samples whose TS-VQA confidence falls within bin b. We then select the mode that maximizes accuracy for that bin:
mi = arg max Acc(m, b) MEM
(20)
where M (TS-VOA, LLM LLM 1L, This creates discrete mapping from confidence bins to optimal interaction modes, ensuring that we only invoke the LLM (and only with a specific number of candidates) in regions where it demonstrably improves accuracy on held-out data.
Deriving Thresholds and Smooth K(c) Function. From the bin-wise optimal policies determined in Step derive the continuous thresholds and candidate selection function:
Step 2:
Lower threshold I: We define as the upper boundary of the highest confidence bin where "LLM-only (Top-0y" achieves the best accuracy. This identifies the region where TS-VQA has minimal domain knowledge and the L.M should answer without potentially misleading candidates.
Upper threshold u: We define as the lower boundary of the lowest confidence bin where "TS- VTS-VQA only" achieves the best accuracy. This identifies the region where TS-V TS-VQA sufficiently reliable to answer without LLM consultation.
Dynamic candidate function K(c;): For the intermediate region [t. u]. we fit a smooth, monotonically decreasing function that maps confidence scores to the optimal number of
25

---

Published as a conference paper at ICLR 2026
Table 6: Delegation percentage for hybrid models match LLM-Only accuracy (72.03%) COCO-QA
dataset.
I.LM-Only
Uni-VOA (Ours)
Pythia
ViLBERT
VisuaBERT
CLIP-VIL ViL
100
18.14 (8186%)
8.06 (-91.94%)
25.56 (-74.44%)
12.13 (-87.87%)
answer candidates. We use the exponential form presented in eq. (2) of the main paper, where the parameters M and W are learned by fitting to the per-bin optimal K values determined in Step 1, subject to the constraints that (1) as M (maximum candidates the lower threshold) and (()) a threshold).
(minimum candidates near the upper
Rationale and Trade-ofTs: This data-driven approach offers several advantages: (1) Accuracy- optimized: By selecting the best-performing mode for each confidence region on validation data, ensure that delegation decisions are evidence-based rather than heuristic, (2) Efficiency through structure: The threshold naturally limits LLM usage to cases where it provides value, as high- confidence samples are handled by the calibrated TS-VOA, (3) Adaptive candidate selection: The smooth function (es) avoids abrupt changes in the number of candidates provided, ensuring that LLM receives appropriate amounts specialized knowledge based on TS-VQA uncertainty.
G ABLATION STUDY
G.1
ADDITIONAL EXPERIMENTS COCO-QA DATASET
Table presents additional resuls the COCO-QA dataset, illustrating the extent of LLM- delegation necessary for the hybrid models to attain equivalent accuracy as the LLM (Mistral-7BB on the COCO-QA dataset for each TS-VOA model. BETT-3 is omitted since the BETT-3 TS-VOA already surpasses the accuracy of Mistral-7B model on COCO-QA.
Number Samples
4000
4000
Number ofmbereetSampee
0.000
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Confidince
Confiidnce
(a) Standard VQA
(b) Calibrated (Ours)
Figure 11: Confidence distribution of incorrect answers in a) Standar, and b) our Calibrated VQA.
Number of Samples
#0000
oumber Sumples
15000
20000
8.00
8.00
Confidence
Confisence
(a) Standard VQA
(b) Calibrated (Ours)
Figure 12: Confidence distribution of correct in Standard, and b) our Calibrated VQA.
G.2 EFFECTIVENESS OF DIVERSE ENSEMBLE (DE)-BASED VQA CALIBRATION TOWARDS CALIBRATED VQA
In this subsection, we analyze the effectiveness the DE-based framework in improving the cali- bration of TS-VQA models compared to standard and VectorScale- baseed VQA models. We present reliability diagrams for four VQA architectures to illustrate the differences in calibration perfor- mances. Fig. 10 clearly shows that standard VQA models are overconfident and poorly calibrated, while VectorScale based VQA models exhibit only a slight improvement in calibration, still suf- fering from overconfidence. In contrast, Expected Calibration Error (ECE) and overconfidence compared to the baselines, resulting in sub- stantially improved reliability. These findings underscore the importance of employing effective
proposed DE-based VQA significantly reduces the
26

---

Published as a conference paper at ICLR 2026
Table 7: Performance comparison of Uni-VQA with baseline TS-VQA models and LLM across four architec- tures, evaluated using multiple calibration metries including ECE, ACE, Brier score, and NLL.
VOA-2
COCO-OX
Model
ACCT
ECED
ACET
Brier
BLL
ACCT
ECE]
AS2]
Brieet
BELL
LLM-only (Mistral-7B) 7B)
69.09
0.31
0.34
0.30
0.91
72.05
0.27
0.48
0.27
0.94
Standard VQA
0.19
0.30
VectorScale
0.61
Pythia
Calibrated (Ours)
66.15
0.06
0.05
0.17
0.53
68.64
0.02
0.02
0.15
0.47
Uni-V/CA (Ours)
71.00
0.05
0.05
0.17
0.53
74.78
0.06
0.13
0.17
0.51
Standard VQA
0.18
0.20
0.87
70.38
0.15
0.71
CLIP-VIL
VectorScale
0.67
Calibrated (Ours)
70.05
0.0R
0.07
0.16
0.52
69.94
0.02
0.03
0.15
0.47
Uni-VoA (Ours)
72.98
0.07
0.07
0.17
0.53
74.95
0.06
0.13
0.17
0.50
Standard VQA
66.98
0.19
0.17
0.21
0.89
69.23
0.20
0.18
0.21
0.99
VectorScale
VILBERT
Calibrated (Ours)
66.90
0.05
0.04
0.16
0,89
70.59
0.02
0.03
0.15
0.46
Uni-VOA (Ours)
7166
0.07
0.07
0.17
0.53
75.63
0.06
0.12
0.16
0.49
Standard VQA
64.92
0.14
0.13
0.19
0.68
65.28
0.19
0.16
0.21
0.87
VisuaBBET Calibeeeed (Ours)
VectorScale
0.16
65.26
0,03
0.03
0,16
0,49
67.38
0.01
0.02
0.16
0.48
Uni-VQ8 (Ours)
70.95
0.08
0.08
0.18
0.53
74.34
0.06
0.14
0.18
0.52
calibration techniques, such the DE framework, enhance the reliability of VQA models and enable more accurate uncertainty estimates not only for ensuring reliability of the entire Uni-VQA framework, but also for effective integration with the LLM model.
G.3 EFFECTS OF DE-BASED VQA ON REDUCING OVERCONFIDENCE IN INCORRECT PREDICTIONS
Figs. (a) and (b) present histograms of confidence scores for incorrect predictions, respectively, made by the standard, and our DE-based Calibraa ibrated VQAs. Our proposed method, assigns low confidence scores to the majority of incorrect answers, while the standard VQA produces very high confidence scores for a large number of the incorrect answers. This observation confirms that our DE-baaed Ca ibrated VQA significantly reduces the overconfidence, by pushing the majority of incorrect answers towards lower confidence scores.
G.4 COMPREHENSIVE EVALUATION CALIBRATION USING ALTERNATIVE CALIBRATION METRICS
ECE (Expected Calibration Error) is a standard metric commonly used to assess model calibration; however, it has several known drawbacks, including sensitivity to binning choices, the inability to capture local miscalibrations effectively, and ignoring the distribution of prediction probabilities within each bin. To comprehensively demonstrate the robustness of our proposed calibration ap- additionally proach in improving the calibration of VSAVQA models across various architectures, evaluate it using alternative calibration metrics: 1) Adaptive Calibration Error (ACE), is an exten- accurately capture local miscalibration, 2)
sion of ECE that adaptively determines bin sizes Brier Score measures the squared difference between predicted probabilities and actual outcomes. assessing both calibration quality and sharpness of probabilistic predictions, and 3) Negative Log Likelihood (NLL) quantifies the negative log probability assigned to true outcomes, heavily penaliz- ing confident yet incorrect predictions. These metries provide complementary perspectives essential for robustly evaluating calibration quality. Table 7 summarizes the results, clearly indicating that our calibration method consistently enhances performance across all evaluated calibration metrics.
G.5 PERFORMANCE COMPARISON OF LLM vs. TS-VOAS IN VARIOUS CONFIDENCE RANGES
In this experiment we compare the performance TS SSQA models, LLM without answer candi- dates, and LLMs augmented with answer candidates (2 candidates are given) across all four archi- tectures, for both standard and our Calibrated VQA models. We evaluate the performance of these models in terms of accuracy for samples whose confidences, determined by the respective TS-VOA (Calibrate ed or standard), fall within three different confidence ranges: 1) low (0-0.1), 2) moderate (0.4 0.5), 3) and high (0.95 1). Results presented in Table
27

---

Published as a conference paper at ICLR 2026
Table 8: Comparison between predictive performances of LLM and our Calitreted TS-VQA in low- and high- nigh-confidence regimes. In low confidence levels, total delegation to LLM yields higher aceuracy, while is misled when presented with the answer candidates from the VQA model. On the contrary, in high confidence levels, VQA model outperforms LLM, suggesting that high-confident questions can be answered more efficient manner by the VQA.
e 0.
[0.4.0.5
C 0..55
Model
13-VOR
LLM candidates
LLM
15-VO4
LLM
w.
TS-VOA
LLM
condidutes
candistaee
Lal Pythia
14.23
39.41
46,29
46.83
90.95
86.85
Callbcated
CLIP-VIL
6.95
8.58
16.17
37.01
39.45
36.52
89.88
87.94
83.64
Calibrated
49.30
46.75
91.41
Calibrated
ViuaaBERT
7.12
13.16
20.18
47.56
54.44
52.48
93.19
92.02
90.05
Standard
Pythia
334
6.42
12.29
32.13
35.88
37.31
83.94
84.43
8137
Sunddrdd
31.91
81.50
Standard
VILBERT
234
5.57
12.56
26.49
30.68
33.25
79.68
81.14
7787
Standard
VisuaBERT
333
8.38
14.16
31.36
36.44
38.52
85.08
84.39
82.02
For our Calibrated models, we observe that in the low confidence range, the LLM alone is the most effective. In the moderate confidence range. providing answer candidates from the TS TS-VQA generally improves the performance of the L.M. However, in the high confidence range, the TS-VQA outperforms the LLM. This suggests that answering hgih- agih--confidenc questions using the TS-VQA model, rather than the LLM, not only reduces the burden on the LLM and improves efficiency, but also benefits the hybrid approach in terms of improving the accuracy.
In contrast, when using a standard VQA as the TS VQA, we observe that the LLM achieves the highest accuracy in both the low and moderate confidence ranges. The lower accuracy of the LLM with answer candidates indicates that the provided top-k answer candidates reduces the accuracy as compared when candidates are provided, suggesting poorer quality of the answer candidates set.
In the highest confidence range, the LLM with answer candidates generally performs better than both the LLM alone and the TS-VOA. This behavior makes the effectiveness of a hybrid approach suboptimal for any delegation confidence threshold when using standard VQA model.
These findings highlight the importance of calibrating the TS-VQA model using the diverse ensem- ble, enables more effective hybrid approach that leverages the strengths of both the TS-VQA and the LLM in different confidence ranges. By delegating low-confidence question to the LLM, incorporating answer candidates for moderate-confidence questions, and relying on the TS-VOA for high-con high-confidence questions, our proposed approach improves both accuracy and efficiency in the VQA task.
6.6 EFFECTIVENESS OF THE DYNAMIC TOP-K SELECTION
To evaluate the effectiveness of the proposed uncertainty-guided dynamic answer candidate selec- tion, we compare the performance of the Uni-VQA framework against the same approach with fixed top-K answer candidates provided for all confidence levels. In all methods, the TS-VQ2 model is our Calibrated VQA, trained according the diverse ensemble. We refer to these variants as LLM-Calibrated (Top-K), where K represents the number of answer candidates provided to the LLM model.
Fig. 13 presents the VQA accuracy with respect to the delegation thresholds for various K values, across all 4 architectures. The figures suggest that the dynamic approach, i.e.. Uni-VOA, achieves the highest overall accuracy for any delegation threshold. Additionally, for any given accuracy, the dynamic approach achieves the lowest delegation percentage among the other variants, while also achieving higher accuracy than the highest achieved by the fixed top-K K answer candidate variants, at certain delegation thresholds.
A comparison between the accuracy of the methods at fixed thresholds for thresholds below 0.2 highlights the effectiveness of the LLM-only prompting when no answer candidates are provided (Top-0). The VQA accuracies of the LLM-Calibrated (Top-1), and LLM-Ca Cilaaiibbeed (Top- 10) variants at lower thresholds suggest that providing answer candidates this region confuses the LLM, compared when those answer candidates are not present. This can be attributed to the
28

---

Published as a conference paper at ICLR 2026
UnivQA (Ours)
LLM Calibralbttdd (top-0)
LLM-Calbrated
Cal-Calbrddeed
Pythia
72
72
74
71
VQA Accuracy
70
68
66
0.2 0.4 0.6
041
0.8
0.2
0.6
0.2 D41
0.8
Delegation Threshold
Figure 13: Performance comparison of the proposed Uni Uni- against answer candidates, with respect to the delegation threshold.
with fixed top- K
answer candidates being random guesses in the low-confidence region, indicating the model's total lack of knowledge.
These findings demonstrate the superiority of the dynamic top-K selection approach employed by Uni-VOA. By adaptively selecting the number of answer candidates based on the confidence of the TS-VQA model, Un VVQ achieves higher accuracies and lower delegation percentages compared to fixed top-K variants. Furthermore, the results emphasize the importance of relying solely on the LLM for low-confidence questions, as providing answer candidates in this region can hinder the LLM's performance. The dynamic approach effectively leverages the strengths of both the TS-VOA and the LLM, leading to improved overall performance the VQA task.
G.7 UK TQQA HYPERPARAMETER GENERALIZABIIJTY
To validate the robustness of our Uni-VQA framework and demonstrate that it does not require careful per-model hyperparameter tuning, we conducted an extensive cross-model hyperparame- ter transfer analysis. This analysis evaluates whether hyperparameters optimized for one TS-VOA backbone can effectively transfer to other architectures without significant performance degradation.
For each VQA model in our framework, we apply the hyperparameters {1, u, K((c) [c;)) originally tuned for that specific model all other models in our evaluation set. This cross- crossaappliottion tests whether our delegation mechanism maintains consistent performance across architectural variations. The hyperparameters include: (1) the delegation threshold that determines when to invoke the LLM,, (2) The dynamic top-K bounds (1, u) that control answer candidate selection, (3) The confidence- adaptive function K ((;) that adjusts selection based on model confidence.
Key Findings. The analysis reveals remarkable robustness in our hyperparameter selection. The maximum deviation from optimal performance across all cross-model transfers is only 1.24% (CLIP-ViL using BEÎT-3 hyperparameters), with most deviations below 0.6%. This demonstrates that hyperparameters are not overly specialized to individual architectures. Additionally, the sym- metry in the transfer matrix (e.g., ViLBERT Pythia and Pythia ViLBERT both maintain high accuracy) confirms that the hyperparameter robustness is bidirectional, not dependent on specific source-target model pairs.
These results validate our claim in Section 5.3 of main paper, that the Uni-VQA framework is not sensitive careful hyperparameter tuning, making practical and scalable solution for real-world VQA applications. The framework's 's ability to maintain consistent performance across diverse ar- chitectures with shared hyperparameters addresses critical deployment challenge in VQA systems.
G.8 ALTERNATIVE UNCERTAINTY MEASURES FOR UWIIVQQ ENTROPY
While our main approach uses confidence scores (i.e., maximum output probability) to guide knowl- edge exchange between TS-VOA and LLM, we also explored alternative uncertainty measure to
assess robustness of our delegation strategy. A natural alternative to confidence score is entropy, which is widely used in uncertainty quantification literature [_akshminarayanan et al. 2017 Kendall
29

---

Published as a conference paper at ICLR 2026
Table Cross-mens model hyperparameter generalizability COCO-QA. Each cell shows accuracy when model in the corresponding row uses hyperparameters (HP) tuned for model the column. Bold values indicate model- specific tuned parameters. Max Dev shows maximum deviation from optimal performance.
Model
HP from CLIP-ViL
HP from
HP from ViLBERT
HP from VisualBERT
HP from
Max Dev
Evaluated
Pythia
BETT3
CLIP-ViL
74.95
75.08
75.11
74.98
73.71
1.24
Pythia
74.76
74.78
74.78
74.82
74.41
0.37
ViLBERT
75.57
75.61
75.63
75.55
75.06
0.57
VisualBERT
74.28
74.25
74.25
74.32
73.72
0.60
75.90
& Gal 2017). it provides measure of the prediction uncertainty by quantifying the dispersion of the probability mass across possible answers.
To empirically evaluate the effectiveness of our approach using "entropy" sure for delegation and knowledge exchange, we implement an entropy- bassed delegation variant of Uni-VQA and compared it with confidence-based approach. Table TO compares performances of our Uni-VV using the two uncertainty-measure, in terms of accuracy and LLM delegation percentage for ViLBERT TS-VOA on VQA-v2 dataset, and shows that both uncertainty measures achieve comparable performance, with confidence-bart based delegation showing slight advantage in both accuracy and efficiency.
uncertainty mea-
Table 10: Performance comparison between confidence-based and entropy-based uncertainty measures for knowledge exchange in Uni-VÇA using ViLBERT VQA-v2 dataset.
Uncertainty Measure
ACC (f)
DiM-Deelgatiin (%) (1)
Confidence-based
71.6
79.1
Entropy-based
71.5
80.0
Callbrated (Our)
0.75
Accures
5.75
0.50
0.25
0.25
Average Entropy
Confidence
E-trapy
Standard VOA
1.90
ECS-0.10 Acc=B7.0 %
0.50
3,50
0.25
0.25
0.5 Confisance
E100p7
Confidence
0.6
(a) Relationship between accuracy and uncertainty measures (confidence entropy) for Ca 1 ibbrted (top) and Stan- dard VQA (bottom) TS-VOA models.
(b) Inverse relationship hetween model predic- tion confidence and output entropy, averaged over 30 confidence bins.
Figure 14: Visualizations of confidence and entropy relationships in ViLBERT on VQA-v2 dataset.
Fig. 14a illustrates the relationship between both uncertainty measures (entropy and confidence score) and accuracy, standard and our DE-based Cal ibbated TS-VOAs. For low-entropy (corresponding to high-confidence) regions, our calibrated models consistently achieves higher ac- curacy compared to high-entropy regions, indicating that the Calibrated model's answers are more reliable in low-entropy regions, confirming that both measures effectively identify samples where the TS - VQA model can be trusted. Additionally, Fig. [4b]depicts the relationship between average answer confidences and probability entropies, calculated in 30 equally spaced confidence in- tervals, illustrating a clear inverse trend where higher confidence values correspond to lower entropy in the predicted distributions.
30

---

Published as a conference paper at ICLR 2026
While entropy can also serve proxy for uncertainty, we choose confidence primary uncer- tainty measure for several practical reasons: (1) Interpretability: Confidence is bounded between O and 1 with intuitive probabilistic interpretation, where in well-calibrated model confidence of 0.9 suggests a 90% probability of correctness. On the contrary, entropy ranges between 0 and (Cg)(C) where C' is the number of answer classes, which makes setting and interpreting thresholds less intuitive. (2) Direct relationship with calibration: confidence score is a widely used measure in calibration literature. Additionally, calibration metrics including ECE are specifically designed to measure the alignment between confience scores and accuracies (both bounded between 0 and 1), hence confidence score is natural choice for our framework. (3) Simplicity: Using confidence scores for both calibration assessment and delegation decisions leads to a simpler framework. Also, confidences are straightforward to obtain, while computing entropy introduces additional computa- tional overhead.
G.9 DIVERSE ENSEMBLE DISTILLATION
The Uni VQQ framework is designed to reduce overall computational costs by reducing dependence on large-scale LLM models. Although effective during the inference phase, the use of an ensem- ble model increases the computational costs of inference by TS-VQA and may potentially lead to higher latency. To address this issue, inspired by the findings of Alleen-ZZhh & Li (202)); Hebbal- aguppe et al. (2024) on the advantages of ensemble learning and knowledge distillation to transfer predictive accuracy and calibration, we use knowledge distillation to transform the calibrated diverse ensemble model (DE) into single VQA model, with the same architecture individual ensemble components, and is trained to learn from the ensemble's output distribution instead of the target labels, thereby preserving both the ensemble's accuracy and enhanced calibration.
The distillation process minimizes the Kullback-Leebler divergence between the output distributions of the ensemble and the distilled model, expressed follows:
N
fa(x))) T
fe(axe)
(kD(X:0.) 2
KL
110
where T' is the temperature parameter used to smooth the probability distributions, and a represents the softmax function. This process enables the distilled model to retain the ensemble's strengths while reducing the operational costs associated with deploying multiple models.
Accuracy & Calibration Performance Preservation. Table II highlights the effectiveness of this approach in preserving calibration and predictive performance across four VQA architectures, on VQA-v2 and COCO-QA datasets. The distilled model maintains the accuracy and calibration prop- erties the DE, while significantly reducing the computational overhead associated with ensembled models. As shown in Table [2] the increased inference time caused by ensembling is effectively remedied when the ensemble model is distilled into a single VQA model. These latency measure- ments were obtained by running models on a single A6000 GPU, with a batch size of 32, averaged over 3 runs.
Integration with Uni- LQA Framework. While all results presented in the main paper utilize the original ensemble models, we validate that distilled models can serve as efficient alternatives within the ini-VQs framework. Table [3 presents a direct comparison on the COCO-QA dataset, show- ing that distilled models not only maintain comparable accuracy but also achieve more efficient delegation patterns. Specifically: (1) VilBERT and VisualBERT demonstrate
6% reduction in LLM delegation while slightly improving accuracy (+0.26% and +0.39% respectively), indicat- ing enhanced confidence in local question answering. (2) CLIP- ViL maintains robust performance with minimal change in delegation behavior (+0.55%), preserving the ensemble's 's already efficient delegation pattern.
G.10 EVALUATION ON RECENT TRANSFORMER-BANE VQA ARCHITECTURES
We extend our evaluation to include ViLT (Vision-and-Langee Languag Transformer) (Kim et al. 2021 state-of-the-ass tranformer-bbsed model that represents recent advances in vision- language un- derstanding. Unlike the earlier architectures evaluated our main experiments (VisualBERT, ViL- BERT, ViL). ViLT employs simpler design that processes raw image patches directly through
31

---

Published as a conference paper at ICLR 2026
Table 11:
Performance comparison of diverse en- semble and distilled VQA across four architectures. "Diverse Ensemble requizes three VQA since comprises three independently trained models.
Table 12: Average inference latency (ms) comparison between the Diverse Ensemble (DE), and the distilled VQA model.
total parameters Distilled
Model
ACCT
Diverse Ememble EC24
ACC1
Disilled
VQA
Model
Diverse Ensemahle
Average Latency (mus)
Distilled VQA
Pythia
06.13
0.06
65.92
005
YQA-12
CLIP-VIL
Vii BERT
66.90
70.08
0.05
0.07
67.20
09.64
0,5
0107
Pythia
CLIP-ViL
59
4.29
3.71
VisulBERT
66.26
0.05
65.40
0.05
VILBERT
18.51
9.84
Pythia
65.51
0.02
68.06
002
VisuaBEET
15:49
9.61
COCO-QA
CLIF-VIL
BERT
69.94
70.00
70.43
0.08
0.05
VisuBBERT
67.38
0.02
Table 13: Performance and delegation comparison of diverse ensemble and distilled VS-VOA models on COCO-QA dataset, showing accuracy and LLM delegation percentages.
Diverse Ensemble columm corresponds to the results in the main paper presented in Table[] where Distilled VQA corresponds using the Distilled model the Ca ibbattdd TS-VOR.
Model
Diverse Easemble
Distilied VQ4
ACC1
Deleg v
ACC1
Delleg
ViLBERT
75.63
67.19
75.89
61.68
VisuaBBERT
74.34
13.46
74.73
67.66
CLIP-ViL
74.95
65.44
a transformer, without relying on pre-extracted region features, making it more representative of modern end-to-end vision-language architectures.
We train our Calibrated ViLT models using the same diverse ensemble configuration as other archi- tectures, with DRO hyperparameters À E {2, 3, 43 for COCO-QA and À E {8, 20, 100} on VQA-v2. All other training hyperparameters follow the original ViLT implementation.
Table 14 presents comprehensive comparison of Standard ViLT, our Calibrated ViLT, and Uni- VQA integration on COCO-QA, alongside the LLM-only baseline. The results demonstrate that diverse ensemble approach effectively improves calibration for modern transformer architec- tures. Our Calibrated ViLT achieves substantial calibration improvements, reducing ECE from 0.17 to 0.02, while maintaining comparable accuracy to Standard ViLT. Furthermore, when integrated into the Uni-VQA framework, ViLT achieves the highest accuracy of (76.33% on COCO-QA and 74.31% % on VQA-v2) and efficient LLM delegation of (70.47% on COCO-QA and 65.31% % on VQA- v2), demonstrating that our approach maintains its effectiveness on modern transformer-bas based ar- chitectures.
G.11 ROBUSTNESS TO DISTRIBUTION SHIFT AND GENERALIZATION
A critical concern for real-world VQA deployment is whether calibrated confidence scores remain reliable under distribution shifts, or our calibration approach, we conduct experiments on the AdVQA dataset (Sheng et al. 2021). an adversarial out-of-distribution benchmark specifically designed to challenge VQA model robustness through carefully constructed adversarial question-ansen ansser pairs.
questions. To evaluate the robustness of
We evaluate VQA models trained on VQA-v2 directly on the AdVQA test set without any finetuning, creating true -ut-t--disti evaluation scenario. This setup tests whether our diverse ensemble calibration maintains its advantages when facing distribution shifts that differ from the training distributions. We compare Standard VQA models (trained with cross- entropy loss) against our Calibrated models across four architectures: Pythia, CLIP-ViL, ViL- BERT, and ViSUalBERT.
Experimental Setup:
Out-of-Distribution Calibration Performance: Table 15 presents the performance of Standard and Calibrated VQA models on the AdVQA dataset. As expected, all models experience significant accuracy degradation and increased calibration error compared to in-distribution performance (Table []. However, the critical finding is that our Calibrated models consistently maintain better calibration than Standard models across all architectures.
32

---

Published as a conference paper at ICLR 2026
Table 14: Performance comparison of Uni-VQA with T8 VQ- models and L.M on ViLT architec-
tre..
VQA-22
COCO-QA
Model
A3CT
BCE4 LIM-De Dogg (W))
ACC1
RCE4, LLM-De log (S)
LLN-only (Mistral-BB)
72.03
0.27
Standard VQA
66.00
0.21
73.61
0.17
VILT
Calibrated (Ours)
66.44
0.07
73.89
0.02
Uni-VQA (Ours)
65.31
76.33
0.03
Table 15: Out-of-distribution performance comparison on AdVQA dataset (tsst))lit) All models are trained on VQA-v2, and evaluated on AdVQA to assess robustness of our Calibrated and Uni -VOA models under distribution shift.
AdvOA
Model
ACCT
SCE
SLM M----Deeen Deleg (50)
ACCT
SCEL
LLN- Pelee (903)
M.I.Monnly (Mistral-79)
69.09
0.31
1001
38.98
0.53
100
Standard VQA
0.36
Pythia
Cal sbrated (Ours)
66.15
0.06
31.5
0.12
Uni-VDA (Ours)
71.00
0.05
78,77
41.05
0.19
98.08
VQA
0.23
CLIP-VIL
Cal Calibrated (Ours)
70.05
0.08
31.95
0.06
Uni VDA (Ours)
72.98
0.07
69.86
38.11
0.11
99.86
VQA
VILBERT
Ca ibbated (Ours)
66.90
0.05
32.07
0.20
Uni UDA (Ours)
71.65
0.07
79.06
40.21
0.19
91.76
Standard VQA
0.14
31,41
0.36
VisuaBBERT Celibrated (Ours)
65.26
0.03
31,53
0.14
Uni-VgA (Ours)
70.95
0.08
77,87
40.77
0.16
96.03
Notably, we observe that across all architectures, our Calibrated models achieve lower ECE compared to Standard models on AdVQA. This demonstrates that the calibration benefits of diverse ensemble training are not limited to in-distribution data. While all methods exhibit higher ECE on AdVQA compared to VQA-v2, which is expected behavior under distribution shift, the relative advantage of our calibration approach remains consistent.
To get insights on how calibration behaves at a more granular level under distribution shift, we analyze the confidence distributions of correct and incorrect predictions on AdVQA. Figs. 15] and 16] present confidence histograms comparing Standard and our Calibrated ViLBERT models.
Analysis of Confidence Distribution Under Distribution Shift:
As illustrated in Fig. 15 Standard VQA exhibits severe overconfidence incorrect predictions, with a pronounced spike the highest confidence bin (around 1.0), indicating that the model is overcon- fident many incorrect answers. In contrast, incorrect predictions toward lower confidence regions, with substantially higher concentration in the low-confidence bins (particularly in the 0.0-0.3 range). The overconfident spike confidence 1.0 is greatly reduced in the Ca ibrated model. These patterns mirror the in-distribution behav- iors observed in Figs. [ 12 confirming that diverse ensemble training continues to shift incorrect predictions to lower confidence regions even under distribution shift.
Calibrated model shifts the distribution of
Implications for RAG integration: The above analysis reveals an important opportunity for inte- grating Retrieval-Augmenttee Generation (RAG) methods (Lewis et al. 2020 Guu et al. 2020) with Uni-VOA. Our diverse ensemble calibration reliably pushes OOD and knowledge-intenns
ques- tions toward the lowest-confidence region (precisely the regime where the LLM is invoked without TS-VOA candidates). To further validate this, we evaluated our calibrated TS-VQA (trained on VQA-v2) on OK-VQA (Marino et al.) 2019). knowledge-base based VQA benchmark requiring ex- ternal world knowledge. On OK-VQA, 56.6% of incorrect predictions from our calibrated model (accuracy: 20%, ECE: 0.11) fall below the lower confidence threshold, compared to only 27% for standard TS-VOA. This confirms that knowledge-heav questions are reliably routed to the lowest-confidence region. In Uni-VOA, this is exactly where RAG augmentation could be most beneficial -anefical-enhanccinn LLM accuracy on knowledge-intensive questions while avoiding the cost of invoking RAG every query.
33

---

Published as a conference paper at ICLR 2026
10000
Number of Samples
1000
750
750
300
300
250
250
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.50
0.75
1,00
Confidence
Confidence
(a) Standard VQA
(b) Calibrated (Ours)
Figure 15: Confidence histograms for incorrect predictions on AdVQA (OOD;
trained on VQA-v2): (a) Standard VQA and (h) Cal lbaatedd (ours).
Number Samples
Number of Samples
150
P50
1.00
250
8.50
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
Confidence
Confidence
(a) Standard VQA
(b) Calibrated (Ours)
Figure 16: Confidence histograms for correct predictions on AdVQA (OOD;
trained VQA-v2): (a) Standard VQA and (b) Calibrated (ours).
G.12 COMPREHENSIVE COMPUTE COST ANALYSIS
To provide complete picture our framework's efficiency, we present detailed breakdown of both training and inference costs. While training introduces upfront computational overhead, the significant inference savings in production deployments justify this initial investment.
Our analysis considers three primary computational costs: (1) Training Cost: One-time GPU hours required for ensemble model training, (2) Distillation Cost: Additional training to compress en- sembles (optional), (3) Inference Cost: Per-sample latency at inference time.
Table 16 presents the costs associated with training of our Uni-VQA components. distillation employed (optionally), it adds approximately one-third of the ensemble training time (equivalent to training a single model). Table 17 presents the effective inference costs our hybrid system, accounting for selective delegation.
Key Cost-Benefit Findings. Inference costs dominate real-world computational expenses in pro- duction systems. While training the ensemble models requires upfront investment of 15-366 GPU hours depending on the chosen backbone, this cost is quickly amortized in production deployments that process millions of queries daily. For instance, framework's 's improved inference efficiency translates to savings of approximately 11,000-35,000 GPU hours monthly compared to LLM-only inference.
scale of 10 million queries per day,
Table 16: Comprehensive compute cost breakdown for Uni-VOS components on VQA-v2 dataset. Calibrated models ensemble of 3 independently trained mod- els. Training measured on A100 GPUs, inference latency on A6000 GPU.
Model
Parameters (M)
Training Time (GPU Hours)
Avg Inference Time (ms/sample)
Calibrated Models (Ensemble 3)
Pythia
3 147
3 5 15
3 3 =
ViLBERT
3 250
3 3 47
141
3 9 27
VisualBERT
3 x 114
3 19 = 57
3 9 = 27
CLIP-ViL
3 256
3 122 = 366
3 23 = 69
BEiT-3
x 900
3 x 72 216
3 9 27
Reference: I.M--only Baxeline
Mistral-7B
0* Pre-trained model used without additional training
7,000
534
34

---

Published as a conference paper at ICLR 2026
Table 17: Effective inference latency comparison between Uni-VQA and LLM-only baseline. Delegation % X (LLM)-
indicates frequency of L.M invocation. Effective latency computed as: ivQA +
Delegg
100
TS-VQA
Latency
Delegation
Effective Latency (ms)
Speedup
Backhone
(ms)
%
LLM-only
Mistrul-7B only
100%
534
1.0x
Pythia
9
78.8%
426
ViLBERT
27
79.1%
447
1.19x
VisualBERT
27
77.9%
440
1.21x
CLIP-ViL ViL.
69
69.9%
440
1.21x
BEİT-3
27
35.9%
217
2.46x
Table 18: Hyperparameters for training our Calibrataa ibbrttdd VQA models.
VQA Model
Aa
Aa
Pythia
100
1000
ViLBERT
8
20
100
VQA-v2
VisualBERT
10
20
100
CLIP-ViL
20
100
1000
BETT-3
200
500
Pythia
4
200
ViLBERT
3
COCO-QA
VisualBERT
2
3
5
CLIP-ViL
2
50
BETT-3
0.5
G.13 EXPERIMENTS REPRODUCIBILITY
In this section the hyperparameters used for training the Diverse ensemble based Calibrated model. The DRO loss can be computationally expensive to optimize. To mitigate this, similar to the approach in (Sapkota al. 2024). we employ regularized version of the loss function, defined as:
(pl
1 N
I
((0)
max
Σ
Wn, θ) AD,
(21)
W. T
which has a closed-form solution demonstrated in (Sapkota et al.
2024
((O)
w 1xn 9)
(22)
where, wn is given
W exp( ((x...)) A
(23)
Dj=1 exp( 1(x,,
In this setup, hyperparameters the À values corresponding to the diverse models in the ensemble. For all of our experiments, we set the ensemble count to 3, resulting in three hy- perparameters: Åι, λ2, and Ag. For training our Calibrated TS-VOA models. We A E {8, 10, 20, 50, 100, 200, 500, 1000) in our experimentation, and select the final parameters based on the performance on the validation set, to obtain the desired ECE. The final values of hyperparameters are given in Table 18 Due to computational overhead of LL-M-bssed inferences, we report results based on single run.
H QUALITATIVE ANALYSIS
Fig. 17 presents qualitative examples illustrating how Uni-VQ2 VQA performs confidence-guided knowl- edge exchange between TS-VQ2 VQA. and the LLM. For each example, show the input image- question pair, the TS-VOA model's 's initial prediction with its confidence score (spanning low to high confidence bins). and the top answer candidates proposed by TS-VOA. We then report the LLM's prediction (and its correctness) when prompted with different numbers of answer candidates, reflecting the behaviors discussed in section 3.3
35

---

Published as a conference paper at ICLR 2026
Q: Which estrological eign represented by this animal?
Q: What is the large white thing bahind head?
Lowest Confidence
TB-VOA
TS-VQA
0.05
Condidates: notrinng no, dartt
Candieortac: oounter oebinot. sable
Water heater
nothing
LLM
unknown
LLM
counter
nothing
counter
Q: What is the doing?
Q: What the bowl made
Low Confidence
IS-VOA
Anamar: exting
ISAGA
Anawer: ceraiic
Coné:
Candidates: oating standing amiling. looking, posing, teliingg daklig prctoss diikkng stoing watching
Candidatasi cssumiic wood, poroelaln, metil planti, glase. staak stons, marblu, clay
Yot... playing
Ext
eating
ceramic
LLM
smiling
metal
Taiding
Moderate Confidencel
Q: la tattii gourmet dish?
Q: Are there leaves with
IS-VOA
Answert 0.58
brown edges?
TS-VO4
yoa
Candidates: no, yos, unlinows, not surs, don't know nothing, plezs, food, idon't know, canttsal
Candidetsa: yea, no. unimnoo, 6,not suro, whiks, groon,
nothing: 2
yes yes
LLM
spoont com
yes
Q: Are these people in
High confidence
ISAGA
Conf: 3:58
doing?
TEANQA
Answer: stingg
Conft
Can dinddetee: yess, unknoww, dent'i
Candidates: sVits mooebcrriigg
jumoing Ryng skinting,
nothing
deing stancling mothing yos
tet
Ext
sking
LLM
piting
you
1e10
yes
flying
Figure Qualitative examples demonstrating the knowledge exchange various confidences TS-VQA
These examples reveal a consistent trend across confidence regimes. In the lowest-confidence bin, both the TS-VQA prediction and its candidate set often misleading; consequently, providing these candidates can steer the LLM toward an incorrect answer. In this regime, prompting the LLM without answer candidates (top-0) frequently yields the correct response. In the low-confidence regime, the LLM benefits from receiving larger candidate set (e.g., top-10), which can provide useful partial signal despite the TS-VOA's uncertainty. As confidence increases, fewer candidates become sufficient (and can be preferable) since the top candidates are more reliable and additional candidates may introduce noise. Finally, the highest bighest-confidence regime, where the calibrated TS-VQA prediction is most reliable, the system can accept the TS-VQA answer directly without delegating to the LLM, saving expensive LLM computation while maintaining accuracy.
I BROADER IMPACT STATEMENT
Modern large language model (LLMed systems have revolutionized Al applications, demon- strating remarkable capabilities diverse domains, including healthcare, finance, and creative in- dustries. Yet their widespread adoption comes at substantial environmental cost, raising concerns about sustainability and their environmental impacts. Studies (Strubell et al. 2020 Patterson et al. 2021) have highlighted the environmental costs of training and deploying these models, highlight- ing the significant carbon footprint associated with large-scale AI, emphasizing on the need for more
energy efficient Al solutions. Furthermore, reports (Patterson et al. 2021 Weidinger et al. 2022
36

---

Published as a conference paper at ICLR 2026
Luccioni et al. 2024) indicate that inference accounts for substantial AI workloads, often exceed- ing the energy costs of model training and development, due to their usage at scale. This underscores the urgent need to develop AI systems that balance computational efficiency with performance.
In line with the principles of Green AI (Schwartz et al. 2020 prioritizing innovation while min- imizing resource consumption and computational costs our work proposes a framework that lectively and dynamically utilizes LLMs when their unique capabilities are truly needed. Our ap- proach identifies opportunities use smaller, task-specific models for routine tasks while reserving resource-intensive LLMs for complex queries that demand their advanced capabilities. This selec- tive deployment strategy can significantly reduce the environmental footprint of Al systems without compromising their performance.
While our approach improves trustworthiness through calibration, and efficiency of using LLMs by reducing overreliance on the LLMs, several negative aspects merit further discussion. Firstly, cali- brated confidence scores are critical in domains like medical, autonomous driving, or surveillance, where incorrect answers can have serious consequences. Although our framework improves relia- bility, high model confidence does not guarantee correctness, and such high-stakes scenarios, human supervision must make informed decision. If such confidence scores are interpreted as definitive indicators of correctness (especially by non-expert users), this could lead to overtrust and potential harmful decisions in sensitive contexts. Secondly, framework involves dynamic dele- gation of queries to LLMs, which may reside in third-party systems. In scenarios involving sensitive private visual data, delegation to external L.M (particularly serious privacy risks. Moreover, unless made explicitly transparent to users when delegation occurs, this can lead unintended data exposure and ethical concerns around informed consent.
not hosted locally), poses
LIMITATIONS AND FUTURE WORK
Our study has several limitations. First, while our approach employs confidence-based delegation from TS-VQA to the LLM with answer candidates, it does not leverage additional mechanisms, such answer consistency checking or refinement techniques (Srinivasan al., 2024, Khan al., 2024 Prasad et al. 2023), which could further boost the performance, when answering is delegated to LLM. Second, our approach still lacks the systematic way of providing the well-calibrated uncertainty estimates on the L.L--generteed answers. While calibrated confidence estimates of our Calibrated USQVA provides better reflection on the question difficulty, accurate confidence estimation of the LLM-generated answers can be important, particularly in safety-critical domains such as medical, or security surveillance. As uncertainty quantification in LLMs remains an ongoing research challenge, we leave the development of more robust LLM calibration strategies for future work.
K SOURCE CODE
The source code is available at: http:///gg.aatccccc//aaaaaaaaaaa
37


Reviews:
[{"review_id": "oREWUGSGXP", "reviewer": "Reviewer_7cPv", "summary": "This paper proposes a novel framework called Uni-VQA for visual question answering, aiming to address challenges in directly applying large language models to VQA, such as suboptimal performance in specialized domains, high computational costs, and lack of uncertainty quantification. The key contributions include: (1) developing a calibration technique based on a diverse ensemble to improve the reliability of confidence estimates for task-specific VQA models; (2) introducing a confidence-guided knowledge exchange mechanism that dynamically decides whether to delegate to an LLM and how to provide candidate answers based on TS-VQA confidence scores, optimizing accuracy and efficiency; and (3) validating the framework through theoretical analysis and extensive experiments on VQA-v2 and COCO-QA datasets, showing superior performance over using LLM or TS-VQA alone while significantly reducing computational overhead. The paper also explores knowledge distillation for faster inference and provides analysis on carbon emissions and latency, highlighting environmental sustainability benefits.", "questions": "1.How does Uni-VQA handle modal missingness or distribution shifts? For example, if image quality is poor or questions are out-of-distribution, does confidence calibration remain reliable? \n\n2.The paper mentions that confidence thresholds are determined using a validation set. Could you provide more specifics about the optimization process? What objective function was used to balance accuracy and efficiency during threshold selection?\n\n3.While the paper demonstrates inference efficiency, could you provide more details about the training computational costs of the diverse ensemble approach? How does the training time scale with the number of ensemble members?\n\n4.How does the framework handle concept drift or distribution shifts over time? Have you considered mechanisms for continuous adaptation of the confidence thresholds?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper novelly integrates confidence-guided mechanisms with LLM-VQA collaboration, differing from traditional model cascades or simple delegation. The dynamic candidate answer selection based on confidence intervals is a creative combination, underexplored in existing work.\n\nThe method has theoretical depth and experiments are comprehensive, covering multiple VQA modelsand datasets with consistent results. Ablation studies validate component importance. Writing is concise, and figuresintuitively explain complex concepts, with clear mathematical derivations.\n\nThe work directly targets AI scalability and sustainability, reducing LLM carbon footprint, with potential impact on high-stakes applications , aligning with green AI trends.", "weaknesses": "While tested on VQA-v2 and COCO-QA, the paper does not include more diverse datasets (e.g., medical VQA or long-tailed distributions), potentially limiting generalizability. Also, LLM usage is limited to Mistral-7B and LLaVA, without extension to larger models , failing to fully assess scale effects.\n\nThe calibration technique relies on diverse ensembles, increasing training overhead, and although distillation mitigates this, it may affect deployment ease. \n\nComparison with recent VQA methods (e.g., Transformer-based variants) is limited; the paper focuses on traditional baselines.\n\nTheorem 4.2 relies on inverse relationship between entropy and confidence, but strict proof in multi-class settings depends on uniform distribution assumptions, which may deviate in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tCkmFZdDwf", "reviewer": "Reviewer_H4gC", "summary": "The paper proposes Uni-VQA, a method employing an LLM and a task-specific VQA model to efficiently answer questions. The task-specific VQA model is calibrated to provide reliable confidence scores. With the confidence score, the framework whether the answer from the task-specific VQA model should be processed by the LLM. Extensive experiments show the effectiveness of the method.", "questions": "1. What are the advantages and disadvantages of the proposed method compared to RAG methods?\n2. What is the training costs of the task-specific VQA models and the distillation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation for introducing task-specific VQA model is reasonable and practical.\n2. The calibration of task-specific VQA model provide reliable confidence score, enabling the interaction between the two VQA models.\n3. Extensive experiments show the effectiveness of the method.", "weaknesses": "1. RAG (Retrieval-Augmented Generation) methods are not discussed and compared. The task-specific VQA model serves as a role of providing specific knowledge for LLM, which is similar to external and up-to-date information in RAG methods. The advantages and disadvantages of the proposed method compared to RAG methods should be discussed and compared.\n2. The training cost of the task-specific VQA models and the distillation is not reported. It is unclear whether it would be the bottleneck of the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6T46SED3Um", "reviewer": "Reviewer_nhPm", "summary": "This paper proposes Uni-VQA, a hybrid VQA framework that uses calibrated confidence scores from a task-specific VQA model to decide when and how to involve an LLM. High-confidence questions are answered locally, low-confidence ones are delegated to the LLM, and intermediate cases use candidate answers for collaboration. The approach improves accuracy while reducing reliance on costly LLM inference.", "questions": "See weaknesses please", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The related work section is well organized and clearly positions this framework among prior VQA, calibration, and LLM-augmented systems, making it easy to understand its practical motivation.\n\n- The theoretical derivations and more complex details are moved to the appendix, which helps maintain good readability in the main paper.\n\n\n\n- The experimental evaluation is comprehensive, including multiple backbones, datasets, and ablation studies.", "weaknesses": "- Confidence threshold decisions could use more clarification\n  - The routing strategy depends on two confidence thresholds (𝑙,𝑢). It would be helpful to include more explanation on how these values are selected and how sensitive the method is to different threshold settings across datasets or models.\n\n- The effectiveness of TS-VQA candidates may vary depending on the confidence level\n  - When the TS-VQA model has low confidence, its proposed candidates may negatively influence the LLM’s reasoning. The current strategy of suppressing candidates only in low-confidence cases is reasonable, but additional analysis on when candidates are beneficial vs. harmful would provide deeper insight into this interaction.\n\n- LLM output reliability is not fully addressed\n  - The reliability of LLM outputs is not modeled. Since the LLM handles the most uncertain samples, having some form of uncertainty estimation or error check on the LLM side could further improve the robustness of the overall system.\n\n- Some figures have relatively small, which affects readability. For example, Figure 3 and Figure 14.\n\n- There is no Ethics Statement or Reproducibility Statement in the paper, which are required by the ICLR submission guidelines. Authors may miss this information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XwGmzPdjLc", "reviewer": "Reviewer_FgQE", "summary": "The paper proposes a calibrated, rule-based router to combine TS-VQA and LLM to balance power consumption and performance. The first step is to apply the existing DRO method to calibrate confidence. Then the calibrated confidence is used to route questions to different decision models with varying capability vs power consumption combinations.", "questions": "Please give the exact form of $w$ and its dependence on $\\lambda$.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper offers a practical cost-aware collaboration between TS-VQA with an LLM to save energy consumption.\n- The motivation and trade-off framing are convincing.", "weaknesses": "- Modest novelty. The proposed system is essentially a confidence-controlled routing. \n- There seems to be a lack of comparison between ensemble-aware fusion vs a distilled single model, to fully understand the tradeoff between reliability and latency.\n- The routing is hard-coded. There is no comparison to a learned router or agentic alternatives (that add a light verifier before calling the LLM).\n- Sustainability is a central motivation. But the benefit is not sufficiently evidenced with proper accounting.\n- The writing can be improved. For example, the DRO paragraph introduces $\\lambda$ abruptly, leaving the mechanism unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)