Human Uncertainty-Aware Data Selection and Automatic Labeling in Visual Question Answering
Authors: Jian Lan
Zhicheng Liu
Udo Schlegel
Raoyuan Zhao
Yihong Liu
Hinrich Schuetze
Michael A. Hedderich
Thomas Seidl
Venue: ICLR 2026 Poster (2026)
Keywords: Large Vision-Language Model
Visual Question Answering
Primary area: generative models

Abstract: Large vision-language models (VLMs) achieve strong performance in Visual Question Answering but still rely heavily on supervised fine-tuning (SFT) with massive labeled datasets, which is costly due to human annotations.  Crucially, real-world datasets often exhibit *human uncertainty* (**HU**) — variation in human confidence across annotations, but standard SFT simply optimizes toward the most frequent label, disregarding HU distributions.  This leaves two open questions: *How does HU affect SFT*, and *how can HU be effectively leveraged in training?*  In this work, we first conduct a systematic evaluation of VLMs across varying HU levels. We have two key findings:  (i) surprisingly, high-HU samples contribute little, or even degrade, model performance, and  (ii) naively training on the full dataset yields under-calibrated models that fail to capture HU distributions.  Motivated by these findings, we introduce **HaDola**, a **h**uman uncertainty-**a**ware **d**ata selection and aut**o**matic **la**beling framework.  HaDola operates in four stages: **discriminate**, **self-annotate**, **error trigger**, and **training**, to iteratively identify harmful samples, prioritize informative ones, and bootstrap from a small seed set (5% of data).  Our approach substantially reduces reliance on costly HU annotations and makes VLMs more accurate and better calibrated.  Extensive experiments on VQAv2 and VizWiz datasets demonstrate that HaDola consistently matches or outperforms state-of-the-art baselines, with less training data.  Our work highlights the importance of explicitly modeling HU in SFT, suggesting better utilization of HU is more effective than merely scaling up dataset size.

HUMAN UNCERTAINTY-AWA DATA SELECTION AND AUTOMATIC LABELING IN VISUAL QUESTION ANSWERING
Jian Lan Zhicheng Liu', Udo Schlegel".,, RRoouan Zhao Yihong Liu
Hinrich Schütze!? 1.2 Michael A. Hedderich Thomas Seidl 1.2 University of Munich (LMU), "Munich Center of Machine Learning corresponding author: lan.dbs... ifi. mallmmu.d. de
ABSTRACT
Large vision-language models (VLMs) achieve strong performance in Visual Question An- swering but still rely heavily on supervised fine-tuning (SFT) with massive labeled datasets, which is costly due to human annotations. Crucially, wallworld datasets often exhibit human uncertainty (HU) variation in human confidence across annotations, but standard SFT simply optimizes toward the most frequent label, disregarding HU distributions. This leaves two open questions: How does HU affect SFT, and how can HU be effectively lever- aged in training? ? In this work, we first conduct the systematic evaluation of VL.S across varying HU levels. We have two key findings: (i) surprisingly, high-HU samples contribute little, or even degrade, model performance, and (ii) naively training on the full dataset yields mnder-calibrated models that fail capture HU distributions. Motivated by these findings, we introduce HaDola, human uncertainty aware data selection and automatic labeling framework. HaDola operates in four stages: diseriminate, self-annotate, error trigger, and training, to iteratively identify harmful samples, prioritize informative ones, and bootstrap from a small seed set (5% of data). Our approach substantially reduces reliance costly HU annotations and makes VLMs more accurate and better calibrated. Extensive experiments on VQAv2 and VizWiz datasets demonstrate that HaDola consistently matches outperforms of-thoof-the-at baselines, with less training data. Our work highlights the importance of explicitly modeling HU in SFT, suggesting better utilization of HU more effective than merely scaling up dataset size. Code: http://ggitubb.cmmmmmuuunnhnhn
1 INTRODUCTION
Large vision-language models (VLMs) (Wang et a..] 2023 Liu et al., 2024h Wang et al. 2024b Bai
ct al. 2025) have achieved impressive progress on multi-modal tasks, with Visual Question Answering (VQA) (Goya et al., 2017) being a critical benchmark for evaluating vision-language understanding and generation Zhang et al. (025). h. Despite their strong performance, VLMs trained with the prevailing strategy of supervised fine-tuming (SFT) face two major limitations. First, they rely heavily massive comaan-nnnotttd data and simply scale up the dataset size without investigating how individual samples contribute training. Some studies have explored sample difficulty in VQA (Karamcheti et al.) 2021 Tan & Bansal 2019). but they still solely depend on SFT and offer no effective solutions to data selection or reducing annotation costs. Secondly, for each sample, current VLMs optimize only for the most frequent answer label while ignoring alternative answers and, crucially, overlooking human confidence distributions (Wang et al., 2022aa: Liu et al. 2024a Wang al. 2023). Human confidence, also known as human uncertainty (HU) (Lan et al. (025). ) refers to the fact that, for a sample, different annotators may provide diverse answers with varying confidence levels. HU has been demonstrated as an important and negliggii factor for VQA in their work. As

---

Inputs
Confidence
Evaluation
Question: What colore are the uniforms?
Answer
Confidence Scores
HaConf
VLMs Prediction
VQA.Acc
Nave Blue
Neny Blu
Bluek Wrike
0.96
0.5
0.99
0.827
Blonk White
Black
0.285
Figure 1: Illustration of human uncertainty in VQA. Different annotators may provide different answers with varying confidence levels. The HaConf Score is the average score of all annotators for each answer [T] On the right, it shows the VQA-Accuracy 4.1 of different model generations. The metric fails to reflect the HU difference, and leaves an open question: Is that an accurate metric to evaluate VL.Ms?
shown in Figure I given question-image pair input, different humans have different answers, and even for humans providing the same answer, their uncertainty label can be different. Neglecting HU can leave models mis-calibrated (Guo et al. 2017 Baan et al. 2022) and drive indiscriminate training data expansion, instead of exploiting HU for more effective training strategies.
To explore how HU can be appropriately incorporated in training, this work raises the following research questions: RQI: To what extent does HU influence the SFT VLMs, and what are the beneficial or harmful samples? RQ2: How can HU be incorporated into the training process to achieve high accuracy and better calibration? RQ3: Given the prohibitive cost of large-scale HU annotations, how to design efficient strategies that leverage only small portion of HU-labeled data while maintaining strong performance?
To answer these questions, we first conduct a comprehensive evaluation of VLMs on datasets novelly stratified by different HU levels. Surprisingly, our analysis shows that high-HU samples contribute little or even harm SFT, and indiscriminate training on the full dataset yields under-calibrated models that fail to capture real-world HU distributions. These findings highlight the role of HU: it can serve valuable signal for guiding data selection and training strategies. Building on this insight, we introduce HaDola (human uncertainty-awwat data selection and automatic labeling), novel framework that explicitly integrates HU into VLM training. HaDola follows a four-stage pipeline with advantages: (1) discriminate, to identify harmful versus beneficial samples; (2) self-annotate, to automatically refine annotations guided by HU; (3) error trigger, to detect and correct potential error accumulation; and (4) training, to fine-tune VLMM with our novel loss function. Starting from a small HU-annotated seed set (around 5% of all data), HaDola iteratively expands training supervision in a self-evolving manner, substantially reducing reliance on costly HU annotations.
Moreover, we find that the widely used evaluation metric VQA-Accurace (Antol et al., 2015). being purely frequency-based, overlooks HU by treating all answers equally. To better align with HU, we propose a complementary measure, HU-acc, which weights predictions by human confidence and reveals that low- and medium-HU samples provide more effective supervision.
We conduct extensive experiments on two widely used benchmarks, VQAv2 (Goyal et al. 2017 and VizWiz Gurari et al. 2018), with recent state-of-the-an art (SOTA) VLMs including LLaVA (Liu et al. 2024b). Qwen- VL (Bai et al. 2025), Intern (Wang et al. 2024b), and BEiT3 (Wang et al. 2023). Results show that HaDola consistently outperforms strong baselines, achieving both higher accuracy and more reliable calibration, while requiring substantially less annotated data.
Our Contributions are: (1) For VQA, we provide the first systematic study of how HU affects SFT in VLMs, identifying the essential role of HU as both source of harm and useful training signal. (2) We propose
2

---

HaDola, human- aaaaaannnnncettnty-wwwwaaa generalizable and data-efficient framework that integrates data selection and automatic labeling into VLM training. requiring only a small portion of HU annotations. We validate HaDola on VQAv2 and VizWiz with multiple SOTA VLMs, demonstrating that it achieves superior aceuracy and calibration compared to baselines. (3) We identify the limitation of the standard majority vote-based VQA-Accuracy metric, which ignores HU, and for the first time, advocate for incorporating the non-negligible HU into evaluation protocols to better assess VQA in real-world settings.
2 RELATED WORK
VQA and Human Uncertainty. VQA was first introduced by Antol et al. (2015), and follow-up work (Tan) & Bansal 2019 Li et al. 2022 [aoo et al. 2022; Wang et al. 2023) has mainly optimized the standard VQA-
Accuracy metric. BEiT3 Wang et al. (023) remains the SOTA task-specific model but lacks zero-shot generative ability. Recent VLMs (Bai et al. [2025] Liu et al. 2024b Wang et al. (024b) rival or surpass BEiT3, yet still rely primarily on supervised tine-tuning (SFT) without explicill modeling HU. The non-negligible HU has only recently received research attention. Lan et al. (2025) introduce the HUD score to quantify HU in VQAv2 with BETT3, but leave open how HU can be exploited in training and whether it benefits beyond accuracy. Only few datasets explicitly provide HU annotations. For example, VQA 2.0 (Goyal 2017) and VizWiz (Gurari et al. 2018) include both answers and uncertainty labels from ten annotators ("How certain are you: yes, maybe, no?"). In contrast, other common datasets such as OK- VQA (Marino ct al. 2019) and GQA (Hudson & Manning 2019) do not provide HU labels. Yet prior studies (Goyal et al. 2017 Guran et al. 2018 Tann & Bansal [019] Wang et al.) 2023) have largely ignored this information, training models only on the majority label to maximize VQA- AQA-Accuracy. More recent work shifts attention to model uncertainty calibration across multiple answers (Yang et al., 2024 Xiong et al. [002] Lann et al.) 2026). However, a key challenge remains: HU is rarely available at scale, making it difficult to assess whether model predictions faithfully reflect human uncertainty. This motivates the need for methods that can leverage HU more effectively and, ideally, reduce dependence on costly human annotations.
ct al.
Sample-aware training in VQA. Recent work has examined what data and how much data to use for training. Karamcheti et al. (202T) study this via active learning (AL) (Lowell et al. (219), showing that difficult samples cause AL. to fail without offering ways identify or handle them. Tan & Bansal (2019) measure sample difficulty by annotator label diversity, but only analyze label frequency and ignore HU. Lan et al. (2025) are the first explore HU in VQA, evaluating BEiT3 Wang et al. (2023) but lcaving generative VLMs and HU's role in training under-explored. To reduce costly VQA annotation, [Sun et al. (2024) augment medical-domain data with DPO (Rafailvv et al. 2023). but assume perfect training distributions and overlook HU. Other efforts, such as domain generalization with customized loss (Huang et al. (025), ) semi-supervised learning for video-QA (Mitra & Soundararajan 2024), and selective prediction for reliability (Dancette et al. 2023: Whitehead et al. (022) likewise aim to reduce annotation cost or improve robustness. While these approaches improve efficiency and robustness, they typically assume perfectly reliable data and do not investigate the potential benefits or harms of training samples with high HU. This leaves open questions of how to systematically select and utilize HU-aware samples to improve both model accuracy and calibration.
3 BACKGROUND
Measuring Human Uncertainty. In a training sample, image-question-answe triplet, 8 == (i, q, A). the (i, q) is image-question input pair, and A is answer label set with 10 independent humans' annotations. In each human's annotation h2s one annotator gives their answer an to q and also a confidence level en indicate whether this annotator is confident that a, is correct. Ca belongs to pre-defined category ['yes", "no", 'maybe"]. To quantify cn. the latest study Lan et al. 2025)) assigns different confidence scores Cn. mapping 'yes" to 0.99, 'no' to 0.01, and 'maybe" to 0.5:
[aa G,) 10 mn Cn [0.99, 0.5, 0.01]).
3

---

Within A, annotators who provide the same answer d' may assign different confidence scores Cn. We denote the average of these scores as ''- the human average confidence of d',, (abbreviated as HaConf, not a new metric but a shorthand for clarity). Building on this, Lan et al. (2025) introduce HUD (human uncertainty across disagreement), which aggregates HaConf scores across all different answers of a sample to quantify its HU degree. HUD is the most recent measure for capturing sample-level HU. The above are denoted as (note that HaConf is baased and HUD is sample-based);
1
HaConf(d(m) Cm
Σ 10:0 =0m
Ca
HUD($) = m
(1)
in On d'm
Distributions and Calibrating towards Humans. For a model M, the prediction distribution M(s) corresponds the probabilities (s) y|X = 8) the model assigns to each class y. In VLMs, PM is derived from the last hidden state of the neural network over the well-known logits, denoted as L(s) 1/, lg .... bm], with m as in Eq. PM is represented by applying the standard Softmax normalized function to L(s): (s) = Softmaz ( L (s)). The human distribution // (s) [ç,..... cm is defined by the HaConf scores across different answers (Eq. [T. The standard way to assess alignment between human and
model distributions is the -L-vivergence (KL) (Kullback & Leibler 1951). Given a human distribution (s))
and model distribution (s), the KL score measures how much H (s) deviates from (s):::
(H((((()) |I PM(()) Σ H(s) log H(s) PM)
m
(2)
4 HUMAN UNCERTAINTY-AWAR DATA SELECTION AND AUTOMATIC LABELING
Studying different HU levels requires dividing a dataset into varying sets. We first introduce our novel split and evaluation based on HUD and HaConf, and then introduce HaDola.
4.1 SAMPLES WITH HU LEVELS AND HU HU-BASED ACCURACY
Following prior work TTa & Bansal 2019 Lan et al. 2025). we adopt a three-level categorization of HU (low, medium, high). In (Lan et al. 2025). datasets are partitioned into three equal subsets by HUD scores, but this does not reflect reality, where low-HU samples dominate. Therefore, unlike their work, we divide HU by evenly splitting HUD intervals: [0.01,0.35 0..33 (high), (0.33,0.66) (medium), and [0.66,0.99] (low). As shown in Figure aa), our splits align better with samples' HaConf upper bound, yielding smaller variance and more balanced means. By contrast, prior splits blur medium and high sets, inflate the low set, and produce higher variance. Our design therefore achieves a sharper and more realistic distinction across HU levels.
Moreover, measuring VQA model accuracy is challenging. The standard metric VQA-acc AAtol et al. 2015)
is defined as VQA-Acc(a) min #humans that said a 1 } where only the majority vote is considered for a
3
model-generated answer a. However, ignores HU: even when multiple annotators agree, their confidence may remain low, yet the sample is still rewarded with high score. To address this, we propose HU-acc, a VQA-Ace(a), which incorporates both label
HU-weighted variant defined as HU-ace(a) a) = HaConf(a) frequency and HU. As shown in Figure bb. bb). HU-acc yields a clearer separation across subsets: training on low-, medium-, and high-HU data leads to steadily lower accuracy as HU increases. This reinforces that high-HU samples harm the SFT gained on the other two sets. Combining low- and medium-HH -HU samples further improves performance, while adding high-HU data again impairs training, confirming their limited reliability. In contrast, VQA-ace shows only marginal differences and fails to reveal the harmful effect of high-HU samples. This discrepancy highlights the insufficiency of frequency- -base evaluation and motivates the need for HU-sensitive measures. Accordingly, replace the conventional VQA-ace supervision signal
4

---

Comparison Data Distributions Training Set
Gest ip
Lo
MaConl Upper Bound
Annoladon Upper
10 Mediuon-Mi HQ
Lowe-W0 H0
Figure 2: Results VQAv2, (a): comparison of data distribution with different set splits design. (b): Effects training samples with different HU degrees, and comparison with VQA-ace. The L, M. H stands for low, medium, and high. We downsample the L and M subsets to match the sample size of H. VizWiz results reach consistent findings and are Appendix [A
with HU-acce during SFT to enable fairer and more informative comparison. These HU-based splits and metries form the foundation for introducing HaDola, our HU-aware training framework.
4.2 HADOLA PIPELINE
Overview. HaDola is model-agnostic and generalizable framework that uses HU to reduce annotation cost while matching or surpassing SOTA performance and simultancously improving calibration. The framework operates in four key stages -discrimiaat, self-annotate, error-trigger, and training summarized in Algorithm | and illustrated in Figure the following, we describe each stage in detail, starting from the initialization of the HaDola.
Initialization. To capture the inherent and non-negligible HU data, HaDola requires a small seed set with human annotations as an anchor for confidence distributions. Let Sn be small labeled seed set (randomly selected 5% of data from the entire dataset for human annotation). Minit an initial VLM without SFT, and S, the remaining unlabeled data. HaDola first fine-tunes Miit on Sp to obtain Mas which is duplicated: one copy fixed as the HU reference model, the other used as the initial model for the subsequent iterative training.
Discriminate. HaDola aims to identify and exclude high-HU samples in each step since they are assumed to contribute neither to learning nor efficiency, which instead consumes additional human and computational resources. HaDola calculates the average KL scores between Mh, and humans So:
T1 = = [DKL (a) || Man( T2 Buems, [DkL (a) (na(a]) h = Eoo Um [Mn (a)] (3)
In Eq. 3 Is, and ms, are low- and medium-HU subsets of Sg. and are their average KL scores between the human reference model and true HU distribution, with Ti < T2. h is the mean human confidence distribution over Is, U m..- For each unlabeled sample E S, selected in this step / (1% of S,), HaDola
computes its KI. score kl, between current round M, and hu: kl, = DKL. (hw M((u), and retains only those with kl, [T] σ, 72 + o]. Samples outside this interval are considered high-HU or outliers and discarded, where a is the standard deviation of all KL scores in Eq. 3
Self-Annotate. After the unhelpful instances discarded in the discriminate stage, HaDola annotates each remaining instance u by leveraging the last round model M,-1: i.. = (M)-1(u), where Iu is the response of M,-1 when answering It then constructs the pseudo training pair (u, U.).
Error Trigger. To avoid the potential accumulation of errors pscudo training pair (u, I.). we introduce an error trigger mechanism, where use gradient consistency (Mirzadeh al. 2020) and TracIn-mini (Pruthi jet al. 2020), where the former ensures that the gradient direction yielded by (u, Da) is consistent with human- labeled data with the current model parameters, thus serving theoretical criterion filter out pseudo labels
5

---

Algorithm 1: HaDola Pipeline.
Input: Seed set So (5% labeled), unlabeled set S,, initial model Mass total rounds T.
Output: Final model My.
Initialization: SFT Mia on So to obtain Mau: duplicate as reference Mau and training copy Mg. Compute thresholds T1, T2. hy once on Sy using Miu (Egg. B
( = to do
Discriminate: each u S., compute kd, under Mc retain if kl, [T] o, 72 a].
Self-Annotate: For each assign pseudo-lahel B. Mr-1 (M) and form (o)) Error Trigger: For (u, fu), compute 5y and Smaciai keep if 8q 2 Ty Suacio S тi Training: Fine-tune Adr- on retained samples from all rounds with Ciaaela (Eq 8 update Mt.
return My.
that conflict with reliable supervision; the latter traces the influence of (u, Dw.) across checkpoints to quantify its contribution to the model, enabling us to down-weight or discard harmful pseudo pairs. Together, these two components provide principled mechanism to safeguard against enror accumulation in anlo-annttteed training. More theoretical derivations are in Appendix B.2 For pseudo-labeled sample (u, 9n) at the tth round with model parameters t2, we compute its gradient and the reference gradient, the average gradient over the human-labeled seed set So as:
g(u, tu. 0z) Vo, (fo, The gradient consistency score is then measured by cosine similarity:
Jeet (0,) S. Vo, ((fo, (x),
(4)
(g(u, Uns 0z), gret (0.))
Sy(u, Dus 0z) (9 u, Bui 0e)1 l (ft)ll
(5)
In addition, to capture the global effect of training, we employ a simplified TracIn estimator that only requires the initial model 0g and the current model 0g:
Suraci (u, Fu) = (g, Gu: 0o). (Oo)) + (g(u, ju: 0c), Vo
(6)
where ((a is the validation loss. pseudo-labeled sample is retained only it passes both criteria:
keep(u, u,, fu) T[ [[1 Gu: 02) Tg Surcin(M, Uu) S T.).
(7)
where thresholds Tg and T, are calculated based on low-/mediun-HHU subsets. We leave the details of the selection of Tg and 5 in Appendix B33
Training, We design training objective that jointly pursues predictive accuracy and uncertainty calibration by leveraging a human-uncertainty (HU) reference model. The overall loss consists of three terms:
CtaDola E(CE(y, Mo)] + 8 X (DKL(H || Mo) D H MMu)).
(8)
T DRI((MuU 1:) Mo(- ( (=)).
The first term is the standard cross-entropy ensuring Mo predicts labels correctly. The second term regularizes Mo against the HU reference MuU, preventing from drifting too far from the HU-informed baseline. The final term compares alignment of Mg and MHс with human distribution H, encouraging M3 to better approximate human uncertainty and thus improve calibration.
5
EXPERIMENTS
5.1 SETUP DETAILS
Datasets and set split. We use two open-sourced VQA datasets with human confidence annotations:
VQAv2 (Goyal al. 2017) and VizWiz (Gurari et al. 2018 VQAv2 contains 443,757 training and 213,954
6

---

validation samples based on MSCOCO images Lin al. 2014), and is widely used benchmark. VizWiz has 20,523 training and 4,319 validation samples, collected from blind users via mobile photos (e.g., asking for expiration dates). Unlike VQAv2, VizWiz poses greater challenges due to blurred or unconventional images, though entirely unanswerable ones are filtered out (details in Appendix C.T). Test annotations for both datasets are unavailable and excluded from our experiments. All results are reported on validation sets.
Models and Baselines. We evaluate the latest SOTA VLMs: S0we2.5VVL-22B. (Bai et al., 2025 LLaVA 1.6-7B (Liu et al.] (024b),
(Wang et al. 2024b) alongside the task specific SOTA BEiT3 (Wang et al.) 2023). We mainly study the larger models and smaller models performances are reported in Tab. This selection covers both zero-shot VLMs and the task-specific model. For baselines, we consider: (1) VLMs' zero-shot outputs; (2) SFT with LoRA (Hu et al. 2022]; (3) Meta Pseudo-labeling (Meta PL) (Pham et al., [021]] [Mitra & Soundararaj an (024). while excluding weaker semi-supervised learning methods like Maxch (Sohn et | 2020) and UDA (Xie et al.) 2020 (4) Active Learning (AL) for VQA
with least-confidence sampling (Karamcheti er al. 2021); (5) DPO training for VQA (Sun et al. (024);; and (6) selective prediction LYP (Dancette et al. 2023). We reasonably do not consider more baselines since to the best of our knowledge, they already span recent VQA training strategies, ensuring fair and extensive comparison. More explanations and baseline setup details are in Appendix C.T
Evaluation Metrics. Traditional VQA evaluation only adopts VQA-accuracy metric. In this work, we assess (i) accuracy via HU-acc (Sec. 4 and (ii) confidence alignment via KL-divergence (Sec. 3). We exclude EnCCE (Guo et al.) 2017). as prior work (Lan et al. 2025) shows that it is less effective than KL for VQA.
Reproducibility and Implementation Details. We use open-sourced code bases, model checkpoints, and datasets. We provide their information together with all the implementation details, including training in Appendix C.2. Therefore, we emphasize our method and results are highly reproducible.
5.2 RESULTS AND DISCUSSION
Comparison with baselines. Figure compares HaDola with baselines the selected metrics and datasets. On accuracy, we observe that for the three zero-shot capable VLMs, HaDola achieves the best performance: with only 5% of labeled data, it even surpasses SFT with 100% annotations. This demonstrates that HU-based supervision outperforms the traditional quantity-basee strategy. On BEiT3, which lacks inherent zero-shot ability and heavily relies large-scale supervision, HaDola does not outperform SFT, but still reaches a comparable level while substantially outperforming all other baselines. On KL divergence, both HaDola and LYP consistently outperform the remaining methods. Between the two, when LYP performs better, HaDola still achieves comparable performance while surpassing all other baselines. This result suggests that leveraging fewer but more informative samples leads to better calibration and thus more reliable models. Compared with LYP, while LYP attains lower divergence by discarding samples it deems hard to answer, HaDola achieves competitive calibration without explicitly removing data, making it more broadly applicable.
HaDola Training Rounds. Figure 4 we present the training dynamies of HaDola on VQAv2 and VizWiz. A clear S-shaped curve emerges across all backbones: models exhibit rapid performance improvement when only 5-10% of the data is labeled, followed by moderate gain between 10-15% and eventually plateau after 15%. This finding highlights two important insights. First, substantial gains can be achieved with very limited supervision, showing that HaDola is highly label-efficient. Second, further annotation beyond 15% yields diminishing returns, suggesting that conventional large-scale supervised fine-tuning may lead to unnecessary annotation costs. Together, these results demonstrate that HaDola not only improves absolute performance, but also provides principled way reduce the reliance on costly laheled data.
Ablation Study. Table I presents the ablation results on Qwen2.5VL-7B and consistent performance degradation across both backbones once any component of HaDola is removed or
We observe
7

---

HU-acc VQAv2
KL on VQAv2
InDola (ours)
18.79
18:13
11.93
FaCdl jouns)
435
0.4
682
DPO
14.76
13.09
73.30
DPQ
6:98
0.00
0.7
0.70
NrtsPt
$2.40
68:43
$0.30
Netw
1.84
681
680
08
Wnlls
68.30
72.69
71.83
A81
Ven 1/
6.82
4.40
0.5
350
0wam2.358.22
ULNA 6878
Intemil 3:08
VizWiz
VizWiz
HaDola (curs)
6.72
03.35
FaCole Jours.
L33
150
20
D0
11.47
00.28
U77
1.58
133
148
152
63:77
ALConl
64.33
05.02
03:43
AL-Conl
146
147
2.0
Netart.
NeterL
2.75
SFT
64:93
14:41
03.38
SFT-
184
1444
2.11
14
HANA 6:78
bownitt 2.88
Figure 3: Heatmap comparison of different training methods across four backbone models VQAv2 and WizWiz, under HU-acc and KL. For HU-acc, darker colors indicate higher accuracy, while for KL, lighter colors indicate smaller divergence. Red boxes highlight the best- best-performing method for each model. Our method (HaDola) consistently achieves competitive or superior performance across both datasets.
Habola Training Trend vgav?
HaDnla Training Trend on Visuil
20
Qench
15.3
FSEVAL
123
203
1abeled
Dath
Figure 4: Training dynamics of HaDola. We observe an S-shaped improvement: a rapid increase with 5-10% labels, slower gain between 10-15%, and convergence after 15%.
replaced, validating the necessity of our design. Replacing the Selectors with random sampling substantially reduces accuracy, confirming that HaDola effectively identifies helpful samples. Substituting Self-Annotate with human labels leads the smallest drop, yet still demonstrates that the model benefits from iteratively self-refining supervision signals beyond human annotations. Removing the Error Trigger causes the largest decline, showing that unchecked erroncous labels accumulate and thus harm training. Finally, replacing our tailored loss with standard cross-entropy prevents effective alignment with human preference, yielding suboptimal results. These results highlight that each component plays a complementary and indispensable role in HaDola.
when applying simple SFT on the four models, we consistently observe monotonic trend across both metries. Models trained the low-HU subset (L) achieve the best performance, followed by the medium-HH subset (M), while the high-HU subset (H) yields the worst results. This ordering (L M > H) holds not only training subsets but also validation subsets,
Effects from different HU degrees. As shown Figure 5
8

---

Method (T=15)
VQw2 (HU-Acc)
VizWiz (HU-Acc)
Qwen
LLaVA
Qwen
LLaVA
HaDola (full)
76.75
77.63
65.72
66.58
Replace Selectors with Random Samples
72.23
73.51
62.11
63.02
Replace Self-Annolate with Hurnan Lahels
73.91
75.02
64.53
64.88
w/o Emor Trigger
71.56
72.47
60,92
61.73
Replace Our Loss with CE
72.37
73.28
61.89
62.15
Table 1: Ablation study on HU-Acc across VQAv2, VizWiz and Qwen2.5VL-7B, LLaVAL66-7B We use Qwen and LLaVA for short in the table. Performance consistently drops when replacing or removing each component, showing the necessity of our design.
(Hiiver)
Bem Mijeec
Oweed ML. IEL
Lod 28 DELI
HitrrE H8L
2013 OEL
Train-L
Train-M
Train-H
Figure 5: Radar charts of SFT performances across VQAv2 different training and validation subsets with varying HU levels. The upper shows HU-acc (higher is better), and the lower row shows KI divergence (lower is better) for five models. The three training settings are distinguished by line styles and colors: Train/Val-L, -M, -H means Training or Validation Low, Medium, or High HU subsets.
where lower HU leads to higher accuracy and lower KL divergence. These results suggest that samples with higher HU contribute less to learning, and high-HU is indeed challenge for VLMs. More results and analysis are in C.3 due to page limitations.
6 LIMITATION AND CONCLUSION
This work demonstrates that human uncertainty (HU) is not merely a by-product of annotation but valuable training signal. By systematically analyzing HU, we show that high-HU samples are detrimental, while low- and medium-HU subsets drive both accuracy and calibration. Building on this insight, we propose HaDola, a generalizable framework that integrates HU-aware data across multiple VLMs and two benchmarks. HaDola achieves superior performance with drastically reduced annotation needs. More broadly, highlights a new direction for VQA training: rather than indiscriminately scaling labeled datasets, we advocate leveraging human uncertainty more efficient, better calibrated, and ultimately more human-aligned vasion-language models. The main limitation is that HaDola relies well-constructed labeled seed set. Although such seed set is available
study
informative signal to guide both data selection and evaluation, enabling
9

---

with limited cost, studying how use VLMs' zero-shot ability further reduce the reliance on labeled data will be worthwhile future direction.
Ethics statement We anticipate no ethical concerns with this work. We utilized open-sourced datasets and models, which have been cited.
7
ACKNOWLEDGMENTS
Yihong Liu and Hinrich Schütze were supported by the German Research Foundation (DFG, grant SCHU 2246/14-1).
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 1892-1915, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 eenppeeeenlp-eeommm caalpp-main.hhm 124. URL /ttptsss//aaaantoooooy ort.//r////// 2022 emn p ma i n. 124
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https: http://rxivv.oroo
abs/2502. 13923
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vasion-language pre-training with mixture-of-moddaly---xxp Advances in Neural Information Processing Systems, 35:32897-329122 2022.
Haokun Chen, Yueqi Zhang, Yuan Bi, Yao Zhang, Tong Liu, Jinhe Bi, Jian Lan, Jindong Gu, Claudia Grosser, Denis Krompass, Nassir Navab, and Volker Tresp. Does machine unlearning truly remove knowledge?, 2025. https:t::///xxx org. abs 22052/5552222 23270
Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, and Marcus Rohrbach. Improving selective visual question answering by learning from your peers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
pp.
Julian Eisenschlos, Hernán Maina, Guido Ivetta, and Luciana Benotti. Selectively answering visual questions. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 4219-4229, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 18653/v 1/2024. ac1.250. URL hatps: /tll/attss//aaaatttatt http://caaaaatoooocco org/2024 Findings acl 22222//
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6904-69919 2017.
2017.
10

---

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. CVPR, 2018.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Chengyue Huang, Brisa Mancechotesuwan, Shivang Chopra, and Zsolt Kira. Frames-vqa: Benchmarking fine-tuning robustness across multi-modal shifts in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3909-3918, June 2025.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher Manning. Mind your outliers! investigat- ing the negative impact of outliers on active learning for visual question answering. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 7265-7281, Online, August 2021, Association for Computational Linguis- / wwwp::ooooo2222.1111ooooooo
tics. doi: 10. 10.18663//1/2021.accl 564/
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
Jian Lan, Runfeng Shi, Ye Cao, and Jiancheng Lv. Knowledge graph- based conversational recommender system in travel. In 2022 International Joint Conference Neural Networks (IJCNN), pp. 1-8, 2022. doi: 10.1 1111000CNN500044222222 99922176.
Jian Lan, Mingfeng Xue, and Jiancheng Lv. Rulepg: Syntactic rule-enhanced paraphrase generation. In 2023 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2023. doi: 10. 109/1/544 2023.10191854
Jian Lan, Diego Frassinelli, and Barbara Plank. Mind the uncertainty in human disagreement: Evaluating discrepancies between model predictions and human responses in vqa. Proceedings of the AAAI Conference on Artificial Intelligence, 39(4)4446 4454, Apr. 2025. doi: 10. 111.6//aaa....9....33....
URL https
7/ojs. aaaiaaaii..m flojs.aaai.oro org/ ndee php/ AAAT AwA./atticllaaaeeaaaaaaaaaiaiita
ew 324 68
Jian Lan, Udo Schlegel, Tanveer Hannan, Gengyuan Zhang, Haokun Chen, and Thomas Seidl. Unveiling the "fairness seesaw": Discovering and mitigating gender and race bias vision-language models, 2026. URL
https: wrw.a.aiiv....... http://axxxi.....aa../.. abap////b////a 2505. 23798
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, and Luo Si. mPLUG: Effective and efficient languaee learning by cross-modal skip-connections. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 7241-7259, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 11233//11222222mmmppmm UBLL:/ppss/aallattoooooooogooogp wwtp:222222mmmppmamm...... emnlp-main 488
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pp. 740-755. Springer, 2014.
11

---

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 16296-263066 June 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oer, and world knowledge, January 2024b. URL.tttpp//llaaaall...g.
io/blog 2024 01-30 30 11 1lava next/
David Lowell, Zachary C. Lipton, and Byron C. Wallace. Practical obstacles to deploying active learning, In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 21-30, Hong Kong, China, November 2019. Association org
for Computational Linguisties. doi: 10. 10.18653/11/D19. 1003. /tL/sttss////caatttot///
D19 1 10037
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning. Advances in Neural Information Processing Systems, 33: 7308 73200, 2020.
Shankhanil Mitra and Rajiv Soundararajan. Knowledge guided semi supervspe learning for quality assess- ment of user generated videos. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 4251 1251-426 1-4200,22220 2024.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V. Le. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11557-11568, June 2021.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.),
Advances in Neural Information Processing Systems, volume 33, pp. 19920 19920-19930. Curran Associates, Inc., 2020. URL.tttss///rrreeddaaag. neuriis.eerppppp ec/ paper files pttpp/aappppppeeeepp pdd
e 35355399ec3994225333343222288883
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36. pp. 13728-537411 Curran Associates, Inc., 2023. URL https: / hattes:/prroeeeings..uusu cc/paprr files /ptpp//pee////1ee///
a
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Guohao Sun, Can Qin, Huazhu Fu, Linwei Wang, and Zhiqiang Tao. Self-training large language and vision assistant for medical question answering. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (cds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20052-20061 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10118633/1/22224.mmmp... 1119. URL UR..htts:://celaattlloooo..
wtt.::22...mm1mmmmmmmmmm.m emnlp-main
11197
12

---

Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5100-5111, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/11DD1--5555
1514
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang. Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception the world at any resolution, 2024a. URL httptt/tt///arttt
org/ abs 2409. 12191
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint wxiv:2411. 10444, 2024b.
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as foreign language: Beit pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and 2023.
Patter Recognition, pp.
Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly, In European Conference on Computer Vision, pp. 148-166. Springer, 2022.
Tobias Jan Wieczorek, Nathalie Daun, Mohammad Emtiyaz Khan, and Mareus Rohrbach, Variational visual question answering, 2025. TRLLhttps:/aaxii..rrr
09591
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems, 33:6256-6268 2020.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https: http:::///peeeevie....oo wtwp://wwwww...eeeouu...u..aa
gjeOKFxFp2)
Mingfeng Xue, Dayiheng Liu, Wenqiang Lei, Jie Fu, Jian Lan, Mei Li, Baosong Yang, Jun Xie, Yidan Zhang, Dezhong Peng, and Jiancheng Lv. Unifying discrete and continuous representations for unsupervised paraphrase generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13805-13822, Singapore, De- cember 2023. Association for Computational Linguistics. doi: 12016655111002233emmmmmmaa https: ac http:://cclaanhhoooooooooooe wrg. www.::mammmpmmmmmmmmmmmm 852/
URL
Yongjin Yang, Haneul Yoo, and Hwaran Lec. Maqa: Evaluating uncertainty quantification in llms regarding data uncertainty, 2024. URL httpttps//aaaxivoooo///////////
06816
Gengyuan Zhang, Tanveer Hannan, Hermine Kleiner, Beste Aydemir, Xinyu Xie, Jian Lan, Thomas Seidl, Volker Tresp, and Jindong Gu. Avila: Asynchronous vision-language agent for streaming multimodal data interaction, 2025, URL htllhttts://aat http://awwwwv..oooa//////a// 2bt/ 18472
13

---

Technical Appendices and Supplementary Material
A DIFFERENT HU DEGREES IMPACTS ON VIZWIZ
Similar to Figure 2 (b), we provide comparison of HU-acc and VQA-ace VizWiz in Figure In general, model performances drops on VizWiz as this a more challenging dataset, yet we still see similar trend VQA's: HU-acc provides more discriminative view acToss subsets: models trained on low-, medium-, and high-HU data exhibit a monotonic drop in accuracy as HU grows, indicating that high-HU samples undermine the benefits of SFT on the other two groups. When combining low- and medium-HU data, performance further increases, whereas incorporating high-HU data once again degrades training, confirming their limited utility. In contrast, VQA-acc reflects only small variations and fails to capture the detrimental impact of High-HU samples. This divergence underscores the limitations of frequency-based evaluation and highlights the necessity of HU-aware metries.
VQA-acc
80
70
Accuracy
03
65
80
30
1-H0
14-00
H-HO
L-M-NU
L/80
M-503
B-H0
L-98-100
L-H
Annotation Upper
Figure 6: Results on VizWiz: Effects of training samples with different HU degrees, and comparison with VQA-acc. The L. M, H stands for low, medium, and high.
B HADOLA SETUP
B.1 HADOLA PIPELINE FIGURE
Figure 7 provides clear illustration of HaDola pipeline. Details are introduced in 4.2
B.2 MORE THEORETICAL DERIVATIONS
Setup and notation. Let X' be the input space and A the probability simplex over answers. For a sample E X, let (s) E A denote the human distribution (HaConf-based), Ma(s) A the current model distribution, and MHu(s) the HU reference model. Define the supervised risk and calibration gap
R(0)= E( (CE Mo
C(#) = Eg DKa (x) Mo(x))]
The HaDola objective is
LIaDOLa (0) R(() + (( Mo Mav) A(C(0) C(Onu)), where (MMa || MHV) E, [DK((Muufe Mx))) and θHU are the parameters of MaU-
(9)
14

---

Minit
Muu
seed sel
1909
Only for start
So
2
SFT
200
S
Unlabeled set
Discriminate
Labeling
Sampling
Mc-1
M
PJB
SFT
Terminate if 1-T
Model-labeled seed
Error Trigger
Figure 7: Illustration of HaDola pipelines. Technical details are presented in Section
Assumptions.
A1 (Smoothness) For all æ, the map Mo(x)) is L-Lipschitz and p-smooth in parameters.
A2 (Bounded logits) || log l B uniformly over iterates.
A3 (Seed representativeness) The seed Sp is i.i.d. as S,; its low/medium HU subsets yield empirical estimates (Ti, T2, hw) with sub-Gaussian concentration (Eq. 3 in the main text).
Stage I (Discriminate): HU window controls distribution shift. Let kl, (Ku Mc(u)) for u E S...
HaDola retains samples with kl, [T o, T2 + o).
Lemma (Pinsker window). For any retained 2,,
Me() hw S DKI Mc(w)) & (2
Implication. Discrimination confines training total Variation distance-ball around h excluding high- HU/outlier points. In other words, retained samples are guaranteed to stay within bounded neighborhood of the human anchor distribution hws so the model only learns from data that is sufficiently consistent with human confidence, while discarding samples that are too uncertain anomalous to provide reliable supervision.
Stage II (Self-annotate): pseudo-label bias bound. Let Da ~ MM-1(u) and & be CE. Then
Eg., (M)) 0.)] = -(M(-1(w)) + DKL(M,-1(u) My (u)). additionally (M) - hwll S (by Lemma at L 1), then
CE (u), Mo(w)) CE( Ma(w)) < LCB5,
for some LEE depending on B in A2. Thus pseudo-label training is a bounded-bias surrogate of HU supervision within the HU window. Implication. Self-annotation provides bounded-bias surrogate of human
15

---

supervision: as long as samples remain in the HU trust region defined Stage the loss incurred by training on pseudo-labels differs only slightly from that of true HU labels. In other words, even though annotations are generated automatically by the model, they are guaranteed stay close enough human confidence distributions to provide reliable training signals.
Stage III (Error trigger): stability via gradient alignment. Let g(u) V Vol(Mo(u), Bu) and gref B(a.y)ss, (o Mx(x), v)]. If cos(g(u). fret) 2 Tg 0 and TracIn-mini score Stracin (u) 7 < 0, the contribution to validation loss non-increasing (first-order view).
Under A1 and step size 1j s 1/L, updating on samples passing both triggers yields
L S - + O(n²),
for some proportional to E (g(u), V over retained samples.
Implication. The error trigger ensures stability during iterative self-training: pseudo-labeled samples are only retained if their gradient directions are consistent with human-labeled data and their global influence does not degrade validation performance. In other words, this mechanism prevents harmful labels from accumulating, so that each added sample contributes constructively to model refinement rather than introducing noise.
Stage IV (Training): decomposition and effect. From equation 9 C(OHu) is constant; minimizing LHaDola equals minimizing
R(0) + 3 (Ma || MHv) + xC(()).
The last term penalizes misalignment to H, while the middle term constrains drift from the HU reference.
Lemma (Relative-KL improvement). At any stationary point Ó with 3, À O, either Ma - MHU a.e., or C(O) C(OHV) set of non-zero measure.
Implication. The HU-aware loss jointly balances accuracy, stability, and human alignment: cross-entropy drives correct predictions, HU regularization keeps the model anchored to the reference distribution, and the relative KL term explicitly pushes the model to better approximate human uncertainty. In other words, this objective prevents the model from drifting while encouraging calibration beyond the HU reference, yielding VLMs that are not only accurate but also faithfully reflect human confidence distributions.
Proof sketch. Take the inner product of the first-order condition VR + PV + XVC 0 with log (MoM)) and use the Bregman structure DKL-
Putting it together. Combining Lemma (HU-window), the bounded-bin bias surrogate view, Proposition 1 (stability), and Lemma 2 (relative-K) KL improvement) gives:
Under A1-A3, fixed window [T1 or, T2 o). step size 1) L,L, and 3, >>>> 0, the HaDola iterates (0, YT= satisfy
LutDota (0 CumaOLa ( 7 K4 0 B[C(0r)] C(OHu),
and R(Or) S minGsT R(,,)) + O(1/T) with 0r + Σ - 0,. Thus HaDola decreases prediction error and improves human alignment beyond the HU reference while maintaining stability.
B.3 PARAMETERS AND THRESHOLDS
Threshold estimation. Both thresholds Ty and 7. are derived from the low- and medium-HU subsets of the seed set Sg. Formally, let sz(2, yi 02) denote the gradient consistency score of sample (æ,, v) and Stracin (I., y/ 0,) its TracIn-mint score. We define
Tg E ((a))) Ums, ( (=,
T (a,))els Ums, [Stracin (I. y: 8.)]
(10)
16

---

8
X
HU-ace (%)
KI. 4
0.2
0.5
72.38
0.98
0.3
0.7
76.75
0.63
0.5
1.0
73.16
1.05
0.3
0.5
68.43
1.19
0.3
1.0
69.22
1.21
Table 2: Sensitivity of HaDola to different (8, weights. Results are reported on VQAv2 validation set with Qwen2.5VL-7B. We observe only minor variation across settings, demonstrating robustness.
A pseudo-labeled sample (u, Pn.) is retained only if
sy Dui 02) 2 Tg Suacin (u, Tui 02) S Tt-
(11)
In other words, T ensures sufficient gradient alignment with human-labeled data, while 7 enforces that the pseudo-labeled sample does not harm validation performance.
Weighting strategy for LHaDola
Recall that objective is
LaaDola
E(CE(y, Mo)] +B (MoollMHe) +A (DKL,(HIMo))
DKI. -
accuracy
HU-regularization
human-alignment
Normalization. To ensure that the three terms are comparable in scale, we compute their batch means initialization (t = 0):
Ao ECE])
Ro = E[0],
Co = (HDL.(HIMM) DKL. ( (H]))
We then normalize
Bo
An
do Col-
Ao
Rote
with E~~ 10 to avoid division by zero.
(ii) Grid tuning, Around the normalized values, search over small grid
B E (0.33o. fo, 330), X E (0.3Ao, λo 3λO): selecting the best configuration by HU-ace (primary) and KI. divergence (secondary).
(iii) Default values. When normalization is applied, robust defaults are 3 [0.1, (e.g., 0.3) and A E [0.1, 2] (c.g., 0.5-1.0). We monitor HU-acc and KL during training: if accuracy rises but KL stagnates, we increase if KL improves but accuracy drops, we reduce À or strengthen 3.
As shown in Table 2 varying (3, A) within a reasonable range has little effect on both HU-acc and KL
divergence (changes 0.3%). This demonstrates that HaDola robust to the choice of weighting parameters. We therefore fix (3, A) = (0.3, 0.7) for all experiments to ensure fairesss and reproducibility.
B.4 COMPUTATIONAL EFFICIENCY.
HaDola introduces no additional training overhead compared standard SFT. All components (discrimination, self-annotation, error trigger, and loss) reuse existing forward and backward gradient. Unlike standard SFT, which repeatedly trains on 100% of annotated data each epoch, HaDola only uses 5-20% of the data per round (starting from 5% seed and incrementally adding 1% per iteration). This drastically reduces the effective training load while achieving superior accuracy and calibration, so our framework incurs lower computational cost than SFT. A detailed quantification of computational cost is beyond the scope of this work, since our focus is on the effect of human uncertainty on training and calibration, rather than runtime profiling. Thus, method achieves better accuracy and calibration without extra computational cost.
17

---

OKVQA
GQA
Model
Vanilla
HaDola (trained on VQAv2)
Vanilla
HaDola (trained on VQAv2)
Qwen2.5VL-2B
62.6
64.5
72.1
79.8
LLaVAI.6-7B
61.2
63.7
68.4
75.1
InterVL.2.5-2B
59.8
61.0
73.2 66.5
BEiT3
2.4
59.1
59.1
61.5
Table 3: Performance comparison of HaDola and baselines on OKVQA and GQA datasets on HU-Ace.
C MORE EXPERIMENTAL DETAILS AND RESULTS
C.I DATASETS AND BASELINES SETUP
Datasets. We select VQA22 and VizWiz as our main datasets because they are, to the best of our knowl- edge, the only open-source VQA datasets that explicitly provide human confidence annotations, which are indispensable for studying HU, Other widely used datasets such as OKVQA (Marino et al., 2019) or GQA (Hudson & Manning 2019) do not include HU information, and thus cannot directly support our investigation. Nevertheless, to further test the generality of HaDola, we also evaluate models trained on VQAv2 in a cross-domain setting, and observe that HaDola retains strong performance when transferred to other VQA-style datasets, demonstrating its domain generalization ability. Table demoonsttttss that HaDola consistently surpasses all baselines across both datasets and evaluation metrics. In particular, suggests that the benefits of HaDola are not restricted to single benchmark but generalize across distinct domains. This generalization ability highlights that our design choices especially the integration of HU-aware training signals enable the model transfer effectively across datasets with different challenges, thereby ensuring robustness and broader applicability.
Baselines. We compare HaDola with a broad set of representative training strategies, including supervised learning, semi-supervised learning, active learning, reinforcement learning learning-based data augmentation, and selective prediction. To the best of our knowledge, these baselines cover all major VQA training paradigms that allow fair comparison with our setting. We therefore reasonably do not consider additional models, our selection already provides a sufficiently comprehensive and fair evaluation.
C.2 REPRODUCIBILITY AND IMPLEMENTATION DETAILS.
Our work is highly reproducible. All code, pre-processing scripts, and detailed hyperparameters will be released after the anonymous period.
Details include: (i) model architectures and checkpoints are from the following open-sourced code bases: Qwen2.5 5VL LLaVA 1.6 Intern 3 and BEiT3 [ (ii) dataset splits are from the official website:
VQAv2 VizWiz (iii) trainnn hyperparameters (learning rates, batch sizes, optimizers, number of rounds T, etc, in Tab.
https://ggthhh..cmmmmmaiiaaaaiii
wttp://wwiiiiill.l/ooo.oo/o./o/a hanp://gthhb.comarrcnelllllll
"htpss//visualqaoorgg
18

---

Parameter
Value
Hardware
8
A100 80G (single node)
Precision
BF16
Optimizer
Adam W (weight decay: 0.01 or 0)
Learning rate (LLM)
1 x 10
Learning rate (Vision Projector)
x x
Batch size (effective)
48-64
Gradient accumulation
(when batch per device < global batch)
Epochs
3 for VLMs, 10 for BEiT3
Warmup ratio
0.03-0.1
Scheduler
Cosine
Max text length
2048-4096
Gradient checkpointing
Enabled Rank 16, a=32
LoRA
Table Core training configurations used in our experiments.
how there?
Q: who offense?
Q: what color is the surfboers?
Q: how much is the shrimp plate? Human: 6.75
Human:: 5
H. Humans person bottom
Humaan: rid and blue
SFT:
AL: woman
MetaPL: blue
Vanill:: 6
HaDoln: 5
HaDoie: bottom parson
Haibela: red and blue
HaDola: 8.75
HaDola wio our loos: 4
Hatola wio self arnotate: man
HalDola wio enrar trigger: blue
Hatola wil selector: B15
Figure 8: HaDola's generation (also as self-label) together with case study.
C.3 MORE EXPERIMENTAL RESULTS AND ANALYSIS
Self-annotated label Analysis & Case Study. To disentangle the effect of each design choice, conduct an ablation study on HaDola. HaDola's outputs serve as both answers (final round) and self-annotated labels for middle rounds. As shown in Figure g compared with baselines, HaDola consistently yields correct and numan-aligned results. Removing any component leads to clear degradations: the model drifts away from human preferences while the self-annotated iterative supervision weakens, where obvious mistakes left uncorrected. These results confirm that each design choice is indispensable for HaDola's superiority.
Effects from different HU degrees. Consistent with our earlier findings on VQAv2, we again observe under simple SFT clear monotonic trend across both accuracy and KI. divergence all four models. Training on HOw-HU subsets (I.) yields the strongest performance, medium- -HU subsets (M) perform worse, and high-HU subsets (H) degrade performance the most. This ordering (L > M H) appears consistently on both training and validation subsets, indicating that higher HU samples provide less useful supervision and that high-HU remains fundamental challenge for VLMs.
Size selection of seed set.
To further analyze the effect of seed set size, vary it from 1% to 10% and report Hu-acc results in Table 5 As shown in the table, all models on both datasets exhibit rapid gains when the seed set increases from 1% 3%, followed by slower improvements and eventual convergence after round 5. This indicates that the majority performance benefits are already captured in the early stages, while adding more labeled data beyond 5% yields only marginal gains. Therefore, in the remainder of this work choose 5% as the default
19

---

Train-L
Train-V
Train-Ff
Figure 9: Radar charts of SFT performances across VizWiz different training and validation subsets with varying HU levels. The upper row shows HU-acc (higher is better) and the lower row shows KL divergence (lower is better) for five models. The three training settings are distinguished by line styles and colors: Train/Val-L, -M, -H means Training or Validation on Low, Medium, or High HU subsets.
Base Model
Dataset
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
Qwen2.5VL-7B
VQAv2
0.62
0.68
0.71
0.73
0.77
0.77
0.77
0.77
0.78
0.78
VizWiz
0.55
0.60
0.63
0.65
0.66
0.66
0.66
0.66
0.67
0.67
LLaVA1.6-7B
VQAv2
0.60
0.66
0.70
0.73
0.75
0.76
0.76
0.76
0.77
0.77
VizWiz
0.54
0.59
0.62
0.64
0.67
0.67
0.67
0.67
0.67
0.68
Intern IntemV VL2.5-8B
VQAv2
0.58
0.64
0,68
0.71
0.72
0.73
0.73
0.73
0.74
0.74
VizWiz
0.52
0.57
0.61
0.63
0.64
0.64
0.64
0.64
0.65
0.65
BEiT3
VQAv2
0.45
0.58
0.64
0.68
0.72
0.72
0.72
0.73
0.73
0.73
VizWiz
0.39
0.45
0.53
0.60
0.61
0.61
0.61
0.61
0.62
0.62
Table 5:
HaDola top performance (HU-acc) across seed set sizes (1-10%) on VQAv2 and VizWiz. We observe steep gains during the first three rounds, rapid improvement up to round 5, and convergence afterwards.
seed set size, as it strikes favorable balance between model performance and annotation cost. Moreover, a 5% budget is also more realistic and practical in real-world applications. where annotation budgets are constrained in both industry and research settings, and stands also for text-based tasks Chen et al. (025
Xue al. (2023): Lan al. 2203 2022).
Comparison with traditional Calibration
Finally, we briefly compare HaDola with traditional calibration methods, in particular Temperature Scaling (TS). We emphasize that this work is not a calibration paper and HaDola is not designed solely for calibration: rather, our focus on leveraging HU for data selection and training. Nevertheless,
informative to analyze HaDola's improvements in calibration relative to TS. We also note that several recent works have studied calibration (Wieczorek et al. [2225] [isenschlos et al.) [024| Lan et al. 2025).. However, these studies primarily address reliability and human distribution alignment without evaluating model accuracy, and thus do not provide a fair comparison to our setting. Among them, Lan et al. (0025) specifically evaluates TS for VQA. while the other two works have not yet released code, preventing a direct implementation for comparison.
As shown in Table HaDola consistently achieves the lowest KL divergence across all four base models and both datasets, demonstrating clear advantages calibration. In contrast, Temperature Scaling (TS, T=1.2) only partially reduces KL: while it slightly improves calibration on VQAv2 (e.g., Qwen2.5VL-7B
20

---

VQAv2
VizWiz
Base Model
Vanilla
TS
HaDola
Vanilla
TS
HaDola
Qwen2.5VL-7B
0.82
0.67
0.64
1.75
2.45
1.59
LLaVA1.6-7B
0.48
0.41
0.35
1.35
2.52
1.28
0.79
0.77
0.74
1.84
2.58
1.33
ВЕÎТ3
9.58
2.221
0.62
10.46
2.80
1.58
Table 6: KL divergence (lower is better) for Vanilla, Temperature Scaling (TS, T'=1.2), and HaDola across two datasets.
and LLaVAI.6-7B) it even degrades performance VizWiz, where KL values increase compared to the vanilla baseline. This highlights the instability post-hoc calibration under domain shift or noisy annotations. In all cases, HaDola substantially outperforms TS, showing stable and large improvements. The effect is particularly striking on BEiT3, where the vanilla model exhibits extremely poor calibration (KL over 9.5 and 10.4), TS reduces the gap but still remains high, while HaDola lowers KL to 0.62 and 1.58, reaching the same level as recent VLMs. These results confirm that HaDola not only surpasses post-hoc calibration but also achieves robust and reliable calibration improvements across diverse models and datasets. This suggests that explicitly leveraging HU during training is more effective than applying post-hoc calibration methods such as TS.
21


Reviews:
[{"review_id": "rLO013TuQx", "reviewer": "Reviewer_Uh4i", "summary": "The paper introduces HaDola, a framework that improves the training of vision-language models (VLMs) by explicitly accounting for human uncertainty (HU) in annotations. The authors find that samples with high HU—where annotators disagree or are uncertain—often degrade model performance and calibration. HaDola mitigates this by iteratively identifying and excluding harmful samples, generating pseudo-labels for reliable ones, and applying an error-trigger mechanism to prevent noise accumulation during self-training. Using only a small seed set (around 5%) of HU-labeled data, HaDola achieves comparable or superior accuracy and calibration to state-of-the-art baselines on VQAv2 and VizWiz datasets. The work highlights that effectively modeling human uncertainty is more beneficial than simply scaling the dataset size.", "questions": "Please refer to the questions in the weakness section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors propose an interesting approach to leverage human uncertainty for both model training and evaluation.", "weaknesses": "- The compared baselines seem weak. The target task appears to be a semi-supervised VQA task. Then, it would be beneficial to compare with other pseudo-labeling-based semi-supervised learning works, such as FixMatch.\n- In Figure 4, the performance curves with different VLMs are not enough. In a data-efficient VQA work, the authors should have compared against baselines, as this curve could be the main experiment of the paper. \n- Also, as the initial annotation can critically change the model performance, the authors should have considered different seed data settings in the experiment.\n- The novelty of the proposed HaDola seems weak. The proposed HaDola framework is a combination of existing ideas (data filtering, pseudo-labeling, uncertainty calibration) with minor adaptation for human uncertainty. Simply removing uncertain samples is too naïve and risks losing informative and diverse examples. It would be better to include more analysis of the justification for the design choices.\n- In addition, there are various design choices in the method without empirical justification. For example, while the authors select the samples in the region [tau1-sigma,tau2+sigma], it would be better to empirically justify the design choice by sampling different samples.\n- The evaluation is only done with the HU accuracy. However, it would also be useful to evaluate with the conventional VQA accuracy. Comparing the trends between VQA accuracy and HU accuracy might provide informative insights.\n- For Figure 3, it might be better to use a table instead of a heatmap for better visualization. In the heatmap, the baselines' performances look similar. However, when we look at the individual values, there are several settings where the baselines perform better than the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "REwveE0rVU", "reviewer": "Reviewer_nUwP", "summary": "The authors propose a new method to leverage a small pool of human-labeled VQA data along with annotators' confidence scores to bootstrap a larger pool of self-annotated VQA data. This approach outperforms naive SFT at a fraction of the data scale. The authors experiment with their HuDola method on both VQAv2 and VizWiz datasets, demonstrating that HuDola reliably improves performance across multiple model families.", "questions": "1) How does the HuDola method generalize to out-of-distribution data from the source seed data pool?\n2) Are the results of the training runs statistically significant? The authors conduct only a single run for each training method and model family, and the difference in performance between HuDola and SFT is very small for some experiments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, the paper is well-written and the experiments are well-designed. The authors provide empirical evidence that high human uncertainty is detrimental to SFT performance. The experimental scope is satisfactory, covering four different VLM base models with consistent improvements across all models (with the exception of BEiT-3). The authors provide a thorough analysis of the HuDola results. The study on training dynamics shows that performance improves smoothly with additional HuDola-generated samples, and the ablations presented in Table 1 demonstrate that each component of HuDola's pipeline is necessary for achieving the improvements.", "weaknesses": "My main concern is that the authors study this problem only in the context of fine-tuning and evaluating the VLM on data from the same distribution and on relatively saturated benchmarks. The findings would be more impactful if the authors could demonstrate that HuDola improves VLMs on out-of-distribution data and on more challenging benchmarks. To assess the out-of-distribution performance of HuDola, one specific experiment would be to initialize the seed data pool from a single source (e.g., VQAv2) and use it to bootstrap labels for a different source (e.g., VizWiz).\n\nAnother limitation is that the authors conduct only a single run for each training method and model family. Since the difference in performance between HuDola and SFT is less than 0.5 percentage points in most cases, it is unclear whether the improvements are statistically significant. There is likely some run-to-run variability in the results, and reporting the mean and standard deviation would help verify whether the improvements are statistically significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pg7IBSqQQx", "reviewer": "Reviewer_FqsW", "summary": "This paper investigates how human uncertainty (HU), namely the variation in annotator confidence across different answers, affects visual question answering model training. The authors observe that standard supervised fine-tuning optimizes only toward majority labels while ignoring confidence distributions, leading to miscalibrated models.\n\nKey contributions: (1) First systematic study of HU's impact on VLM training; (2) HaDola framework achieving competitive results with only 5-20% labeled data; (3) HU-acc metric incorporating confidence into evaluation; (4) Theoretical analysis with bounded-bias guarantees.", "questions": "See Weaknesses part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As the first systematic study of human uncertainty's role in VLM training, this work addresses an important gap in VQA research. Its strengths include convincing empirical findings (high-HU samples harm performance), impressive data efficiency (5% seed data matching 100% baselines), and comprehensive validation across multiple models and datasets with theoretical grounding.", "weaknesses": "1. The paper assumes confidence labels are reliable across annotators without validation. Different annotators may interpret \"maybe\" systematically differently, yet no inter-annotator agreement is analyzed. Since all thresholds (τ₁, τ₂...) depend on the 5% seed set, miscalibrated annotators in the seed would compromise the entire framework.\n\n2. The mapping yes/maybe/no → 0.99/0.5/0.01 is borrowed from prior work without justification.  These values directly affect HUD distributions and split boundaries, yet zero sensitivity analysis is provided. Testing alternative mappings (e.g., 0.95/0.5/0.05) is necessary to show results aren't artifacts of arbitrary choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BV4QSQf2eV", "reviewer": "Reviewer_VZNq", "summary": "The paper studies how human uncertainty affects VQA fine-tuning and finds that high-uncertainty data often hurts learning while low and medium uncertainty help both accuracy and calibration. It proposes HaDola, a four-stage pipeline that selects data with a KL-based HU window, self-annotates remaining samples, filters errors with gradient and influence triggers, and trains with an HU-aware objective.", "questions": "1. How robust are results to different random seeds for the initial 5% labeled subset. \n\n2. What is the measured training and inference overhead per round and per sample on standard hardware.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The work clearly demonstrates that indiscriminate training on high-uncertainty samples degrades both accuracy and calibration. \n\n2. The HaDola pipeline reduces dependence on expensive annotations by iteratively expanding from a small seed set. \n\n3. Ablations and training-round analyses support the necessity of each component and show an S-shaped gain that plateaus near 15% labels.", "weaknesses": "1. The HU-acc metric may face adoption hurdles and its relationship to downstream utility is not deeply analyzed. \n\n2. The HU splitting scheme and threshold choices introduce design degrees of freedom that could affect reproducibility. \n\n3. Claims of low overhead are not backed by wall-clock or memory profiles for discrimination, triggers, and extra objectives. \n\n4. The method depends on a representative 5% seed set, and its sensitivity to seed variance is unclear. \n\n5. Validation is limited to two VQA datasets, so generality to knowledge-heavy or compositional QA remains uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)