AQuA: Toward Strategic Response Generation for Ambiguous Visual Questions
Authors: Jihyoung Jang
Hyounghun Kim
Venue: ICLR 2026 Poster (2026)
Keywords: Visual Question Answering
Vision–Language Models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Visual Question Answering (VQA) is a core task for evaluating the capabilities of Vision–Language Models (VLMs). Existing VQA benchmarks primarily feature clear and unambiguous image–question pairs, whereas real-world scenarios often involve varying degrees of ambiguity that require nuanced reasoning and context-appropriate response strategies. Although recent studies have begun to address ambiguity in VQA, they lack (1) a systematic categorization of ambiguity levels and (2) datasets and models that support strategy-aware responses. In this paper, we introduce Ambiguous Visual Question Answering (AQuA), a fine-grained dataset that classifies ambiguous VQA instances into four levels according to the nature and degree of ambiguity, along with the optimal response strategy for each case. Our evaluation of diverse open-source and proprietary VLMs shows that most models fail to adapt their strategy to the ambiguity type, frequently producing overconfident answers rather than seeking clarification or acknowledging uncertainty. To address this challenge, we fine-tune VLMs on AQuA, enabling them to adaptively choose among multiple response strategies, such as directly answering, inferring intent from contextual cues, listing plausible alternatives, or requesting clarification. VLMs trained on AQuA achieve strategic response generation for ambiguous VQA, demonstrating the ability to recognize ambiguity, manage uncertainty, and respond with context-appropriate strategies, while outperforming both open-source and closed-source baselines.

Published as a conference paper at ICLR 2026
AQUA: TOWARD STRATEGIC RESPONSE GENERA- TION FOR AMBIGUOUS VISUAL QUESTIONS
Jihyoung Jang' Hyounghun Kim
Graduate School of Artificial Intelligence, POSTECH DDepartment of Computer Science and Engineering. POSTECH (jihyoung, h. h.kkim)postech.cc ac.
ABSTRACT
Visual Question Answering (VQA) is core task for evaluating the capabilities of Vision-Language Models (VL.s).. Existing VQA benchmarks primarily fea- ture clear and unambiguous image-question pairs. whereas real-world scenarios often involve varying degrees of ambiguity that require nuanced reasoning and context-appropriate response strategies. Although recent studies have begun address ambiguity in VQA, they lack (1) a systematic categorization of ambi- guity levels and (2) datasets and models that support strategy-aware responses. In this paper, we introduce Ambiguous Visual Question Answering (AQUA), a fine- grained dataset that classifies ambiguous VQA instances into four levels ac- cording to the nature and degree of ambiguity, along with the optimal response strategy for each case. Our evaluation diverse open-source and proprictary VLMs shows that most models fail to adapt their strategy the ambiguity type, frequently producing overconfident answers rather than seeking clarification or acknowledging uncertainty. To address this challenge, we fine-tune VLMs on AQUA, enabling them to adaptively choose among multiple response strategies, such as directly answering, inferring intent from contextual cues, listing plausible alternatives, or requesting clarification. VLMs trained on AQUA achieve strategic response generation for ambiguous VQA, demonstrating the ability to recognize ambiguity, manage uncertainty, and respond with context-appropriate strategies, while outperforming both open-source and closed-source baselines !
INTRODUCTION
Recent advances in VasiunnL Models (VLMs) (Dai et al. 2023 Liu et al. 2023 Chen (ct al. 2024 Bai et al.) 2025) have significantly improved their performance across a broad range of Visual Question Answering (VQA) tasks (Antol et al., 2015 Goyal et al. 2017; Gurari et al. 2018 Singh et al. 2019 Mathew et al., 2021). Traditional VQA benchmarks primarily evaluate whether models provide correct answers to clearly stated, unambiguous questions paired with well-defined images (Johnson et al. 2017 Hudson & Manning 2019 Biten et al. 2019). While such benchmarks are valuable for assessing basic multimodal reasoning. they fail to capture critical challenge in real-world use: the ability to handle ambiguous or unclear queries. This capability remains underexplored.
In human communication, ambiguity is typically resolved through contextual inference or follow-up questions. For example, when asked "What brand is this " an image with multiple cars, one may seek clarification or infer the intended car from context. Previous research in ambiguous VQA has mainly focused on making models always ask clarifying questions when uncertain (Jian) et al. 2025). While this binary answer-or-ask strategy can be useful, it does not reflect real-world conversational dynamics, where clarification is not always the most efficient approach (Chen al.) 2025). Humans instead adapt their strategy to the situation-somec sometimes inferring intent from text, sometimes offering multiple plausible answers when they are few, and requesting clarification only when necessary.
Our code, dataset, and model are publicly available httptts://aaaaiiiili/i

---

Published as a conference paper at ICLR 2026
What color is this bat?
cp
The bat in the foreground (leaning against the wall, farthest to the right) is black with red handle.
There are several baseball bats visible, so "this" could refer to more than one Could you specily which bat you mean perhaps ......
Based on the image, the bats in the foreground are variety of colors, including natural wood,
The bat in the image is red and black.
Ours
GPT, Gemini, Qwen
Figure I: Examples of model responses ambiguous visual question. In this image, none of the bats is visually salient, making the visual context ambiguous. While GPT, Gemini, and Qwen provide answers by arbitrarily selecting (e.g., the bat in the foreground) despite the ambiguity, our model, which is trained to handle such cases strategically, requests clarification instead.
To bridge this gap, we propose Ambiguous Visual Question Answering (AQuA), novel dataset designed to enable VLMs choose contextually appropriate strategies for ambiguous VQA. Our dataset categorizes VQA instances into four fine-grained levels, based on both the nature and degree of ambiguity: (Level 0) unambiguous questions, (Level 1) questions whose intended referent can be inferred from context, (Level 2) questions with multiple plausible answers where listing options is preferable, and (Level 3) highly ambiguous questions requiring explicit clarification. To our best knowledge, AQUA is the first resource enabling systematic training and evaluation of strategy selection across these distinct ambiguity scenarios.
We empirically show that both open-source models (Bii ct al. 2025 Chen et al. 2024 and high- performing closed-source models -GPT--and Gemini 2.5 Flast fail to properly handle ambiguous VQA, often responding overconfidently rather than adapting to the ambiguity (Figure D Building on these findings, we train models on AQUA to enable them to produce strategy-awars responses that adapt the degree of ambiguity. Since generating strategy-adaptive responses is highly chal- lenging for baseline models, we begin with supervised fine-tuning (SFT) to explicitly teach them the space of possible strategies. SFT provides solid foundation for ambiguity it does not directly optimize for strategic choice. To address this, we further apply Group Relative Policy Optimization (GRPO) (Shao et al. 2024), rewarding models when they produce strategy- aligned outputs and thereby improving their ability to adapt to varying degrees of ambiguity. VLMs fine-tuned on AQUA achieve substantially better performance by developing adaptive ambiguity- handling abilities. Our analysis demonstrates not only whether VLMs genuinely understand ambi- guity and respond strategically, but also why such strategy-based responses are effective.
responses, but
Our contributions in this paper are as follows:
1. We propose AQUA, a novel VQA dataset designed to train and evaluate how VLMs handle ambiguity. AQUA organized into four fine-grained levels based on the degree and nature of ambiguity, enabling systematic analysis of response strategies across different ambiguous scenarios.
2. We fine-tune spen-source models on AQUA, and they outperform larger open- and high- performing closed-source models by autonomously selecting contextually appropriate response strategies.
3. Through extensive analysis, we verify why VLMs fail to generate strategic responses, analyze their error patterns, and confirm the effectiveness of strategic responses in handling ambiguity.
RELATED WORKS
Ambiguity in Question Answering. Traditional Question Answering (QA) benchmarks typically focus on unambiguous question-contex pairs with clear answers, which effectively measure mod- els' basic comprehension but fail to assess their ability to handle ambiguity (Rajpurkar et al. 2016
Joshi et al.] 2017 Kwiatkowski et al.) 2019). In text-batt based QA, ambiguity has been extensively studied (Min et al.
2020; Stelmakh et al. 2022 Kim al.) 2023 Lee et al. [003 Li et al. 20256), whereas visual QA research has only recently begun addressing this gap. For example, Focus Am-
biguity (Chen et al.
2025) analyzes the responses of GPT-4o and Intern VL.2 to ambiguous ques-
tions, revealing that models often generate answers that appear plausible but lack semantic adequacy.
https::/deepmindd.
2

---

Published as a conference paper at ICLR 2026
Q: What toppings on A: The "this" points main item, the hot dog in the bun, and it's topped with mustard and ketchup.
Q: What food baking tray?
A: pizza is the baking tray.
Level (Unambiguous Questions)
Level (Low-Level Referential Ambiguity)
Q: What this player daing right now? A: The refarence might be the Meriners fielder in while at first, or to the rurner in gray with blue helmet and number 49, ......
Q: What shape
furniture?
A: Could you clarify which fumiture you mean? There is a lot of fumiture visible in the room visible in the room, so can't
shape
Level (Multiple Valid Interpretations)
Level 3 (High-Level Ambiguity Requiring Clarification)
Figure 2: Examples of the four ambiguity levels in AQUA.
CIearVQA (Jian et al. 2025) trains LLaVA to ask clarifying questions for ambiguous queries, but adopts rigid binary strategy by always seeking clarification, without adapting to different types degrees of ambiguity, which limits its practicality. duces benchmark specifically designed to evaluate how visual contexts help resolve ambiguous linguistic expressions. To the best of our knowledge, AQUA is the first dataset to provide a fine- grained categorization of ambiguity in VQA, thus enabling systematic evaluation of diverse and context-appropriate response strategies.
contrast, VAGUE (Nam et al. (024) intro-
Uncertainty Handling Strategies. While Large Language Models (LLMs) and VLMs respond with "I don't t know" in uncertain situations, they often show tendency to answer even unanswer- able questions (Guo et al., 2024 Li al. 2025a). Previous research has primarily addressed this problem through binary approaches: training models to respond only when confident and to ab- stain when uncertain (Whitehead et al., 2022 Jian et al. (225).. These methods focus mainly on teaching models when to withhold responses. However, simply abstaining in uncertain situations does not always align with real-world human behavior (Liu al., 2025). Depending on the degree of uncertainty, humans may leverage contextual clues to infer answers (Nam et al.) 2024), provide all possible answers when there are only few viable options, or ask follow-up questions to re- solve ambiguity (Jian et al. 2025). We adopt this perspective in the context of ambiguous VQA, examining how VLMs should respond based on different types and degrees of ambiguity. To our knowledge, this is the first work that enables models to select among multiple response strategies based on specific ambiguous scenarios.
DATASET
In this work, we introduce Ambiguous Visual Question Answering (AQUA), a novel dataset that enables not only comprehensive evaluation but also effective training of VLMs on ambiguity in VQA. Unlike existing datasets that treat ambiguous queries uniform or binary fashion, our dataset systematically categorizes ambiguity into four distinct levels, enabling controlled and fine- grained training and evaluation.
3.1 LEVEL DEFINITIONS
In natural human communication, when confronted with ambiguous questions about visual infor- mation, people do not rely on a single strategy. Instead, they adapt their response according to the situation: asking for clarification when necessary, inferring answers from contextual cues when am- biguity is low, or enumerating all candidates when multiple plausible targets exist. For example, when looking crowded scene and asked, "What is that person wearing?", human might spond "Which person?" if there are several individuals, or directly answer if only one person is prominently visible.
Our goal in designing AQUA is not only to test whether VLMs can request clarification, but also to assess whether they can strategically respond using contextual reasoning when faced with ambiguity. To this end, we construct our VQA dataset with the following four levels (Figure 2):
Level O. Unambiguous Questions: Standard VQA cases with clear, unique answers, such as "What food is on the baking tray?" when there is only one tray with food. This category serves
3

---

Published as a conference paper at ICLR 2026
control group to verify that models can still perform well on conventional VQA without over-applying Sebbiuutt-hhnndling strategies.
Level 1. Low-Level Referential Ambiguity: Questions often involve pronouns like "ir", "this" "that", or "these" " where context makes the referent obvious. For instance, in the exam- ple "What toppings on this?", the term "this" can be resolved from context because the hot dog is the only plausible referent for a topping-related question. Thus, the model is expected to infer that "this" refers to the hot dog and directly provide the corresponding answer, rather than treating the question as ambiguous.
Level 2. Multiple Valid Interpretations: In these cases, offering all reasonable interpretations is preferable while asking for clarification may be unnecessary or inefficient. For example, consider the question "What is this player doing right now?" " in an image where two baseball players are engaged in clearly distinct activities, with one running and the other fielding. At this level, there are only two three plausible interpretations, and mentioning all of them is more efficient than asking for clarification.
Level 3. High-Level Ambiguity Requiring Clarification: Questions that genuinely require clarification due high level of ambiguity. For example, in the question "What shape is this furniture?", the image contains many visually similar objects, including multiple sofas, tables, desks, and lighting fixtures, making it unclear which one the question refers to. In such cases, enumerating all possible candidates would be inefficient, and the most appropriate strategy is to request clarification.
3.2 DATASET GENERATION
We construct our dataset using images from the COCO dataset (Lin et al. 2014) as the visual source. To identify objects and potential sources of ambiguity, we rely the bounding box annotations provided in COCO. These annotations specify the location and category of each object in the scene, enabling systematic identification of potential ambiguity sources. In particular, bounding boxes allow us to quantify both the number and the spatial prominence of objects, providing a principled basis for controlling ambiguity levels.
For Level O, we use randomly sampled images and design unambiguous questions such that the target object is explicitly specified without vague referential terms (c.g., "this", "that", "these"). This guarantees a unique, distraction-free interpretation, corresponding to the zero-ambiguity set- ting. For Level 1, we select images that contain a single salient object. Specifically, Object saliency is determined using structured scoring scheme. For each object, we compute a saliency score as weighted combination of its area ratio and its normalized distance to the image center, with weights of 0.7 and 0.3, respectively. The object with the highest score is regarded date. Based on our empirical analysis, objects with a saliency score greater than 0.6 are considered salient. An image is assigned to Level 1 if exactly one object satisfies this criterion. While other minor objects may be present, their visual prominence is negligible, ensuring that vague referen- tial terms can be resolved unambiguously through context. For Level 2, identify images with small number of salient objects (two to three bounding boxes above the threshold), where multiple plausible answers exist and enumerating alternatives is the most natural strategy. For Level 3, we target complex scenes with larger number of salient objects (five or more bounding boxes, often of similar categories or sizes), where ambiguous questions genuinely require explicit clarification.
the primary candi-
To generate question-anse answer pairs for collected images, we employ GPT-5 with level-specific prompts aligned to the above definitions. This controlled prompting procedure ensures that the linguistic form of the questions and the corresponding answer strategies consistently reflect the in- tended ambiguity level. Please see Appendix K for all prompts used in dataset construction.
3.3 DATASET FILTERING
As AQUA is a synthetic dataset, we apply a rigorous multi-stage filtering process to ensure its quality. For this, we adopt three-stage filtering pipeline: (i) we first verify that each instance satisfies the requirements of its designated ambiguity type: (ii) we then verify if each caadee-question pair better fits different ambiguity level, ensuring that the assigned level is uniquely justified by the visual context; and (iii) we confirm that the image is a real-world photograph and validate both the clarity of the question and the factual correctness of the answer. All three stages evaluated
4

---

Published as a conference paper at ICLR 2026
using GPT-5-mini, and only image question-anae answer triplets that pass all stages retained. Please refer to Appendix K for the dataset filtering prompts.
Through this process, collect 7.2K samples in total: 3.6K for training and 3.6K for evaluation. Each split is evenly balanced across the four ambiguity levels, with 0.9K instances per level. Please Appendix A for additional examples of the AQUA.
To ensure the reliability of the evaluation split, we perform human validation on all samples in this split using Amazon Mechanical Turk (MTurk) For each generated sample, annotators verify triplet conforms to its assigned ambiguity level, providing a
whether the image-question-anwwe binary PASS/FAIL. judgment. Each instance is independently evaluated by two annotators, and only samples that receive a PASS label from both annotators are retained, Further details of the filtering procedure and annotation protocol are also provided Appendix B
judggment
3.4 HUMAN EVALUATION OF STRATEGIC RESPONSE SELECTION
To examine whether the four strategic response levels defined in AQUA align with how humans handle ambiguity in practice, we conduct a human evaluation on Amazon MTTur. Annotators are provided with concise definitions of three response strategies: (I) answering directly when the target is obvious or inferable from context, (2) listing all plausible targets when their number is small, and (3) requesting clarification when the scene is highly ambiguous, corresponding Level 0-1, Level 2, and Level 3, respectively.
For each image-question pair, we collect strat- egy annotations from three independent annota- tors. Final human labels are assigned via major- ity vote, and examples without agreement are treated as disagreements. The evaluation set consists of 200 test examples, randomly sam- pled from the test split with 50 examples from each ambiguity level. Table [T] reports the hu- man evaluation results. Level () and achieve near-perfect agreement, indicating strong align- ment between human judgments and the strate- gic response levels defined in AQUA.
Table I: Evaluation results on human strategic choices.
Level
Agreement
Disagreement
Level 0
50
0
Level 1
48
2
Level 2
18
Level 3
32
18
For Level 2 and 3, agreement remains high overall, although some disagreement arises in highly am- biguous cases, reflecting the inherent subjectivity these scenarios. closer examination identifies two common sources of disagreement. First, in borderline cases involving three or four plausi- ble targets, annotators may differ on whether it is preferable to enumerate all options or to request clarification. Second, disagreements can arise from differing judgments of object saliency, where annotators vary in whether they infer salient target based on visual prominence or focus cues. No- tably, such disagreements results demonstrate that human responses largely align with the strategic response levels defined in AQUA, while highlighting that the boundary between listing multiple possibilities and requesting clarification can subjective in edge cases.
rare in Level I cases, where saliency is clearly defined. Overall, these
4 EXPERIMENTS
We evaluate range of soun-source and closed models on our AQUA to assess their ability to handle ambiguity. In addition, we fine-tune two souresource models to investigate whether VLMs are capable of demonstrating strategic ambiguity-handling.
4.1 MODEL TRAINING
To investigate whether VL.Ms can develop strategic capabilities for handling different types and degrees of ambiguity, we fine-tune Qwen2.5-VL- Bree2.5-VLL-B--nsttrue and IntemVL3-2B-Instrree on AQUA. These models were chosen because (1) they are widely adopted and well-regarded in the research
https: http:://wwwwww..... http://wwwttttt..c//
5

---

Published as a conference paper at ICLR 2026
vahicles visible yallow truck,
Out The main vehide the foreground is yelow truck bold red lettering.
0u: There are four vehidles are wisilee.. So. which one you mean?
multole vahicles
you clarity vahicle you mean?
Model Outputs
R. (0,
12(0g).1)) -1
R.00)==))
color is vehicle?
Judge
Figure 3: Reward assignment process. Since the given image contains multiple vehicles, the correct response is to request clarification. A perfectly accurate clarification receives a reward of 1. If clarification is requested but contains factual error, 0.3 penalty is applied. All other response types are assigned a reward of
community, (2) they perform strongly standard VQA benchmarks, and (3) their parameter sizes offer practical trade-offs between computational efficiency and performance.
Training Strategy. We train all models using two-stage pipeline consisting of supervised fine- tuning (SFT) followed by Group Relative Policy Optimization (GRPO) (Shao et al. 2024). SFT alone does not reliably enforce the correct choice of strategy under different ambiguity levels. To address this limitation, we then apply GRPO, which provides explicit rewards for strategy-aware outputs and thereby strengthens the model's ability to make contextually appropriate decisions.
Reward Design. GRPO is conducted under an LLM-as-a-judge framework, where GPT-5-mini the judge (see Appendix K for prompt). For a generated response y given input (z, I), where x denotes the question and 7 the corresponding image, the reward R(ylæ, I) is defined as (Figure 3):
| 0
if strategy correct but factual distortion detected, if strategy is correct and no distortion, otherwise,
R(yl,,
where À denotes the penalty applied if hallucination or factual inconsistency is detected, and is set to 0.3 in our experiments.
Data Splits. For SFT, we use the training split of AQUA, dividing it into 80% for training and 20% validation, ensuring balanced coverage all four ambiguity levels. For GRPO, we randomly sample 15 instances per level for training and 5 per level for validation from the same split, again maintaining balanced label distribution. Additional optimization details and hyperparameters are provided in Appendix
4.2 EVALUATION METRICS
Our evaluation is performed under an LLM-as-a-judge framework, where GPT-5-mini serves as the judge. To verify the reliability of this automatic evaluation, we sample 400 cases from the test split and compare GPT-5-mini's judgments against an in-house human evaluation, confirming that the automated judgments are highly aligned with human assessment (98.5% agreement). Detailed explanations are provided Appendix D
We report two complementary metrics. facstafactual consistency indicates that the model's response faithful to the content of the given image, even if not all details are included, and is judged in binary manner (Grounded or Ungrounded). Second, strategic accuracy measures whether the response strategy matches the ground-true ambiguity level. If response cannot reliably mapped to any of the four defined levels, it is assigned Unknown label. This metric is computed all responses independent of their factual consistency, since our goal is to evaluate the model's ability to choose the correct strategy rather than remain factually accurate.

---

Published as a conference paper at ICLR 2026
Table 2: Main benchmarking results of various VLMs on AQUA. Unk denotes Unknown.
Model
Factual Ace.
Strategic Acc.
Grounded
Ungrounded
Level
Level
Level
Level
Overall
Unk
Zero-shot
Qwen2.5 Breen.5-VLL-2 -L-tt--nstruet
79.86
20.14
97.11
0.11
33.33
0.78
32.83
104
Qwen2.5- Deon2.5-VVL722
89.33
10.67
99.56
0.56
2.11
0.89
25.78
12
76.63
23.37
96.0
2.33
3.56
1.89
25.95
138
BeemVV.3-78B-Insteee GPT-5
80.5
19.5
96.0
2.11
3.0
5.67
26.7
133
98.4
89.67
0.67
0.33
22.86
178
Gemini 2.5 Flash
91.89
8.11
99.00
5.22
4.44
0.89
27.39
9
Chain-of- Chain-of-Thought (CoT)
Qwen2.5- 111.2.5-V-----nstrree
78.22
21.78
95.89
8.33
5.67
3.78
28.42
Dwen2.5-VL-722. 22s-rust
86.97
13.03
13.78
27.72
76.08
23.92
97.67
2.44
1.33
1.11
25.64
54
79.75
20.25
96.78
5.22
3.67
12.33
29.5
74
GPT-5
98.83
1.17
97.33
3.78
0.67
1.11
25.72
14
Gemini 2.5 Flash
91.64
8,36
98.0
7,89
3.56
0.22
27.42
22
Strategy Prompting
Qwen2.5-VV Owen2.5-VL-3B-
88.08
11.92
99.78
0.22
0,22
1.44
25,42
8
Qwen2.5-VV L-VVL-22---
-nnttuuuttt
91.5
8,5
99.78
5.89
17.11
46.11
42.22
12
ViterVL3-2B---
68.42
31.58
93.33
1.22
4.0
10.11
27.17
152
LttemVL3-78B-BB GPT-S
13.56
96.89
14.11
30.61
64
0.83
94.56
59.0
10.67
42.25
Gemini 2.5 Flash
94.08
5.92
99.11
8.0
10.68
30.11
36.98
35
AQUA Tuned Models
Qwen2.5-VL-3h V--3--Tued
82,22
86.33
86.28
Intem 80.44
19.56
98.78
80.0
59.67
78.0
79.11
RESULTS
Table shows the performance range of VLMs on AQUA. Across all models, factual consis- tency remains quite high, indicating that hallucinations are rare. The primary challenge, however, lies in strategic reasoning, where performance poor across all levels except Level 0. This sug- gests that differences in performance primarily reflect the models' inability to select appropriate ambiguity-hanee hanndinng strategies. Please refer to Appendix for full benchmarking results, including models of other sizes.
Base VL.Ms. Both open-source models (Qwen2.5- --L.----VLL----nr and VL.3 series) and strong closed-source models (GPT-5 and Gemini 2.5 Flash) exhibit similar performance patterns. While these models perform well on unambiguous cases (Level 0), they consistently struggle with ambiguous scenarios (Levels 1-3), showing poor performance when multiple plausible interpreta- tions or clarification requests are required. Notably, even the strongest closed-source models struggle with higher levels of ambiguity, indicating that model scale alone does not resolve the strategic rea- soning challenges posed by our dataset (see Appendix for more details). The same holds for large and Intern VLL3-78B-Instruee which also fail to consistently outperform their smaller counterparts despite their increased size.
open-source variants (e.g.. Dwwen2.-vVL-7BB-Insrre
CoT and Eerategy-Prompting We next examine whether standard prompting techniques improve ambiguity handling. We consider two prompting variants: (i) Chain-of- Thoughh (CoT) (Wei et al. 2022), where we append "Ler's think step by step." encourage stepwise reasoning, and (ii) Strat- egy Prompting, which explicitly instructs the model to choose among four response strategies de- pending on the level of ambiguity (see Appendix K for prompt). As shown in Table CoT provides no meaningful benefit and often reduces performance, since models tend to generate verbose single- answer responses instead of adapting their strategy. Strategy prompting has no effect on smaller open-source models, but yields slight improvements for larger stronger closed-source models. These findings suggest that models cannot handle ambiguity reliably through prompting alone and instead need explicit training datasets like AQUA acquire strategy-awe -ware response abilities.
Trained Models. In contrast, Qwen2.5- Owen2.5-VL-3B. VL-3B-Tuned and Intern VL3-2B 23-2B-TTuned models reach ap- proximately 80% overall strategic accuracy, substantially higher than all baselines and prompting- based variants. Importantly, these models maintain robust strategic reasoning across all ambiguity levels. Unlike base VLMs, which default to overconfident single answers, the tuned models reliably

---

Published as a conference paper at ICLR 2026
Q: What this round the cooling rack?
Q: What this helmat?
How many
the plate?
Zero-shot am not sure what color the helmet is. can be seen yellow, green, black.
Zero-shot: There three the plate: two doughnuts and three strawberries.
Zero-shot: The round cake the cooling rack pink.
Ours: There are many helmets visible different people ..... Could you specily which helmet you mean perhaps by person's position ......
Ours: There are multiple round cakes
Ours: could mean the ring-shaped pastries with red fruit filling the three bright red strawberres
on the cooling rack, Could you
specity which cake you mean
Figure 4: Response comparison of Qwen2.5-VV--3B-Inssme in zero-shot and tuned settings.
Table 3: Performance comparison of models tuned on AQuA with SFT and SFT+GRPO. G, U, and Unk respectively denote Grounded, Ungrounded, and Unknown
Model
Factual Ace.
Strategic Acc.
Level
Level
Level
Level
Overall
Unk
Qwen2.5-VL-33 -und (SFT)
82.78
17.22
99.56
92.22
61.33
82.11
83.81
2
Qwen2.5-VL.: 1..-5-V---B--Tm- (SFT+GRPO)
81.06
18.94
99.56
77.0
82.22
86.33
86.28
1
VLtem (SFT)
66.08
33.92
99.22
82.67
37.67
74.11
73.42
2
BitemVLL-2B-Tumm (SPT+GRPO)
80.44
19.56
98.78
80.0
59.67
78.0
79.11
12
adapt their strategies. This consistent behavior shows that explicit training on AQUA enables mod- els to handle visual ambiguity in human-like and strategy-aware manner. Please refer to Figure for examples of our model's strategic response.
ANALYSIS
6.1 SET AND RL TRAINING
To better understand the effect of each training stage, we conduct an ablation comparing models trained with SFT alone against those further optimized with GRPO. As shown in Table 3 models trained with SFT alone already achieve over 73% strategic accuracy overall, confirming that simple supervised training on ambiguity-awan performance on highly ambiguous cases (Levels and 3) remains lower. Applying GRPO further boosts performance, this stage not only raises accuracy on Levels 2 and 3, but also stabilizes per- formance more broadly, leading to balanced and robust strategic reasoning. However, we observe slight drop in Level 1 performance after applying GRPO following SFT (Figure 5b and 5cp. We find that models trained only with SFT tend to concentrate most of their errors in Level 1, indicat- ing either form of overfitting that level or insufficient understanding of Levels and 3. As GRPO encourages more strategic decision-making across all ambiguity levels, this bias mitigated, and the resulting redistribution of errors naturally leads to a minor decrease in Level accuracy. Please see Appendix G for confusion matrices of Intern VL3-2B based models, Also, please refer to Appendix H for additional results with increased GRPO training samples.
responses is sufficient to yield strong gains. Nonetheless,
6.2 ERROR PATTERNS
Biased Default Strategy of VLMs. Figure presents the confusion matrices of Qwen2.5-VL-3B Instruct and --een..5-VL3B--TUee (SFT+GRPO) evaluated on AQUA. the base model (Fig- ure Ba), observe strong bias toward Level predictions, where the model outputs single confident answer even when ambiguity requires context inference (Level 1), multiple listings (Level 2), explicit clarification (Level 3). This indicates that the model defaults to one-correct-answer
8

---

Published as a conference paper at ICLR 2026
97.11
0.11
2.67
0.00
0.11
Lo
99.56
0.22
0.00
0.00
0.22
Ground Truth 19pe
Ground Truih
66.22
0.11
31.00
0.11
2.56
42.
2.11
92.22
5.00
0.67
0.00
82.
66.78
0.22
27.78
0.78
4.44
0.56
9.56
7.78
82.11
0.00
12
Predicted
Predi
(a) Zero-shot
(b) SFT
(c) SFT+GRPO
Figure 5: Confusion matrices the response patterns of Qwen2.5-VLL-3--Instne on the AQUA.
Level
Level
Level 2 Level 3
Level 3 Level
Q:
What color are this one's eyes? this image, "this one" dearly points to the large black-and-whitt cat
Q: What color is this sofa?
Q: What color this?
GT: could refer to either the two sofas: longer one along the left wall beneath the wide
GT: There are multiple prominent subjects the image, "this" unclear. Could you specity which Ours: The word "this" refers to the grase;
dominating the scene, .....
Ours: could mean the cat ..... has
Ours: There are multiple sofas visible room,
yellow eyes ..... could the wooden abstract sculpture .......
prominent horse chestnut (a rich reddish-brown),
Could you specify .....
Figure 6: Our model responses level- boundaou confusion and salience-driven errors.
strategy regardless of the degree of ambiguity. Similar patterns are observed in other baseline mod- els. However, -ween..5-VL-3B-Instiie shows unusually high proportion of Level predictions
Confusion at Level Boundaries. As shown Figure 5b the SFT model tends to collapse to- ward Level responses, rather than selecting appropriate response strategies. Many error cases misclassified as Level 1, suggesting that the model overfits to specific strategy instead of accu- rately reasoning about the underlying ambiguity. In contrast, after fine-tuning with SFT+GRPO (Figure [c).. the model shows substantial improvements across all ambiguity levels. Notably, the greatly reduced, resulting in predictions that are more evenly dis-
strong bias toward Level tributed across the intended strategies. However, occasional confusions remain near the boundaries between levels, often driven by stereotypes or conventional expectations. the left example is labeled as Level because humans typically associate with the cat, however our model also considers the sculpture's 's "eyes" shifting its response to Level 2. Also, the middle example, our model treats them as distinct objects rather than a single set and requests clarification. These examples highlight how subtle biases and interpretation choices can shift predictions across adjacent ambiguity levels.
shown in Figure 66
Driven Errors. Although uncommon, the model occasionally deviates from the appro- priate strategy by focusing Figure 6, the question "What color this?" is paired with an image containing the sky, rocks, grass, and horse. The ground-truth response requires clarification request because multiple plau- sible referents exist. However, our model interprets the horse as the intended referent, as is the most salient object in the scene, and directly answers with its color. This results in shift from Level to Level 1. Such cases often arise from salient or stereotypical features that lead the model overcommit a single referent instead of requesting clarification or listing alternatives.
salient feature unintended by the query. In the right example of
$Qwen2.5-V V--.3V-----3----- VL-33B-Insttutc tends to answer with "I am not sure It could be A, B, C. when it cannot make clear decision, regardless of the ambiguity level.

---

Published as a conference paper at ICLR 2026
6.3 EFFECTIVENESS OF CLARIFICATION
In cases of high ambiguity, the model tends to ask for clarification. To assess the effectiveness of this strategy, we design experiment in a two-turn question-answer setting. Specifically, we filter 100 Level 3 instances and use GPT-5 to generate a follow-up turn consisting of single disambiguating hint and the corresponding unambiguous answer (see Appendix for examples).
For each response, GPT-5-mini serves as the judge, as- signing binary PASS FAIL depending on whether the model's final answer matches the ground-truth un- ambiguous answer (see Appendix K for prompt). As summarized in Table 4 both models achieve consis- tently high PASS rates, once a clarifying hint pro- vided, demonstrating that Level 3 ambiguity can be effectively resolved with a single clarification turn. These findings highlight the value of clarification:
Table 4: Evaluation results on the clarifica- tion subset.
Model
PASS
FAIL
Bwen2.5-VL-3BB-Tnnse
80%
20%
InternVL--2B-TI 2uned
73%
27%
with a short follow-up, the model can resolve uncertainty and provide accurate, well well-grounded an- swers rather than enumerating all possible answers in the first place.
Table 5: Evaluation results of samples generated using Open Images V7.
Model
Level
Level 1
Level 2
Level
Overall
Unk
Twen2.5-VLL-3B-mmsse
98.0
2.0
1.0
25.25
2
97.0
87.0
76.0
89.0
83.81
2
98.0
1.0
5.0
1.0
26.25
7
Intern WLterWWL3-2B-Tnnee
96.0
86.0
55.0
77.0
78.5
2
6.4 GENERALIZATION BEYOND COCO
AQUA is constructed using images from the COCO dataset, and our model is trained the same source. To evaluate generalization beyond COCO, we test the model on ambiguous image-question pairs from different image dataset. Using the same generation procedure (including human filter- ing), we construct an evaluation set from Open Images V7 with 100 samples per ambiguity level (400 samples in total). As shown Table performance on Open Images follows trends similar to those observed on COCO-based samples. Although trained only on COCO images, our models consistently select appropriate strategies across all ambiguity levels. This suggests that the mod- els generalize beyond dataset-specific visual characteristics. Overall, these results indicate that our approach is robust to dataset shift and can handle ambiguity in VQA across different image sources.
CONCLUSION
In this work, we introduce AQUA, new dataset designed not only to evaluate but also to train VLMs in handling ambiguity in VQA. AQUA defines four fine-grained levels, each aligned with a distinct response strategy. Through this design, we show that current VLMs often fail recognize and adapt to different types of ambiguity, defaulting to overconfident answers rather than reasoning strategically. By fine-tuning open-source models with supervised learning and GRPO on AQUA, we demonstrate that even relatively small VLMs can learn to choose strategies contextually-whetheer whether by direct answering, inference from context, controlled enumeration, or explicit clarification. These tuned models outperform both larger open-source and strong closed-source systems ambiguous VQA, highlighting the effectiveness of strategy-aware training. In addition, we conduct an exten- sive analysis to understand why VLMs fail to generate strategy-aware responses under ambiguity. pair ambiguous, leading
Untuned models often do not even recognize when a question-iman them produce overconfident answers. In contrast, failures in our tuned models mostly arise in boundary cases, where ambiguity levels are difficult to distinguish, or from salience-drive errors, where prominent visual features bias the response. These findings provide deeper explanation of the limitations of current VLMs and point toward the need for models that can reason more flexibly about uncertainty.
https::/sttrrgee leapi httpppss..cmmppeennaaeee///eeees
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank the reviewers and the area chair for their valuable feedback. This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (ITT)) grant funded by the Korea governmen((MSIT) (No.RS-2019-1191906 Artificial Intelligence Graduate School Program(PSSTECH), and National Research Foundation of Korea(NRF) grant funded by the Korea MSVer 22-22222222297777 and Basic Science Research Program through the National Research Foundation of Korea(NRF)) funded by the Ministry of Education(RS- 2025-25433785..
REPRODUCIBILITY STATEMENT
We provide samples of the AQUA and the training code in the supplementary materials. After the review process is complete, we will publicly release the full dataset, model checkpoints, and all source code to ensure reproducibility. In addition. implementation details for training are described in Section 4.1 and Appendix C and all prompts used in this study are provided in Appendix K and can be used fully reproduce our experiments.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502113925 2025.
Ali Furkan Biten, Ruben Tito, Andres Mañaa, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4291-4301, 2019.
Chongyan Chen, Yu-Yun Tseng, Zhuoheng Li, Anush Venkatesh, and Danna Gurari. Acknowl- edging focus ambiguity in visual questions, 2025. htth:ttp:///axxiitt www.wwwwwwwwr.g..aoooo wtt.:os//aaab////////
02201
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 24185-24198, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250-49267, 2023.
Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, and Xin Eric Wang. Grit: Teaching milms to think with images. arXiv preprint arXiv:2505. /5879, 2025.
Yash Goyal, Tejas Khot, Douglas Summers-Stre Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Yangyang Guo, Fangkai Jiao, Zhiqi Shen, Liqiang Nie, and Mohan Kankanhalli. Unk-vqa: A dataset and a probe into the abstention ability of multi-modal large models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
11

---

Published as a conference paper at ICLR 2026
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Pu Jian. Donglei Yu, Wen Yang, Shuo Ren, and Jiajun Zhang. Teaching vision-land vision-language mod- els to ask: Resolving ambiguity in visual questions. In Wanxiang Che, Joyce Nabende, Eka- terina Shutova, and Mohammad Taher Pilehvar (eds.). Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 3619-3638, Vienna, Austria, July 2025, Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10. 10.18653/v1/2025..ce acl-long 182/
182. URL httph:ttt///eee atttpes//aclaattoooooy.oooo orgga///
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Com- putational Linguistics. doi: 10. 10.18653/vL/P17-11 1147. URL https: htt.:s:s/aclaattooogo.oo orgg P17-11477
Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. Tree of clar- ifications: Answering ambiguous questions with retrieval-augmentt large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 996-1009, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 22118553/1/2023.emm 1/2023.emmlppppo... wwweemnnlmmmin...3 main.63. URL
https::/aclaathho.ooo wwtptthhooooooooo.222233.3 2wm.21m1....ppp..mmmmmmmmmmmmppm ma in. 637
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Was Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019, doi: 10. 10.1162/tucla..02276 URL
https: attpss///ccaathhoogog.. wwwpellyyyyooorooooooooo 009g//19-1026/
Dongryeol Lee. Segwang Kim, Minwoo Lee. Hwanhee Lee, Joonsuk Park, Sang-Wos Woo Lee, and Kyomin Jung. Asking clarification questions to handle ambiguity in open-domain QA. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2023, pp. 11526-1154, Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 18.18653/v 1/2023, htt.://2...nnnnngemmmm... findinng--emlll.722 //aclanthology. otg findings emnlp. 7727
https::
Jiaqi Li, Yixuan Tang, and Yi Yang. Know the unknown: An uncertainty-sennitivv method for LLM instruction tuning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025. pp. 2972- 2989, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 979-8- 89176-256-5. doi: 1118653/v1/0255 acl.nng URL bttps::/aclanteee
2r2/ findi ngs lalgg-aac.. ac1. 153/]
Zongxi Li, Yang Li, Haoran Xie, and S Joe Qin. Condambigga: A benchmark and dataset for conditional ambiguous question answering. arXiv preprint arXiv:2502.01525 2025b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
12

---

Published as a conference paper at ICLR 2026
Jingyu Liu, JingquanPeng JingquanPeng, Xiaopeng Wu, Xubin Li, Tiezheng Ge, Bo Zheng, and Yong Liu. Do not abstain! identify and solve the uncertainty. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 17177-17197, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10.1800.110 0083//61322222 acl-long.844 840. URL httppp://aclaaataooooo..o///////
acl ong. 8407
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP). pp. 5783-5797, Online, November 2020. Association for Computational Linguis- tics. doi: 10. 10.18653/v 2ATSS/////222000 emr.pp ma main. /
peaetts://aalaantcccle org/20CC.
Heejeong Nam, Jinwoo Ahn, Keummin Ka, Jiwan Chung, and Youngjae Yu. Vague: Visual contexts clarify ambiguous expressions. arXiv preprint 1xxiv:2411.141::33 2024.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Pro- ceedings of the 2016 Conference on Empirical Methody Natural Language Processing, pp- Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v/DD1-1122
1264/
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0334 033000 2024.
Amanprect Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wes Wei Chang. ASQA: Factoid questions meet long-form answers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8273-8288, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. http:///11/22222emmmmmoeeee emm ma main. 566/
UtLhttps: http:p/ss///aaaaahholloeo org/2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. sfain-of-thoughle prompting elicits reasoning in large language models. Advances in neural information processing systems, 24837,, 2022.
Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than an- swer incorrectly. In European Conference on Computer Vision, pp. 148-166. Springer, 2022.
13

---

Published as a conference paper at ICLR 2026
A EXAMPLES OF AQUA
Please refer to Figure 7 for Level 0, Figure for Level 1, Figure 9 for Level 2, and Figure 10 for Level 3 of the AQUA dataset.
Q: What animal is shown behind the chain-link fence?
A: The animal behind the chain-link fence is dog.
Q: What water sport the person wearing black wetsult performing?
A: The person is surfing.
Q: what the kite shaped like?
A: The kite is shaped like an airplane.
Figure 7: Level 0 examples of AQUA.
Q: What style these cut in?
A: The word "these" refers to the big pile of French fries the foreground, and they are crinkle-cut.
Q: How many pillows are on it?
In this image, "it" refers the large bed dominating scene, and there two pillows it.
Q: What is this made of?
The word "this" points the large round container that dominates the scene-the bowl holding the fruit-and it is made of wood.
Figure 8: Level examples of AQUA.
14

---

Published as a conference paper at ICLR 2026
What is on plate?
A: could refer to the upper plate, which holds slice of pizza topped with leafy spinach and white cheese. Or it could refer to the lower plate, which holds two slices of pepperoni pizza with red pepperoni discs.
Q: What color are these apples? you mean two apples on the raised granite countertop near the center, one is red, and the other greenish-yellow.
Q: Where this person with bat standing?
A: One interpretation is the batter at home plate the left, wearing red helmet and standing in the batter's box beside the catcher and umpire. Another interpretation the on-deck batter in the right foreground, holding bat while standing the dirt near the foul line the edge of the grass.
Figure 9: Level 2 examples of AQUA.
Q: What color is this food truck?
A: There are multiple food trucks visible, so "this" could refer to more than one. Could you specify which truck you mean (for example, by position detail it)? That clarification is needed to identify the correct target.
Q: What color is this skier's 's jacket?
A: There are many skiers visible, so I'm not sure which one you mean. Could you specify which skier you're referring (for example, their position in the image)? need clarification because multiple jackets are equally prominent and "this skier" isn't uniquely identifiable.
Q: How many fruits are this box?
There are several boxes visible, so sure which one you mean. Could you specify the box its position nearby detail? need clarification because multiple boxes are equally prominent and 'this box' could refer to any of them.
Figure 10: Level 3 examples of AQUA.
15

---

Published as a conference paper at ICLR 2026
Instructions
Look the image.
Read question the given answer.
Check the provided Ambiguity Level definition and compare with the example cases:
Please note that random quality check question may appear. If you receive this question, must pass for your work to approved.
Level 0 - No ambiguous terms
Q: What color is the object the girl is holding in her raised hand?
A: The object is white.
Direct factual answer, single clear interpretation.
Level Ambiguous term (e.g., "this", "that", "it", "the thing "), but dominant target What is this doing its trunk?
this scene, "this" means large elephant the middle, it feeding - grabbing eating surrounding vegetation with its trunk.
Ambiguity resolved by explicitly stating referent, then giving answer.
Level 2 Ambiguous with 2-3 plausible targets
Q: What color are these on top?
A: They could be the two small hippopotamus figurines standing near the center of the cake, which are light gray; or the whole strawberries arranged on the cake, which are bright red with green leaves.
v All plausible targets are enumerated, no single best guess.
Level 3 Ambiguous with many possible targets
Q: What color this on the path?
A: There muttiple visible subjects here, I'm not sure which one "this" refers to. Could you specify which item you mean? need that clarification to describe the correct color for the intended target.
Too many possible interpretations politely ask for clarification.
Image
Question:
What color is this towel?
Answer (given): I see multiple towels the image, so I'm not sure which one you mean. Could you specify which towel (for example, its position or what it's next to)? Clarification needed because "this" uniquely identify single object here.
Level (0-3):
Select your evaluation:
PASS
FAIL X
Submit
Figure 11: MTurk Worker Evaluation Screen.
16

---

Published as a conference paper at ICLR 2026
B DATASET FILTERING METHOD
To ensure the quality of AQUA, we designed a three-stage filtering pipeline:
This stage verifies that each generated crestion-answer pair satisfies the requirements of its assigned ambiguity level. For example, Level 0 samples must contain ambiguous terms and allow only one definitive answer, while Level 1 samples must contain at least one ambiguous term but resolve it confidently in the answer. This acts as a strict rule-based gate to filter out obvious mislabeling (e.g., a Level 0 example using "this", or Level 2 answer that selects only one option).
Stage 1 - Level Consistency Check:
Stage 2 Best Fit Validation: Even if sample meets the basic criteria of its assigned level, it may be more appropriately categorized under a different level. This stage checks whether the assigned level is the unique best fit among the four categories. LLM--a---judgee compares the question-answer pair against canonical definitions and applies explicit priority rules. For example, if question uses an ambiguous term but only one dominant object is present, Level I is always preferred. This ensures that each retained sample is not only valid but also aligned with the most specific ambiguity level.
Stage - Seal-World and Quality Validation: The final stage ensures that each sample is suit- able for inclusion in real-world VQA dataset. This includes (i) confirming that the underlying image is natural photograph with sufficient clarity, (ii) verifying that the question refers only to observable properties (e.g., color, shape. size, count) without requiring hidden knowledge, and (iii) checking that the answer is grounded in the image and consistent with the behavioral ex- pectations of its level. This stage also eliminates degenerate cases such as synthetic or corrupted images, hallucinated content in answers.
After applying the three-stage filtering process to all data samples, we further enhance the reliability of AQUA by conducting an additional human validation stage for the evaluation split. This step is carried out on the Amazon Mechanical Turk (MTurk) platform, where we restrict participation to workers with more than 5K previously approved HITs and an approval rate above 95%. Annotators presented with the image, question, and answer, and asked to judge. considering the assigned ambiguity level, whether the sample is acceptable, providing a binary PASS/FAIL decision. Each sample is independently evaluated by two annotators, and only those that receive PASS label from both are retained in the dataset. As an additional safeguard, we inject 10% fake samples into the annotation pool. If worker incorrectly assigns PASS label to any fake sample, all of their submitted annotations are discarded. Please see Figure 11 for the instructions and interface used in the human validation stage.
C IMPLEMENTATION DETAILS
Our training procedure consists of two stages: (1) supervised fine-tuning (SFT) and (2) Group Rel- ative Policy Optimization (GRPO). All trainings are conducted on 8 NVIDIA RTX A6000 GPUs.
For SFT, we fully fine-tune Weeen.55vVL33B--nsse AdamW optimizer, a learning rate of 5 x 10-5, a constant_with_ _ scheduler with a warmup ratio of 0.03, and gradient checkpointing enabled. Training performed for 3 epochs with an auto- fined per-device batch size and a gradient accumulation step of 4, and gradients are clipped at 1.0. For
using the HuggingFace Trainer with the
we also fully fine-tune the model using the official Intern VL training script with the Adam W optimizer, learning rate of 2 x 10-5 5, weight decay of 0.05, a cosine learning rate scheduler with warmup ratio of 0.03, and gradient checkpointing. Training con- ducted for 3 epoch with a per-device batch size of 4 and a gradient accumulation step of 4. We apply early stopping with a patience of 1 for both models and select the best-performing checkpoint accordingly.
For GRPO, we adapt the training scripts released by Fan et al. (0025). The reward function is implemented with GPT-5-mini. We train for 30 epochs with learning rate of 5 x 10-6, batch size of 2, gradient accumulation steps of 2, and 3 = 0.01, using a cosine learning rate scheduler. For each sample, generate four responses, compute rewards for each, and update the model using croup-based advantages combined with KL divergence against a reference model. We select the final checkpoint based on the highest validation reward.
17

---

Published as conference paper at ICLR 2026
Table Full benchmarking results of various VLMs on AQUA. G, and Unk respectively denote Grounded, Ungrounded, and Unknown.
Model
Factual Acc.
Strategic Acc.
G
U
Level 0
Level 1
Level 2
Level Overall
Unk
Zero-shot
Owen2.5-VT. -...---V.----nsstrucc
79.86 86
20.14
97.11
0.11
33.33
0.78
32.83
104
87.97
12.03
98.78
0.78
3.67
3.33
26.64
25
Qwen2.5-VL-7228
89.33
10.67
99.56
0.56
2.11
0.89
25.78
12
IntemVLL3-2B-nsttuu
76.63
23.37
96.0
2.33
3,56
1.89
25.95
138
MitemVLL-8B-IInttune
81.52
18.48
97.67
1.67
2.11
2.67
26.03
94
InterVLL3-78B-
80.5
19.5
96.0
2.11
3.0
5.67
26.7
133
GPT-5
98.4
1.6
89.67
0.67
0.33
0.78
22.86
178
Gemini 2.5 Flash
91.89
8.11
99.00
5.22
4.44
0.89
27.39
Chain-of- Thought (CoT)
Qwen2.5-Vi 11-.55-V--BB--tttreee
78.22
21.78
95.89
8.33
5.67
3.78
28.42
60
Qwen2.5-VI. 1..-5-VL--BB--sttteee
83.69
16.31
88.0
11.46
5.01
2.89
26.85
31
Qwen2.5-VL--7 1tt.5---VLL22-Isstteee
86.97
13.05
93.0
13.78
2.78
1.33
27.72
10
IntermVL.3-2B-Insti
76.08
23.92
97.67
2.44
1.33
1.11
25.64
54
Intern VLtemmLL3-8B-Insst
76.17
23.83
95.22
7.67
3.0
9.11
28.74
127
MitemVLL3-78--nstteee
79.75
20.25
96.78
5.22
3.67
12.33
29.5
74
GPT-5
98.83
1.17
97.33
3.78
0.67
1.11
25.72
14
Gemini 2.5 Flash
91.64
8.36
98.0
7.89
3.56
0.22
27.42
22
Strategy Prompting
Qwen2.5-VV -ree2.5-VL-3BB-o
88
11.92
99.78
0.22
0,22
1.44
25.42
8
Swen2.5-VL-7BB
90.64
9.36
99.67
0.78
1,33
10.33
28.03
16
Qwen2.5-VL-P7 VL-25-- Instrucct
91.5
8.5
99.78
5.89
17.11
46.11
42.22
12
IntemmVL3-22-Instne
68.42
31.58
93.33
1.22
4.0
10.11
27.17
152
78.03
21.97
90.67
11.11
9.67
17.11
32.14
57
Intem
86.44
13.56
96.89
5.56
5.89
14.11
30.61
64
GPT-5
99.17
0.83
94.56
59,0
10.67
4.78
19
Gemini 2.5 Flash
94.08
5.92
99.11
8.0
10.68
30.11
36.98
35
AQUA Tuned Models
Qwen2.5-VI.s -1..5--VV------11 Tuned (SFT)
82.78
17.22
99.56
92.22
61.33
82.
83.81
2
Qwen2.5-VL. 3-V--3BB-TTee (SFT+GRPO)
81.06
18.94
99.56
77.0
82.22
86.33
86.28
IntemmLL3-2B-- (SFT)
66.08
33.92
99.22
82.67
37,67
74.11
73.42
Intern Mi----L-----Teeee (SPT+GRPO)
80.44
19.56
98.78
80.0
59.67
78.0
79.11
12
D VERIFICATION FOR LLM-AS-A-JUDGE
To verify the reliability our LLM-as-a-judge framework, we conduct an in-house evaluation on a sample of responses from Dree2..-VLL3B--nsttee Instruct and Siwen..-VL-33B-TTT
Specifically, randomly sample 400 responses: 100 classified as Grounded and 100 classified as Ungrounded for factual consistency, and 50 from each ambiguity level for strategic accuracy. Human annotators then independently assess whether GPT-5-mint judgments are correct.
The results show high degree of agreement between GPT-5-mini and human evaluation. Out of the 400 sampled cases, only 5 are misclassified in factual consistency and in strategic accuracy, resulting in an overall agreement of 98.5%. This strong alignment demonstrates that GPT-5-mini serves as reliable judge for our evaluation protocol and confirms that our automatic evaluation is trustworthy for large-scale benchmarking.
E FULL BENCHMARKING RESULTS
Please Table for full benchmark results for range of VL.ss.
18

---

Published as a conference paper at ICLR 2026
Table Evaluation results of iterative prompting with GPT-5 using repeated judging and correction.
Stage
Level 0
Level
Level 2
Level 3
Overall
Unk
Stage
95.0
60.0
11.0
3.0
42.25
Stage
97.0
77.0
14.0
8.0
49.0
Stage
98.0
83.0
19.0
15.0
53.75
Stage 4
97.0
89.0
25.0
16.0
56.75
6
Stage
98.0
91.0
34.0
14.0
59.25
11
Stage
99.0
91.0
35.0
16.0
60.25
Stage 98.0
94.0
41.0
17.0
62.5
Stage
100.0
97.0
39.0
16.0
63.0
Stage
100.0
96.0
39.0
20.0
63.75
Stage 10
100.0
95.0
43.0
19.0
64.25
96.00
0.78
2.89
99:22
0.22
Lo.
0.56
Ground Type
Ground Truth Type
22
91:33
2.33
2.50
0.44
3.33
Ground Truth
L2.
3.44
$2.07
8.44
5.44
0.00
L2:
2,44
80.00
13.33
3.89
0.33
12
11.22
26.11
L2-
59.67
1,67
1,22
13.44
74.11
4.78
16.00
78.00
Unk
Le
Unk
12
Vnk
(a) Zero-shot
(b) SFT
(c) SFT+GRPO
Figure 12: Confusion matrices of the response patterns InternVL3-2B -nIstructt on the AQUA.
F ITERATIVE PROMPTING ANALYSIS
In our experiments, we observe that even the strongest closed-souree source models struggle with strategic handling of visual ambiguity. To examine whether improved prompting or more explicit guidance can help such models better resolve ambiguity, this setup, GPT-5 repeatedly generates, judges, and revises its responses based on predefined strat- egy guidelines. We conduct experiments 200 test examples (50 per ambiguity level) over 10 refinement iterations.
also evaluate iterative prompting setup. In
As shown in Table while iterative prompting leads modest improvements, it remains ineffective on highly ambiguous cases and incurs substantial computational and latency overhead. Overall, this model, suggesting that
approach fails to achieve the level of consistency observed in the fine-tuning is necessary for robust, level-aware strategic responses to ambiguity.
G ANALYSIS OF ERROR PATTERNS
The confusion matrices of Intern Innssett and Intern VL3-2BBTuned (SFT+GRPO) on AQUA are shown Figure
H INCREASING GRPO TRAINING SAMPLES
After SFT, we train the model using GRPO with 60 samples (15 examples per level). To examine how performance depends on the number of GRPO training samples, conduct additional experiment using 120 samples (30 examples per level). As shown Table increasing the number of training samples from 60 to 120 leads improved performance strategic choice.
19

---

Published as a conference paper at ICLR 2026
Table Evaluation results with different numbers of GRPO training samples. the model names, "60" and "120" indicate models trained with 60 and 120 GRPO samples, respectively.
Model
Level
Level 1
Level 2
Level
Overall
Unk
99.56
77.0
82.22
86.33
86,28
99.44
86.44
73.33
92.11
87.83
2
Intern -1.---B-TTnnd---
98.78
80.0
59.67
78.0
79.11
12
Intern -L3-2B--TnBe 120
96.0
90.3
57.89
85.33
82.39
29
Qambiy: How many people are standing to the left of this person? Astartry: I'm not sure which person you're referring to to-there are several people visible. Could you specify which one you mean so can count accurately? Clarification is necessary because the number changes depending on the chosen person.
Presolved: Count the number people standing to the left of the person holding red umbrella the center. A Agota: There three people standing to the left of the person with the red umbrella.
Qambig What color this player's jersey?
Actarity: There are several players visible, so can't tell which person you mean. Which player are you referring to? need clarification because "this player" could be any of them and the image doesn't indicate a specific target.
P The reference to the jersey color the player lying on ground the foreground being attended by two men black.
Agoia: The player's jersey is red.
Figure 13: Examples from Level 3 of the AQUA dataset, where follow-up questions used to resolve ambiguity and arrive at the gold answer.
These results suggest that increasing the number of GRPO training samples can further benefit per- formance. In this work, we strategically adopt 60 samples balance training time and computational cost. Since the goal this work is not to identify the optimal sample size, we not explore smaller substantially larger sample settings. We leave more systematic investigation of the effect of GRPO sample size to future work.
1 EXAMPLES OF CLARIFICATION SUBSET FOR LEVEL 3
Please refer to Figure 13 for Level 3 subset examples, where follow-up questions help resolve am- biguity and guide the model the intended gold answer.
J THE USE OF LARGE LANGUAGE MODELS
We use large language models (LLMs) throughout this work support tasks such as data genera- tion, filtering, and reward modeling. During these processes, the authors remain actively involved, carefully reviewing the process and outputs to ensure that results not solely produced by LLMs but refined through human review and curation. We also use LLMs during the paper writing stage for grammar checking and minor language refinement. However, the overall writing of the paper conducted directly by the authors without relying on LLMs.
20

---

Published as a conference paper at ICLR 2026
K
PROMPTS LIST
Prompt for Level 0 Sample Generation
### Instruction:
Given an image, create exactly one question- pair that is clear, factual, and unambigu- ous, such that only one correct answer exists.
### Requirements:
Must:
Ensure the question has a single, definitive interpretation.
Base the question entirely factual, observable elements in the image.
Phrase both the question and the answer in complete, clear sentences.
Ensure the answer is definitively determinable from the image without external knowledge.
Avoid:
Any ambiguous terms like "this", "that", "it", "there".
Subjective or interpretive elements (e.g., opinions, feelings, aesthetic judgments).
Questions that allow multiple plausible answers.
### Output Format:
(Question: [Your generated question], Answer: [The definitive answer]} If the image does not allow for such a guestion-answer pair, output exactly: none
Do not include explanations additional text.
### Response:
Prompt for Level Sample Generation
### Instruction:
Given image, create exactly question-answer pair where:
The question uses an ambiguous term, but the image context makes the intended subject completely clear.
The answer must explicitly resolve the ambiguity first and then give the factual answer. ### Must:
Use at least one ambiguous term (c.g., "this", "that", "it", "the thing", etc.).
Ensure there is exactly one clearly dominant object in the image that stands out from all others in size, position, or salience.
In the answer, naturally explain what the ambiguous term refers to in this specific image, then provide the definitive descriptive answer.
Write the answer in full sentences.
### Avoid:
Questions that would remain clear without ambiguous terms.
Scenes with multiple objects of equal prominence.
Answers that only give the fact without clarifying the referent.
Overly short or one-word answers.
21

---

Published as a conference paper at ICLR 2026
Beginning with fixed templates such "Here, 'this' refers to ". Each answer must be phrased naturally and vary across samples.
### Output Format:
[Question: [Your ambiguous question], Answer: [Your natural clarification plus the definitive descriptive answer])
If the image does not meet requirements, output exactly: none
Do not include explanations any extra text.
### Response:
Prompt for Level 2 Sample Generation
### Instruction:
Given an image, create exactly one question-answer pair where:
-he question is ambiguous and could reasonably refer to multiple distinct objects in the scene.
-The answer lists all plausible interpretations rather than choosing only one.
### Must:
Ensure the image contains at least two and at most three reasonable target objects.
Use ambiguous terms (e.g., "this", "that", "they", "these", etc.) in the question.
Clearly describe each possible target in the answer so that the user can decide.
Make each description factual and directly observable from the image.
### Avoid:
Questions that clearly refer to only one object.
Scenes where one object completely dominates.
Scenes with more than three equally plausible candidates.
Random guessing or adding details not visible in the image.
### Output Format:
(Question: [Your ambiguous question], Answer: [Natural, descriptive sentences listing each plausible interpretation]}
If the image does not meet requirements, output exactly: none
Do not include explanations or any extra text.
### Response:
Prompt for Level 3 Sample Generation
### Instruction:
Given an image, create exactly one question answer pair where:
The question contains ambiguous references, and the image provides no clear clues to identify the intended target.
The answer requests clarification without attempting to guess or list possible options.
### Must:
Include at least one ambiguous term (e.g., "this", "that", "it", "the thing", etc.).
Ensure there are multiple equally prominent objects in the image.
22

---

Published as a conference paper at ICLR 2026
Make the question about clearly visible, observable properties (e.g., color, shape, size, posi- tion, visible text, count, material, etc.).
In the answer, acknowledge that there are multiple possible targets and politely ask which one intended.
-Briefly explain why clarification is necessary. ### Avoid:
Listing all possible targets.
Making any guesses or inferences.
Using scenes where one object is clearly dominant.
Asking about non-observable or speculative properties (e.g., device state, hidden contents, functionality, temperature, brand unless clearly visible).
### Output Format:
(Question: [Your ambiguous question], Answer: [Your clarification request]]
If the image does not meet requirements, output exactly: none
Do not include explanations or any extra text.
### Response:
Prompt for Filtering Stage 1
### Role You are the LEVEL CHECKER. Decide ONLY whether the given (Level, Question, Answer) correctly demonstrates the REQUIRED behavior for the assigned Level.
If any requirement is not satisfied, output FAIL. Do NOT suggest other Levels or reassign.
### Level Requirements (must ALL hold)
Level 0 (Clear VQA): Question: clear, specific, and unambiguous; no demonstratives like "this/that/it".
Answer: direct, factual, determinable from the image; full sentence allowed.
PASS if: ambiguous references and only one definitive interpretation.
-FALL if: ambiguous terms appear OR multiple interpretations exist.
Level (Context-resollved ambiguity):
Question: contains at least one ambiguous term (e.g., "this", "that", "it", "the thing"".
Context: exactly one clearly dominant target makes the reference obvious.
Answer: must explicitly clarify what the ambiguous term refers to, and then provide the factual description in natural full sentence.
-PASS the answer both (1) resolves the referent of the ambiguous term and (2) provides a definitive, image-grounded answer in natural language.
FAIL if: no ambiguous term OR multiple objects are equally prominent OR the answer skips the clarification step OR the answer is just single word/short fragment.
Level (List all plausible options):
Question: ambiguous with 2-3 plausible targets.
Answer: enumerates ALL plausible targets (do NOT pick one best guess); each described clearly and factually. phrased in natural sentences rather than bullet points.
PASS if: 2-3 plausible targets exist and the answer lists them all natural descriptive sen- tences.
23

---

Published as a conference paper at ICLR 2026
FAIL if: target dominates OR more than three plausible targets OR the answer picks single guess OR the answer is bullet/fragmentedd list style.
Level (Clarification required):
Question: ambiguous with many or equally plausible targets; no reasonable best guess.
politely requests clarification, acknowledges multiple possibilities WITHOUT list- ing them, and briefly explains why clarification needed.
Answer:
-PASS if: inference is not reasonable and the answer requests clarification (no listing, guessing).
-FAIL if: one object is clearly more prominent OR reasonable best guess exists OR the answer lists options.
### Universal Quality Checks (all Levels must satisfy):
Question concerns visible, observable propertics (ebbrrslaccsszeppptttoooooo/osbe text/materiial).
No speculativehhidden-sttte queries.
Answer is phrased naturally and consistent with image-grounded behavior for its Level.
### Output Return exactly one token: PASS or FAIL.. No explanations.
### Item to Evaluate Level: [Level] Question: (Question) Question: (Question)
### Your Evaluation:
Prompt for Filtering Stage 2
### Role You are the BEST-FIT VALIDATOR. Decide ONLY whether the assigned Level is the BEST FIT among A/B/C/D for given (Question, Answer).
If ANY other Level fits better than the assigned Level, output FAIL. Do NOT relabel or suggest a new Level.
### Canonical Level definitions (for comparison only)
Level O: no ambiguous terms; single clear interpretation; direct factual answer.
Level I; ambiguous term present; exactly one dominant target; answer explicitly clarifies what the ambiguous term refers and then provides the definitive descriptive answer in a natural full sentence.
Level 2: ambiguous with 2-3 plausible targets; answer enumerates ALL in natural descriptive sentences (no single best-guess).
Level 3: ambiguous with many/equally plausible targets; no reasonable best guess; answer po- litely requests clarification without listing options and briefly states why clarification is needed.
### Best-Fit Priority Rules
If no ambiguous term + prefer O.
If ambiguous term and one dominant target - prefer 1.
If 2-3 plausible targets and the answer lists all - prefer 2.
If many/equally plausible targets and the answer requests clarification (no listing) prefer
3.
IIf multiple seem possible, choose the most specific by these rules.
### Task PASS iff the assigned Level is the unique best fit.
FAIL any other Level appears more appropriate or equally/more consistent.
24

---

Published as a conference paper at ICLR 2026
### Universal Sanity Checks (must hold; otherwise FAIL)
Question about observable visual properties only.
No speculativ/hhidden-sttte queries.
Answer phrased naturally and consistent with image-grounded behavior.
### Output Return exactly one token: PASS or FAIL.
### Item Evaluate
Assigned Level: Level
Question: Question
Answer: Answer
### Your Evaluation:
Prompt for Filtering Stage 3
### Role
You are the QUALITY VALIDATOR. Decide ONLY whether the (Image, Question, Answer) is suitable for a real-world, image-grounded VQA dataset.
Do NOT re-evaluate or change the assigned Level. Fail on quality issues only.
### Pass Conditions (ALL must hold)
REAL-WORLD IMAGE:
Single real-world photograph (not drwwinggCGGlloollleeepppiststrrr screenshot),
Sufficient visual information: blank, extremely blurredloverexpooeddtoo dark, not extreme crop.
OBSERVABLE-ONLA QUESTION:
About clearly visible, observable properties (color, shape, size, position, count, visible text, obvious material).
No hidden states, internal contents, functionality, temperature; no brands unless clearly visi- ble.
IMAGE-GROUNDED ANSWER:
Consistent with what can be seen; no external knowledge.
Wording aligns with the assigned Level's expected behavior (O=definitive; 1=confident; 2=liits:3=llrrfiess, (Do NOT reclassify-only fail if clearly contradictory.)
BASIC SANITY:
QQ-A refers elements plausibly in-frame; off-frame references.
No unreadable OCR claims; no unsafe/privacy--volating content.
### Fail Conditions (ANY FAIL)
worl--soddd/coooossi image; insufficient visual info.
Question requires inference beyond visible evidence.
Answer contradicts mnage-groundednes or Level's behavioral form.
References to objects/text not legibly present.
#### Output
Return exactly one token: PASS or FAIL.
25

---

Published as a conference paper at ICLR 2026
### Item Evaluate
Assigned Level: Level
Question: Question
Answer: Answer
### Your Evaluation:
Prompt for Factual Consistency
### Instruction:
You are judge evaluating whether model's answer is factually correct given the image.
Ignore style, grammar, or completeness.
PASSS if the answer correctly refers to something that is actually visible in the image, even if it does not mention everything or omits other details.
FAIL only if the answer mentions something not present in the image, contradicts the image, or hallucinates details.
### Output:
Return exactly one label:
-PASS
-FAIL
### Visual Question and Answer:
Question: {Question}
Answer: {Answer}
### Response:
Prompt for Strategic Accuracy
### Instruction:
You are an LLMMas-jjdge evaluating how a model's answer handles ambiguity.
Ignore the question completely. Judge only the style and content of the answer. Classify the answer as Level 0, 1, 2, 3, or according to the definitions below. Do not provide explanations reasoning - output only the type label.
### Type Definitions:
Level 0 Clear VQA
Answer is direct, factual statement.
No mention of resolving ambiguous terms.
Reads as straightforward description of what is visible.
Level Context-resolved Ambiguity
Answer explicitly clarifies what an ambiguous term refers to, then provides the definitive factual description.
Typically has two-part structure: clarification the referent + final descriptive answer.
Must be natural full sentence, just fragment.
26

---

Published as a conference paper at ICLR 2026
Level 2 List All Plausible Options
Answer enumerates 2-3 distinct possibilities.
Each possibility is described in natural sentences (not bullet points).
No single option is selected as the "best guess".
Level 3 Clarification Required
Answer does not attempt to guess or enumerate.
Politely requests clarification from the user.
Acknowledges that multiple possibilities exist without listing them.
Briefly explains why clarification is needed.
Level 4 Unknown
The answer does not fit any of the above patterns.
Use this if the answer is irrelevant, nonsensical, off-topic, mixes multiple types in way that does not clearly align.
### Answer:
[nnser))
### Response:
Prompt for Strategy-Prompting
###Instruction:
Look at the image and the question, and respond strategically based on the level of ambiguity.
If there is no ambiguity, answer clearly and factually.
If the question uses ambiguous term but context makes one target obvious, first clarify what the ambiguous term refers to, then provide the definitive factual answer in a natural full sentence.
If the question allows two or three plausible targets, describe all of them in full sentences without choosing single best guess.
If the question has too many or equally plausible targets, politely ask for clarification.
###Question:
{question)
###Reppons:
Prompt for Clafrification Subset
### Instruction:
You are data constructor for Visual Question Answering (VQA).
Given (1) an ambiguous question about an image and (2) clarification response, generate resolved annotation in JSON format.
### TASK:
Your output must include:
attr type: the attribute type of the question (choose from: color, shape, position, count, visi- ble text, material, etc.)
27

---

Published as a conference paper at ICLR 2026
Hint: sentence that uniquely identifies the target object in the image
--reoolved: the clarified sentence (not question type), rewritten to match the resolved mean- ing while keeping the same attribute type
confident, single-sentence answer grounded in the image (no hedging or uncer- tainty)
A_gold:
### CONSTRAINTS: -attr-tppe must be exactly one of the listed categories.
Hint must uniquely describe the object using clear visual cues (category, position, relations, or visible text).
resollee must stay aligned with attr-type
A_gold must one confident sentence, no ambiguity, no hedging.
-Ouiput valid minified JSON only.
### INPUT:
Ambiguous Question: {Question}
Clarification Response: (Response)
### SCHEMA: {"attr type":"..." """"", "". ""..."" resolved".."... ""..",""A gold"."..." ""."""
### Response:
28


Reviews:
[{"review_id": "plW4SBkuOM", "reviewer": "Reviewer_xhCh", "summary": "The paper introduces a new fine-grained dataset AQuA with ambiguous questions that requires vision-language models to recognize when they cannot answer a question. Most models answer ambiguous questions confidently instead of abstaining or seeking clarifications. The paper defines a 4-level categorization of question ambiguity, and an answering strategy for each level.  L0 being easy unambiguous qs, L1 qs are also unambiguous but require the model to resolve the salient referent, L2 qs have 2/3 possible answers, and L3 has qs where enumeration is inefficient and the model must request clarification.\n\nAQuA is built using COCO images and uses provided bbox annotations to control ambiguity levels, for ex. L1 images have a single salient obj (exactly 1 bbox covering at least 20% of the image). GPT-5 is used to generate question-answer pairs for each level. The dataset is filtered to verify ambiguity level and answer correctness using GPT-5-mini (7.2K samples in final dset). The eval split is further filtered with human annotators who verify if each sample belongs to the assigned ambiguity level.\n\nExperiments are performed on 4 open-source models (qwen2.5 & internvl3 family) and on GPT-5, Gemini-2.5-Flash.  Pretrained models are evaluated using zero-shot prompts, CoT prompts, and a strategy prompt. Furthermore, qwen2.5vl-3b & internvl3-2B models are finetuned on AQuA to handle ambiguous questions, first in a supervised manner followed by RFT using GRPO. In GRPO, a generation gets a reward (from GPT-5-mini as a judge) of 1 for grounded answer & correct strategy, and partial reward for correct strategy. Results are shown on the eval set of AQuA, where the trained models show better ability to choose the correct answer strategy.", "questions": "- In RFT, rewards are provided by GPT-5-mini. What is the computational overhead of this? How does training time compare to simpler alternatives like a locally hosted judge, or simpler format based rewards (for example looking for words similar to \"clarify\" in answers to L3 questions)?\n- RFT is performed using just 60 training samples. Is there any merit to choosing more data? \n- Performance on Out-of-Domain data. Are AQuA-trained models generalizable? Evaluation on OOD ambiguity datasets such as VQ-FocusAmbiguity[1], ClearVQA[2], and on hallucination benchmarks like POPE[3], AMBER[4], HaloQuest[5] would strengthen claims.\n- Strategy Prompting seems effective in generating grounded responses. It would be interesting to see qualitative outputs of the same.\n\nI believe the formulation of AQuA with strict 4-level taxonomy and a single \"correct\" strategy limits its practical utility (as mentioned in pts 2&3 in weaknesses). Furthermore, the strategic acc metric measures conformity to the proposed levels rather than measuring the model's ability to answer ambiguous queries. The factual acc metric only checks for groundedness and not for correctness of the answer. In light of these issues, I believe AQuA does not provide a practical, reliable way to quantify performance under ambiguity.\n\n\n[1] Chen, C., Tseng, Y., Li, Z., Venkatesh, A., & Gurari, D. (2025). Acknowledging Focus Ambiguity in Visual Questions.  \n[2] Jian, Pu et al. “Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions.” ArXiv abs/2507.13773 (2025): n. pag.  \n[3] Li, Yifan et al. “Evaluating Object Hallucination in Large Vision-Language Models.” Conference on Empirical Methods in Natural Language Processing (2023).  \n[4] Wang, Junyang et al. “An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation.” ArXiv abs/2311.07397 (2023): n. pag.  \n[5] Wang, Zhecan et al. “HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning.” ArXiv abs/2407.15680 (2024): n. pag.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The paper identifies and attempts to tackle the issue of overconfident predictions by vision-language models for questions that are ambiguous.\n- The data generation pipeline of AQuA is described in detail. Human filtering on eval split is performed to ensure clean samples.\n- The paper is well written and easy to follow.", "weaknesses": "- The dataset is not meaningfully 'fine-grained'. There are only 4 categories of ambiguity, with real-world objects (also not from fine-grained categories)\n- AQuA has a single fixed \"correct\" answer strategy for each level which is unrealistic. In real interactions multiple strategies (or combinations of them) are also appropriate. For instance AQuA says the only acceptable strategy for answering L3 questions is to ask for clarifications, whereas realistic answers could involve making a best-guess based on stated assumptions, followed by alternative coarse answers, followed by user clarifications etc.\n- Samples are strictly classified into 4 categories, which is not reflective of real scenarios that can fall in between categories. Many cases lie between levels, for ex, an image with 5-10 apples falls in b/w L2 & L3, for which acceptable solutions can involve enumerating a few options and then asking for clarifications.\n- Issue with metrics:\nThe *strategic accuracy* metric measures the ability of the model to conform to the categorizations made by AQuA and does not measure the true ability of the model to handle ambiguity. As discussed above, having a fixed strategy as ground-truth is unrealistic. The factual consistency prompt does not check for correctness of the answer and only measures groundedness. Better evaluation metrics are needed to measure a model's effectiveness for ambiguous questions (including checking correctness of answers).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CD0IdUUG6m", "reviewer": "Reviewer_tdiC", "summary": "This paper tackles an interesting problem: teaching VLMs to handle ambiguous questions through strategic responses across four ambiguity levels. The motivation is solid and the experiments are thorough, but I have concerns about the heavy reliance on GPT-5 throughout the entire pipeline, the small dataset size, and some questionable design choices. The core idea has merit, but the execution has limitations that affect how much we can trust the results.", "questions": "Please answer the questions in the weaknesses section", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The core idea is well-motivated\n- Getting 3B models to outperform 72B+ models shows this training approach works.", "weaknesses": "- Generation, filtering, and evaluation all use GPT-5 variants. This creates circular logic—you're essentially teaching models to mimic GPT-5's behavior and then using GPT-5 to judge success\n- 3.6K training samples from COCO only. Will this generalize to other domains?\n- Why 20% bounding box area for Level 1? Why not 15% or 25%? No ablation studies to justify these choices.\n- How do humans perform on strategic selection?\n- Performance drops from 92.22% to 77.0% (Fig. 5). The \"redistribution\" explanation feels unsatisfying—this is a big drop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uCGRGtSXg8", "reviewer": "Reviewer_D7BQ", "summary": "The paper introduces AQUA, a dataset trying to solve the ambiguity issues in VQA. There are four fine-grained levels. This work fine-tunes VLMs with SFT followed by GRPO using an LLM-as-a-judge reward. Fine-tuning small models substantially improves the accuracy across ambiguity levels, outperforming larger open- and closed-source baselines.", "questions": "1. Since LLMs can be used a a judge for reward assignment, one question is, why cannot this ambiguity problem solved via prompting techniques? If GPT-5-mini can serve as the judge, can gpt-5-mini, with the proper prompts, just directly solve the ambiguous VQA problem? Is there really a need for fine-tuning? E.g., make this iterative -- let one LLM first give an answer, then use gpt-5-mini judge it, then iteratively let the LLM refine its answer. What will be the results of this approach? What's the trade-off here?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research problem is clearly framed, with 4 levels of categorization\n2. The dataset construction pipeline has human validation", "weaknesses": "1. The importance of the problem in real-world settings. In Figure 1, the other models' answers still seem reasonable to me. So I wonder about the significance of the problem in the VQA setting.\n2. The rationale/completeness behind the 4 different levels. How can you tell whether there aren't other ambiguous questions?\n3. It seems that the difference between the levels is simply the number of salient objects, which can be quite subjective or prone to errors. You need to pre-define a size threshold, which seems arbitrary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sNKNox0U8z", "reviewer": "Reviewer_fMCK", "summary": "The paper proposes a new benchmark dataset for training and evaluating Vision–Language Models (VLMs), specifically on how to handle different types and degrees of ambiguity in visual questions. The differentiating idea is to extend the binary abstention decision to a four-level taxonomy of ambiguity. Also, the dataset is labelled with four optimal response strategies: direct answer, context-based inference, enumerating plausible alternatives, or explicit clarification.", "questions": "How does the trained model perform on non-COCO datasets?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The four-level taxonomy of ambiguity is very reasonable and novel.\n- The dataset is well structured. The label validation is thorough. Overall, solid methodology-wise.\n- The dataset also aligns well with two-stage training, with SFT and GRPO subsets.\n- The failure mode analysis is informative.", "weaknesses": "- Like other datasets derived from existing labels and GPT models, there would be potential biases. More discussion on bias mitigation would be helpful.\n- The four-level taxonomy is better than binary decisions. But it can still be too rigid. More experiments and analysis on ambiguous cases would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)