Right this way: Can VLMs Guide Us to See More to Answer Questions?
Authors: Li Liu
Diji Yang
Sijia Zhong
Kalyana Suma Sree Tholeti
Lei Ding
Yi Zhang
Leilani H. Gilpin
Venue: NeurIPS 2024 poster (2024)
Keywords: visual accessibility
self-knowledge
vision language models
Primary area: generative_models
TL;DR: Our study explores VLMs capabilities in assessing answerability and guiding information acquisition, introducing a novel task with  a benchmark dataset and a data-efficient training framework to improve the model's performance.

Abstract: In question-answering scenarios, humans can assess whether the available information is sufficient and seek additional information if necessary, rather than providing a forced answer. In contrast, Vision Language Models (VLMs) typically generate direct, one-shot responses without evaluating the sufficiency of the information. To investigate this gap, we identify a critical and challenging task in the Visual Question Answering (VQA) scenario: can VLMs indicate how to adjust an image when the visual information is insufficient to answer a question? This capability is especially valuable for assisting visually impaired individuals who often need guidance to capture images correctly. To evaluate this capability of current VLMs, we introduce a human-labeled dataset as a benchmark for this task. Additionally, we present an automated framework that generates synthetic training data by simulating ``where to know'' scenarios. Our empirical results show significant performance improvements in mainstream VLMs when fine-tuned with this synthetic data. This study demonstrates the potential to narrow the gap between information assessment and acquisition in VLMs, bringing their performance closer to humans.

Right this way: Can VLMs Guide Us to See More to Answer Questions?
Li Liu"
Diji Yang"
Sijia Zhong Yi Zhang Leilani H. Gilpin
Kalyana Suma Sree Tholeti
Lei Ding
University of California, Santa Cruz
(11iu112, dyyul1,,,yang39,. wwwoong szhong16, ktholeti 1iiinn2n,,.. yiz, piilliilllliillleel... edu
Abstract
In question-answering scenarios, humans can assess whether the available in- formation is sufficient and seek additional information if necessary, rather than providing forced answer. In contrast, Vision Language Models (VLMs) typi- cally generate direct, one-shot responses without evaluating the sufficiency of the information. To investigate this gap, we identify a critical and challenging task the Visual Question Answering (VQA) scenario: can VLMs indicate how adjust an image when the visual information is insufficient to answer a question? This capability especially valuable for assisting visually impaired individuals who often need guidance to capture images correctly. To evaluate this capa- bility of current VLMss, we introduce human-labeled dataset as benchmark for this task. Additionally, we present an automated framework that generates synthetic training data by simulating "where know" scenarios. Our empir- ical results show significant performance improvements in mainstream VLMs when fine-tuned with this synthetic data. This study demonstrates the potential to narrow the gap between information assessment and acquisition in VLMs, bring- ing their performance closer to humans. Our dataset and code are available at: https: github........ cwm/LeoLee/ wtw.oooieee//ireiiionnnn www.rrrrreetooaauuuddaeeeeeee
Introduction
In recent years, Vision-Language Models (VLMs) have made significant strides in general multimodal tasks such as visual recognition and Visual Question Answering (VQA) [1, 49]. This progress has opened up vast potential for various applications, including enhancing visual accessibility for visually impaired individuals [4, 20], supporting decision-making in autonomous systems [50, 32], enabling interactive technologies [39], etc. Despite these advances, VI.Ms still fall short of human capabilities. Humans can intuitively assess whether the available information sufficient to answer a question and seek additional details when necessary [36, 11 [1]. In contrast, VLMs typically tend to provide direct, single-response outputs even when information insufficient to answer the question accurately. This limitation reduces their effectiveness in real-world applications [13]. J. To address this issue, recent studies have explored ways to teach VLMs to assess information sufficiency [42]. These studies aim to have VLMs either provide concrete answers or label questions unanswerable, using benchmark datasets from real user questions like VizWiz [20].
However, significant gap remains in handling mannsweaable cases: deciding what actions to take unanswerable. Humans naturally possesses the ability to seek additional details when faced with unanswerable questions challenge often encountered in real-world VQA tasks due poor image quality, ambiguous questions, loss of context [3, 9]. To
when VL.Ms identify a question
"qqual contribution.
Corresponding author: Igilpin @ ucsc.edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Question: what my
Question: What room &s this?
Question:
2165- don't have to
Somy, I'm not sure Can you please move camera to right?
Sorry, Moving camera seoms helpful. try...
Figure 1: The examples of the Directional Guidance task. The model utilizes self- eeffkknoleegge to distinguish between known and unknown information and provides guidance on where to find more information.
the best of our knowledge, no existing benchmarks focused on "what to do" after the model identifies information insufficiency. This active process of information acquisition, fundamental to human cognition, has not been replicated in VLMs and remains largely unexplored.
To narrow the gap between VLMs and human intelligence, we suggest going beyond improving accuracy on answer generation or merely deciding on information sufficiency. Instead, we focus on enhancing the model's capability provide constructive feedback when encountering unanswerable questions. In response to this challenge, we introduce a novel VQA task aimed at providing Direc- tional Guidance, which aligns with real-world needs, particularly for visually impaired individuals. As indicated in previous studies [9], a common issue is that many images taken by visually impaired users ill fll-fremed. Our task aims guide users on how reframe their images during the interactive VQA process. This task evaluates the model's ability to understand visual direction and determine potential direction to obtain more relevant information.
Moreover, to empower VLM with such guiding capability, we propose an automatic VQA data augmentation framework. This framework begins by prompting a pretrained VLM to filter set of answerable questions from the given VQA dataset. The corresponding images are then perturbed using predefined rules that crop relevant visual information, making it more challenging for the model to answer the questions correctly. Finally, the VLM is fine-tuned using this augmented dataset, with the task of providing Directional Guidance resolving the predefined perturbations. This approach simulates information inadequacy scenarios and holds promising potential for enhancing the model's ability guide users in acquiring relevant information.
To validate the effectiveness of the approach, we contribute a manually labeled test set containing the Directional Guidance for real-world unanswerable datasets with images taken by visually impaired individuals. Our experiments on three popular open-source VLMs show significant improvements in the models' performance the Directional Guidance task after fine-tuning with synthetic training data. Notably, the best-performing model outperforms GPT-4o (CoT) [31] by 3% accuracy score,
The contributions of this study are:
Directional Guidance task: We define a novel VQA task. As shown in Figure 1, the proposed task assesses the model's ability identify the information sufficiency and provide Directional Guidance when needed.
Directional Guidance dataset:
We create a human-labeled test set to benchmark the sendance-providdnng capability of VLMs.
Directional Guidance framework: We propose data-efficient framework for training models on the Directional Guidance task. This framework includes synthetic training data generation and model fine-tuning, which can be generalized with any VQA dataset with grounding information.
2 Related Work
Directional Visual Understanding. Many studies have identified that current VLMS struggle to interpret and understand spatial relationships within an input image, especially on fundamental visual concepts like relative directions [15, 40]. This ability is important for interactive VQA applications

---

like autonomous agents [18, 17], visual navigation, and assistive technologies designed for visually impaired individuals [24, 13]. To enhance VLMs' capability to understand directional relationships. researchers construct extensive training data [44], add assistive visual prompt [47, 30], or include collaborative VL.Ms to communicate and ensemble their decisions [8]. However, these methods often require heavy data collection or introduce additional models. Previous studies have investigated visual learning through simulation [18], but they rely on virtual interactive environments that may not accurately reflect real-world scenarios. Another trend involves generating training data by asking questions about directional relationships in existing images [29, 27], but this also requires additional involvement of advanced models. In our study, aim to improve the model's directional understanding with simple data augmentation methods, using images collected from real users.
Assistive technology for visually impaired individuals. Over the past decade, applications like VizWiz [4] and Be My Eyes [13] have used real-time video connections to enhance visual accessibility. VLMs present a more accessible and responsive solution to satisfy the user's needs, as they can provide immediate responses when given a photo-query pair. However, as noted in [9], visually impaired users often face challenges in capturing clear images. In real-world conditions, many images suffer from quality issues such as blurriness, obstructions, and improper exposure, making them difficult to recognize. These issues often result in divergence in human annotations [9, 3]. Moreover, even when the images are clear, the questions may still be difficult to answer due to the off-framing of the target objects [9]. Addressing these challenges typically requires multiple rounds of queries and adjustments to properly frame the key object. For VLMs, these difficulties may be amplified because their training data typically lack examples of unrecognizable images. Additionally, to align with the multiple adjustment interaction offered by human operators in Be My Eyes applications, VLMs need to offer honest and effective guidance to navigate to target objects.
Self-knowledge. Self-knowledge refers to the model's ability to recognize what is known and unknown [22]. When confronted with unanswerable questions due to ambiguity or insufficient information, VLMs/LLMs often generate hallucinated responses [26, 46]. Previous research has introduced methods to help LLMs understand limitations regarding unknowns [48, 2, 34, 41J. Subse- quent studies, such as [12]. have explored explaining unanswcrability by constructing known and unknown datasets through data augmentation and refining base models with self-curation method. For VLMs. [25] presents a robust visual instruction tuning dataset that includes negative instructions at different semantic levels, i.e. nonexistent object manipulation, existent object manipulation, and knowledge manipulation, all implemented by GPT-4. Although these studies validate the benefits of data augmentation, they have focused generating negative unknown data primarily within the language modality, Instead, our study extends this exploration into multimodal data by incorporating the vision modality.
3 The Cognitive Question: From What's Unknown and Where to Know
To understand how statistical model conceptualizes the world, one effective approach is to draw an analogy to human cogmition. In the meta task in NLP (i.e., Question- -antiion-n as other core NLP tasks can be transformed into QA), human cognitive processes in problem-solving and learning multifaceted, involving not just the retrieval of stored information but also the recognition of one's knowledge boundaries and the strategic acquisition of new knowledge. To simulate these processes, we propose a hierarchical cognitive process pattern comprising three levels:
1. Response Generation (knowing what's known): At the foundational level, the model utilizes its existing knowledge base and basic analysis capabilities to generate responses to queries. This process mirrors the human cognitive function of retrieving known information from memory, akin to recall or recognition tasks in cognitive psychology[28. 35J. It reflects the model's ability to combine available information into coherent answers.
2. Awareness of Knowledge Limits (knowing what's unknown): The second level reflects the model's metacognitive ability to evaluate its own knowledge state, recognizing when it lacks sufficient information to answer a question accurately [14, 36, 37]. This awareness is crucial for intellectual honesty and mirrors the human cognitive process of monitoring and evaluating one's 's understanding and capabilities, a key aspect of metacognition [51, 38, 11].
3. Knowledge Acquiitionn Direction (knowing where to know the unknown): At the most advanced level, the model identifies pathways for acquiring new knowledge when existing
3

---

information is insufficient. This ability to seek out and engage in learning opportunities mirrors the human cognitive strategies for addressing knowledge gaps, such as identifying resources, formulating questions, or modifying learning strategies. It signifies the model's s capacity for self-guided learning and adaptation, similar to strategic learning and problem- solving in human cognition.
As mentioned in Section 1, most existing works on VLMs cognitive questions are focused the first two levels [42, 15, 5], and the third level is mostly under-explored. We argue that the challenges lie in the difficulty of collecting suitable data for benchmark and training data: there few VQA samples that exhibit both awareness of knowledge limits and knowledge acquisition direction. Therefore, in our study, we focus on benchmark dataset curation and training data generation.
4 Method
4.1 Directional Guidance Task
We define our Directional Guidance task as follows: in the context of VQA, given image-question pair < I.Q >, the model M should determine whether the image needs to be reframed. To be specific, if the target object is only partially visible and not sufficient to answer the question, the model should give clear guidance for the reframing direction (left, right, up, or down). Otherwise, the model should inform whether the question is already answerable (no need to change) or remains unanswerable even with potential reframing (none of the other options). This task mirrors real-world scenarios where visually impaired individuals need guidance to position their cameras correctly through many attempts. Although the target object might be only partially visible on each attempt, with continuous adjustments under guidance, the user can always capture better view and finally have a better chance to get the question answered. This task goes beyond simply detecting the ill-framing issue of the image: it assesses whether the framing issue impacts the model's 's ability to answer the specific question posed. For example, reframing may not be necessary if the question can be directly answered with the available visual information even if the image is ill-framed. We regard these guide responses as an additional output that complements the original VQA answering process.
This task exemplifies three levels of the hierarchical cognitive pattern discussed in Section 3. Instead of a binary classification of inswerabl/umaasswerbbe as proposed in [9]. this task emphasizes the model's ability effectively utilize available visual information. It requires the model to assess what is known and determine where to acquire extra information, standing in the transition from unanswerable answerable.
4.2 Directional Guidance Dataset
Benchmark dataset. To evaluate model performance in our task setting, we created a benchmark dataset derived from VizWiz dataset families [20, 7]. The VizWiz dataset consists of real VQA queries collected from visually impaired individuals [20]. From this dataset, we used all the unanswerable samples (1.4k) from the validation set as the training set may have potential leakage issues during the pre-training process of VLMs. We invited 26 human annotators to identify ill-framed photos and label the most promising direction to move the camera, by which the reframing action could potentially help to answer the question (more details are available in Appendix A. After cleaning and re-evaluation, we collected 291 samples where reframing could potentially lead to an answer, and 230 samples unlikely to be answered even with reframing. The rest samples are where the human annotators have disagreements. The details of the data collection are presented in the Supplementary Materials. To ensure balanced distribution in the test set, we randomly selected 300 samples from the iiziizggrounding test set and simulated the case where the current image already has sufficient information to answer the question, under the assumption that the visual evidence could be theoretically grounded in the image. Combining those three groups, we get high-quality Directional Guidance benchmark dataset including 821 samples. Despite the size of the dataset being relatively small, this reflects the inherent challenge of the task, where ill-framed images are rare in standard VQA datasets but commonly seen in real-world scenarios. Furthermore, our dataset's diversity and comprehensiveness make suitable for evaluating model performance on the target task, providing a valuable foundation for future studies.

---

-Too Set Generition
Image:
Cropped Image:
"á dog"
Guidance Label: ease itunhwegged
medel
0. moveto night
hore""
"nnoswwerale"
X
Guidance Label: Reversed
Whatis
Question:
grandir
Figure The training set generation framework.
Training dataset. We propose a data augmentation process simulate the ill-framed samples, instead of collecting ad-hoc images that suit the task. Initially, we take all the training samples from dataset pool - the validation set grooudiign dataset [7] as it includes the visual groundings for each answerable VQA query. With that visual grounding information, manual perturbations have been applied to simulate ill-framing. Specifically, we identify the bounding box surrounding the target object and divide it into 10 zones, horizontally and vertically. We then choose specific zone for cropping, resulting in an image that has some missing information while retaining part of the target object. With series of perturbations, we observe the consistency of the model's response the initial VQA query and capture the cases where an ill- framing issue impacts the question -anttioonnnswering.. As the Wiz is open-ended task, use precision as the evaluation:
Ew (P(W) nT(w)] Ew IP(w)
Precision
P(w) and (ww denote word from the model prediction and from the ground-true answer. Precision calculates how many words in the predictions also appear the ground-truth answer, and we threshold to identify the correctness. Following [19], only non-stop words have been taken into consideration. Figure 2 and algorithm 1 outline the process of generating training data with guidance labels.
Another crucial case in the benchmark test set involves samples that remain unanswered even after adjusting the camera. One more data argumentation technique has been placed: we mismatch the questions and images from the same dataset pool to create new pairs with different semantic information. Most questions in the original dataset pool are generic, as a highly frequent question is GPT-4 enabled argumentation
"What is this?" without semantic information. Correspondingly, helped rephrase the paired question and answer. For example, given an image I; with the question Q: "What this?" and answer A; "laptop." the new question Q will be rephrased "What's the color of this laptop?." Then, we mismatch the Q' with another irrelevant image I, to form a new pair. This augmentation generates complex, real-world queries where straightforward answers are infeasible, compelling models to learn deeper semantic information.
Algorithm 1: Generate synthetic training set with data augmentation
Input: A set of image-question pairs (I, Q)
Output: dataset with perturbed image-question pairs and their corresponding guidance
Initialize dataset D empty Define range of perturbation magnitudes P with types (left, right, up, down)
foreach pair (1, Q) do
if model M predicts (1, Q) correctly then foreach perturbation P in do
T' +- Apply P to
if M still correctly predicts (I',Q) then
- 'leave unchanged'
else
Prevese f- Reverse operation of P - Pravnee
Add ())
5

---

4.3 Experiment settings
Model Selection. To verify the feasibility and effectiveness of our approach for different model architectures and sizes. we analyze the experiments of four mainstream open-source large models with different sizes, including: LLaVA-1.5 [27], InstructBlip [10], GPT-4o [31], and CLIP [33]. First, we benchmark the test set on the LL- InstructBlip, and GPT-4o on the zero-shot setting. A series of prompts has been designed to test their zero-shot performances, serving as our baselines. Next, we generate a training dataset using algorithm and apply LoRA [21] fine-tuning on the open-sourced models. We anticipate the effectiveness of our proposed training framework will be reflected by the improvement of model performance compared with the zero-shot baseline.
Task format. To quantitatively analyze the model's ability to provide guidance, we format the task with basic VQA multiple choice template: "<image>(Oril (Original_Quustiinnn image and answer the question, how should the camera be moved? Leave it unchanged. B. L.Left C.Right. Up Down. F. NoNen of the other options. " Each option reflects the model's decision of Directional Guidance: The leave it unchanged option indicates that the current image contains all the necessary information to answer the question. The four directional options suggest that the relevant object is only partially visible, and further image adjustment needed. The None of the other options implies that moving the camera will not help because the question is inherently unanswerable, i.e. due to the ambiguity, or the relevant object is absent from the current image. We use the Fl score and accuracy as the evaluation metrics and also analyze the confusion matrix of the different options.
To improve the
Zero-shot prompt setting. For the zero-shot baseline, we enhance the basic template with addi- tional instructions and explanations tailored for each model. We designed two prompt settings to accommodate their varying capabilities. The first setting is a single-round query where the model makes predictions from six options directly. The second setting is a two-round prompt, following the Chain-of- Chain-of-Thought [43] process. This two-round prompt decomposes the tasks and works as follows: Initially, we prompt the model to determine if the target object is fully present in the image, partially visible, or if the question is unanswerable. The corresponding options are: leave it unchanged, reframe, and none of the other options. If the model indicates that the target object is only partially visible, we then ask it to decide a specific direction for movement: 1eft, right, up, or down. To ensure reproducibility, we include all prompts we used in Supplementary Materials A.5.
Fine-tune setting. In our training framework, we utilize data augmentation to generate potential samples with guidance labels. We assess the consistency of the model's predictions before and after perturbations and categorize the samples into two groups. Samples where the model fails to predict post-perturbation are considered positive, and their Directional Guidance labels are assigned one of four directions: left, right, up, or down. Conversely, samples where the model maintains correct predictions are labeled as negative, with the Directional Guidance label set to leave it unchanged. Upon analyzing these groups, we observed that negative samples predominated the generated training set. To ensure balanced distribution within the training dataset, under-sampled the negative samples to align with the average count of the four directional categories. We also adjusted the number of None of the other options samples to achieve even distribution across the entire training set.
After generating the training set, we format the new pairs into standardized instruction fine-tuning layout: each sample, comprising I', question >, is supplemented with option choices and instructions. Since the task requires models to respond with a single letter, the prediction process equivalent to a classification task. Following the settings in [27], the loss is only computed on the token for the chosen letter and the < eos > (end of the sentence) token. Also, to prevent the model from memorizing the letter distribution, we randomly shuffle the association between letters and options, ensuring each letter (from A to F) is paired with an option randomly in each training sample. When fine-tuning each model, most training configurations follow the officially suggested settings, and more training details are presented in Supplementary Materials A.2.
None-generative Models. Since the task has been simplified to a classification problem, we also investigate whether non non-genertt model with a simpler architecture could suffice. Accordingly, we add linear probe layer onto CLIP and perform a classification head, using a vision encoder (CLIP. VTT-L-336px) aligned with LLaVA-1.5 and text encoder in the default setting. We concatenate the image feature and the text feature as the input for the classification head. Since the CLIP model can not generate open-ended answers, we use the synthetic training dataset generated by LLaVA-115 13b.

---

Up
Down
Left
Right
Up
Down
14 8%
17.2%
Right
38.5%
EEOSS CHECO
What in ove?
What dasuref
Cer rend the nee THanks.
Foo wondering you could resc what the serees? Thope gota. good pletuer
(4)
(M
(C)
(d)
(c)
Figure 3: The distribution of four directions in our benchmark dataset (a) and examples (b-e). The upper caption is the Directional Guidance label and the lower caption is the original question.
GPT-4o
InstructBlip 7b
LLaVA-1.5
LLaVA-L.5
22
PRP Kr Sown up
Fight belt dogg
4
0 rOrE down
24
0 Fight kift down
10
22
Actual
14
6
16
54
Actual
7
48
Actual
14
17
30
30
Actual
2
0
0
86
2.0
2
5
15
10
44
2
5
47
8
16
35
25
5
0
1
58
22
10
0
11
4
6
236
37
0
7
25
157
102
0
14
65
202
15
0
9
0
2
257
28
x
2
19
194
x
19
37
92
x
23
25
69
106
54
172
down loft right
0
down loft right
down ifft right
0
down left right
100
(al)
(a2)
(a3)
(a4)
Zero-shor hascline
CLIIP with lineur probing
InstructBlip 7b
LLaVA-1.5 Tb
LLaVA-1 5 136
0 right litt down up
14
4
11
10
0 right Nit down up
2.
0
5
right left down up
0
0
0 right 10ft down Up
1
8
5
5
10
Actual
Actual
Actual
Artual
14
24
26
30
20
24
17
18
26
1.0
14
46
17
171
42
31
25
15
205
10
17
17
5
239
13
15
5
245
13
x
37
52
35
79
40
x
39
20
22
64
50
13
11
18
8
72
10B
x
12
14
14
5
35
150
-10P
down loft right
right
down left right
right
(161)
(h2)
(133
(b4)
Figure 4: The heatmaps of the model's prediction. (a1)-(a4) shows the baseline performance under zero-shot setting, and (b1 (b1))(b4) shows the performances of fine-tuned models. 'O' denotes the class leave it unchanged, and 'X' denotes the class none of the other options.
Results
5.1
Directional Guidance benchmark dataset and baseline performance
Fig. 3 (a) shows the distribution of four directions in our benchmark dataset. The horizontal directions are the most common, with left at 38.5% and right at 29.6%. The figure displays four typical samples from each direction. We also identified a frequent scenario where users need to take another photo and attempt different question, as shown in Fig. 3 (e). This pattern reveals common challenge for visually impaired individuals: without continuous guidance, the user and assistant can easily lose the context of the original VQA. These findings emphasize the importance of providing clear, sequential dialogue-based guidance for effectively adjusting the camera position.
As mentioned in Section 4.3, we use different prompt settings for each group of models that suit their capabilities. For the 7b models, we use the two-round prompt because these models benefit from more structured, step-by step-by-stey approach, which helps them handle the task more effectively. In contrast, we tested the I.LaVA-1. 13b and GPT-4o model with a single-round of prompting to see if they were capable of this task. The prediction results are presented in Fig. 4, from (al) (a4). We observe that three open-sourced models (I.L. 7b/13b, and Instructblip 7b) show similar behaviors: these models tend to avoid predicting the reframing cases and mistakenly categorize them
7

---

as either leave it unchanged or none of the other options. With the two-round prompt. LLaVA-1.5 7b and InstructBlip 7b make some correct predictions in the 1eft and right categories. However, the correct and incorrect predictions are nearly balanced, with frequent misclassifications in the opposite direction. For example, the number of true left predictions is equivalent to the number of erroneous 1eft predictions that were intended to be right. For GPT-4o, there are fewer errors in categorizing reframing cases into the wrong categories, and more cases within the reframing category correctly predicted. However, contradictory predictions also occur frequently within the reframing cases, The result demonstrates that all the models are generally incapable of accurately predicting the reframing cases under zero-shot prompt settings. However, every baseline model performs well in the none of the other options category. We present the examples from each model in the Supplementary material A.4 to visualize the model's pretrained capability on the Directional Guidance Task.
5.2 Model's performance after fine-tuning
We present the heat maps of the fine-tuned model predictions in Fig. 4. By comparing the prediction results between the zero-shot baselines and the fine-tuned models, we observe significant and consis- tent improvements in prediction performance, demonstrating the effectiveness and generalizability of our proposed method. Although there is considerable potential to improve the overall accuracy, the fine-tuned models reduce confusion between reframing, leaving it unchanged(O), and none of the other options(X). The fine-tuned models are more likely to provide directional guidance on the reframing cases. Moreover, the predictions within reframing cases show noticeable improvement, as indicated by the clear diagonal line in the heat map. Another interesting finding is the substantial reduction in wrong predictions with opposite directions (c.g., predicting an up case as down). This clarity is meaningful, as it lowers the chance of users receiving conflicting guidance, thereby enhancing safety and efficiency in real-worke applications. Overall, the fine-tuned models reduce errors across all options, showing significant improvement in both cross-category and within reframing predictions.
To evaluate our training framework's s sensitivity to different settings, we conducted groups of com- prehensive experiments. We used three metrics to quantitatively assess the model's performance: overall F1 score, overall Accuracy, and Accuracy on the reframing cases denoted as ACC(F). The metries for the baseline models are presented in Table 1. In some baseline experiments, we found that the zero-shot setting did not always ensure a standard output format. In such cases, we performed post-processing and excluded samples with predictions that did not fall within our options. The total number of excluded samples was fewer than 10, and this only occurred in the zero-shot baseline models. The results, as shown in Table 1, include combination of different settings from two aspects: varying perturbation ranges and the impact of shuffling letters and options in the training data. Regarding the choice to shuffle, we observed that randomly mixing letters and options does not consistently enhance performance. For instance, with perturbation range of 0.1-0.9, the unshuffled approach often outperformed the shuffled version, while with a perturbation range of 0.3-0.7, shuffling generally resulted in inferior performance.
The CLIP with linear probing method achieves comparable performance with the zero-shot perfor- mance of InstructBlip, but it's still not able to provide informative guidance (the accuracy for random choice for a six classification task is 16.6%). This suggests that simple CLIP-based encoders, lacking integration with language model, may not be sufficient for this task. While the task largely relies on the model's perception of salient features, the contextual information within questions is also essential. For instance, the VizWiz dataset includes many generic questions such as 'What is the color of this?' Many of these questions are answerable even though the photos are heavily ill-framed. Since these questions do not concern spatial details, the appropriate guidance is to leave it unchanged. This underlines key distinction between our task and other image quality detection tasks, especially those focusing ill- illfframing solely on image modality.
6 Discussion
In this section, we analyze the effect of different settings, including perturbation range and shuffling operations, on the generation of training data. detailed heatmap of the model's predictions is presented in Supplementary Materials with Figure 6. The perturbation range determines the crop
8

---

Table 1: Model's performance with different settings. Fl and ACC denote the Fl score and accuracy score, respectively. ACC(F) refers to the aceuracy of reframing directions, excluding the categories 'Leave it unchanged' and 'None of the other options. N/A indicates not applicable experiments due to limitations on the model's accessibility or incompatibility with the experiment design.
Shuffling
W/O shuffled options
With shuffled options
Baseline
Perturbation range
0,3-0.7
0.1-0.9
0.3-0.7
0.1-0.9
Metries
F1
ACC
ACC(F)
F1
ACC
ACC(F)
FI
ACC ACCEP)
F1
ACC
ACCIP)
F1
ACC
ACC(F)
CinPalinear probe
0.30
0.31
0.19
0.32
0.32
0.18
NIA
Instractllip7hh
0.41
0.35
0.46
0.47
0:45
0:43
0.42
0:35
0.45
0.46
0.27
0.27
0.31
LLaVAL.-76
0.33
0.57
0.58
0:44
0.55
0.56
0.36
0.52
0.54
0.31
0.42
Model
LLaVA1.5-13B
0.63
0.63
0.54
0.52
0.56
0:57
0.37
0.43
0.53
GPT-4v
0.59
N/A
GPT-4o
0.60
ratios used to generate the training samples, ranging from 0.1 (minimal crop) to 0.9 (maximum crop). We observed that positive Guidance samples tend to cluster at high crop ratios, while lower ratios often correspond to negative samples (where the Guidance is leave it unchanged). Therefore, a range of 0.3-0.7 leads to a more balanced selection of training data, while a range of 0-190.9 provides a more comprehensive and varied dataset. This setting can affect the model's performance due to the balance between the diversity and complexity of the generated training samples, and the trade-off works follows: when the perturbation becomes more severe (e.g., a ratio of 0.9), images are aggressively corrupted. This increases the chance of obtaining positive Guidance samples, as the model is more likely to fail in predicting these heavily perturbed samples, which it could have predicted accurately without perturbation. However, it also results in significant information loss and greater challenges in detecting objects, making a harder sample to learn. Conversely, moderate perturbation ratio results in less aggressive cropping, allowing the model to access more information and better respond the original question. However, this can lead fewer positive Guidance samples, as the perturbation does not sufficiently challenge the predictions. The differences with the shuffling settings could be attributed to the regularization effect, which prevents the model from memorizing fixed patterns and increases training difficulty, especially when training data is scarce. scenarios with less training data, shuffling acts as a form of data augmentation increasing the diversity of training examples and making the model more robust. However, shuffling might add unnecessary complexity to a larger training dataset derived from a wider perturbation range, making it harder for the model to learn effectively. In such cases, the unshuffled approach allows the model to quickly identify and leverage consistent patterns, facilitating faster and more efficient learning processes.
Ablation Study To gain more comprehensive understanding of how different perturbation ranges affect model performance, we conducted a more fine-grained ablation study with LLaVAI 1.b Specifically, evaluated perturbation ranges of 0. 0.1-0.3, 0.3-0.5, 0.5-0.7, and 0.7-0.9, alongside our main experiments of 0.1-0.9 and 0.3-0.7. The results, presented in Table 2, reveal that as the perturbation range increases from 0.1-0.3 0.5-0.7, both overall F1 scores and overall accuracy show substantial improvements, stabilizing around 0.49. However, at the highest perturbation range of 0.7-0.9, we observe a slight decrease the ACC(F) metric, suggesting that overly aggressive perturbations may introduce excessive complexity, hindering the model's ability to accurately identify relevant objects. Notably, the broader range of 1-0. achieves the highest overall Fl and accuracy scores, suggesting that a wide perturbation range strikes an effective balance between data diversity and sample complexity. Additionally, all perturbation ranges demonstrate improvements ACC(F), with enhanced reframing direction performance as perturbation increases, except at the highest range. These findings support our initial discussion by emphasizing the trade-off between data diversity and the complexity of perturbed samples. Future work could explore optimized strategies for selecting perturbation ranges, potentially employing dynamic or adaptive methods to further improve model performance based on specific dataset characteristics.

---

Table 2: Model performance under different perturbation ranges with 1.5 7b.
Perturbation Range
Overall
Overall Accurncy
ACC(F)
0-1-0.3
0.19
0.24
0.31
0.5-0.7
0.49
0.49
0.43
0.7-0.9
0:50
0.49
0.40
Limitation and Future Work
In this study, we focused specifically on guiding image reframing directions as a proof of concept. Although reframing is one of the most common needs when assisting visually impaired individuals, some other aspects that impact the VQA process could also be explored, such orientation, exposure, and focus. Our data augmentation framework can be extended to these aspects and generate training data in a similar way, and we plan to explore this in future work. Second, the directional guidance has been simplified to a classification task on directions, which may not fully capture the complexity of real-world scenarios. For example, effective reframing might require combining multiple direc- tions-such as moving both up and left-or even zooming out. A more informative guidance in practice would also consider additional parameters like the magnitude of the reframing action. Those complexities can confuse the model, leading to inaccurate evaluations, To enhance clarity and reduce ambiguity in our benchmark dataset, we included only cases that received consistent annotations from multiple annotators, which resulted in a limited size in our benchmark test set. Additionally, our preliminary experiments are designed to validate the effectiveness of our proposed framework rather than to maximize the model's performance. Consequently, the current method cannot fully guarantee the reliability the model's prediction, and still requires cautious deployment in high- risk scenarios. Moving forward, we aim to refine the task design and data generation framework, adapting more effectively to complex, real-world applications. In addition, theoretically, our guidance framework can be extended to more general and quantitative scenarios. By simulating spatial drift, we can customize the ratio of drift and produce synthetic training data with quantitative values. This potential extension would allow models to identify not just the direction but also the extent of camera movement required. This makes such quantitative guidance particularly meaningful in applications such robotics, where precise tracking of target objects is crucial, for example, in calculating the ground truth for the extent of movement needed [16]. Furthermore, we expect our unsupervised data generation framework to alleviate the pressing data needs of studies exploring LLM or VLM for spatial or temporal reasoning tasks [6, 45, 23].
8 Conclusion
In this paper. we introduced a novel task and benchmark dataset within the context of Visual Question Answering (VQA) aimed at improving the self- ceff-knoweedge of Vision-Language Models (VLMs). Our task specifically evaluates how well VLMs can assess the sufficiency of visual information and determine the necessary actions to reframe an image to obtain additional information. To address the challenge of limited training data, we proposed an automated framework that generates synthetic data by simulating unanswerable scenarios through perturbations applied to answerable cases. Our results show that current nigh-performing VLMs, including LLaVA and GPT-4o, struggle with this task, revealing gap in their ability to handle incomplete or ambiguous visual inputs. However, when fine-tuned with the synthetic training data generated by our framework, the models significantly outperformed the zero-shot baseline on real-world data.
This study highlights the importance of self- eelfeknowledge in VLMs, particularly their ability rec- ognize the boundaries of available information and take appropriate actions when confronted with incomplete or misleading data. By mimicking human cognitive processes, our approach presents a promising solution for enhancing models' eeoleenoweedee and robustness, especially in real- real-world applications that require accurate and adaptive responses. This is particularly relevant for assistive technologies, such as those designed for visually impaired individuals, where providing timely and effective guidance is essential. As VLMs continue to evolve, the ability to recognize their knowl- edge boundaries and make informed decisions will be critical for their successful deployment in dynamic environments. Future work can explore additional strategies for refining this seff-knowledge, potentially leading more robust models capable of learning in complex, uncertain scenarios.
10

---

Acknowledgment
We sincerely appreciate all collaborators who contributed to this project for their invaluable insights and support. Special thanks to our human labelers for their diligent efforts in creating the benchmark dataset. We are also grateful to Prof. David Lee and Prof. James Davis for their expert guidance, constructive feedback, and encouragement throughout this research.
We thank the VizWiz community for providing an inspiring platform that significantly influenced our work. We acknowledge the use of the VizWiz VQA and VizWiz Grounding datasets(both under CC-BY 4.0)). Our model development was based on LA (Apache-2.0 license) and InstructBLIP (in compliance with the license terms of LLaMA and Vicuna).
This material is based upon work supported by the Air Force Office of Scientific Research under award number 219550-24-1-0111
References
[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015.
[2] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2023.
[3] Nilavra Bhattacharya, Qing Li, and Danna Gurari. Why does a visual question have different answers? In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4271 4280, 2019.
[4] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers visual questions. In Proceedings of the 23nd annual ACM symposium User interface software and technology, pages 333-342, 2010.
[5] Yuhang Cao, Pan Zhang, Xiaoyi Dong, Dahua Lin, and Jiaqi Wang. Dualfocus: Integrat- ing macro and micro perspectives in multi-modal large language models. arXiv preprint arXiv: 14767, 2024.
[6] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455-14465, 2024.
[7] Chongyan Chen, Samreen Anjum, and Danna Gurari. Grounding answers for visual questions asked by visually impaired people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19098-19107, June 2022.
[8] Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models visual reasoning coordinators. Advances in Neural Information Processing Systems, 36, 2024.
[9] Min Chiu, Yinan Zhao, and Danna Gurari. Assessing image quality issues for real-world problems. In proceedings of the IEFF/CVF conference on computer vision and pattern recogni- tion, pages 3646-3656, 2020.
[10] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale N Fung, and Steven Hoi. Instructblip: Towards general purpose lision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[11] Andreas Demetriou and Smaragda Kazi. Self-awareness in g (with processing efficiency and reasoning). Intelligence, 34(3):977331,, 2006.

---

[12] Yang Deng, Yong Zhao, Moxin Li, See- KeonKiong Ng, and Tat-Seng Chua. Gotcha! don't trick me with unanswerable questions! self-aligning large language models for responding unknown questions. arXiv preprint arXiv:2402. 15062, 2024.
[13] Be My Eyes. Introducing be my eyes virtual volunteer, 2023. Accessed: 2024-05-21.
[14] John Flavell. Metacognition and cognitive monitoring: A new area of cognitive- developmental inquiry. American psychologist, 34(10);906, 1979.
[15] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arviv preprint 2rXiv:2404.13390 2024.
[16] Yunhao Ge, Yihe Tang, Jiashu Xu, Cem Gokmen, Chengshu Li, Wensi Ai, Benjamin Jose Martinez, Arman Aydin, Mona Anvari, Ayush K Chakravarthy, et al. Behavior vision suite: Customizable dataset generation via simulation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 22401 -24122 2024.
[17] Akshay Gopalkrishnan, Ross Greer, and Mohan Trivedi. framee lightweight & effi- cient vision-language models for question answering in autonomous driving. arXiv preprint arXiv:2403. 19838, 2024.
[18] Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering interactive environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4089-4096 2018.
[19] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[20] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608-3617, 2018
[21] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09668 2021.
[22] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Batfield-Dodds Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint 2xXiv:220.005221, 2022.
[23] Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, and Hyunwoo J Kim. Large language models are temporal and causal reasoners for video question answering. arXiv preprint arXiv:2310.15744 2023.
[24] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635-651, 2023.
[25] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigat- ing ballucination in large multi-modal models via robust instruction tuning. The Twelfth International Conference Learning Representations, 2023.
[26] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vasion-language models. arXiv preprint arXiv:2402.00255. 2024.
[27] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[28] George Mandler. Recognizing: The judgment of previous occurrence. Psychological Review, 87:252-271, 1980.
12

---

[29] Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens Van Der Maaten, Learning by asking questions. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 1 11-20, 2018.
[30] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402. 07872, 2024.
[31] OpenAl. Gpt-4 technical report. arXiv preprint urXiv:2303.05774. 2023.
[32] YungYeon Park, MinJae Lee, JiHyuk Kang, Hahyeon Choi, Yoonah Park, Juhwan Cho, Adam Lee, and DongKyu Kim. Vlaad: Vision and language assistant for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 980-987, 2024.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 1748-8766 PMLR, 2021.
[34] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable ques- tions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 784-789, 2018.
[35] Henry L. Roediger and Andrew C. Butler. The critical role of retrieval practice in long-term retention. Trends Cognitive Sciences, 15:20-27, 2011.
[36] Gregory Schraw and David Moshman. Metacognitive theories. Educational psychology review, 7:351-371, 1995.
[37] Nick Shannon and Bruno Frischherz. Metathinking: The Art and Practice of Transformational Thinking. Management for Professionals. Springer International Publishing, Cham, Switzerland, I st edition, 2020.
[38] Zhen Tan, Jie Peng, Tianlong Chen, and Huan Liu. Tuning-free accounaablc intervention for Ilm deployment-a metacognitive approach. arXiv preprint arXiv:2403.05663, 2024.
[39] Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint
arXiv:2312.11805, 2023.
[40] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. arXiv preprint arXiv:2401.06209 2024.
[41] Yile Wang, Peng Li, Maosong Sun, and Yang Liu. Self-knowledge guided retrieval augmentation for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10303-10315, Singapore, December 2023. Association for Computational Linguistics.
[42] Yuhao Wang, Yusheng Liao, Heyang Liu, Hongcheng Liu, Yu Wang, and Yanfeng Wang. Mm- sap: A comprehensive benchmark for assessing self-awareness of multimodal large language models in perception. arXiv preprint arXiv:2401.07529, 2024.
[43] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Sfain-of-thoughh prompting clicits reasoning in large language models. 2022.
Advances in neural information processing systems,
[44] Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, et al. Q-instruct: Improving low-level visual abilities for multi-modality foundation models. arXiv preprint arXiv:231 2023.
[45] Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. Large language models can learn temporal reasoning. arXiv preprint N4N0:22401.08853,2222
13

---

[46] Diji Yang, Kezhen Chen, Jinmeng Rao, Xiaoyuan Guo, Yawen Zhang, Jie Yang, and Yi Zhang. Tackling vision language tasks through learning inner monologues. In Proceedings of the AAAI Conference Artificial Intelligence, pages 19350-19358, 2024.
[47] Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting. Advances in Neural Information Processing Systems, 36, 2024.
[48] Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuan-Jing Huang. Do large language models know what they don't know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653-8665, 2023.
[49] Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and answering binary visual questions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[50] Xingcheng Zhou, Mingyu I Liu, Bare Luka Zagar, Ekim Yurtsever, and Alois C Knoll. Vision language models in autonomous driving and intelligent transportation systems. arXiv preprint 1:X1::2310.14414. 2023.
[51] Yujia Zhou, Zheng Liu, Jiajie Jin, Jian- Yun Nie, and Zhicheng Dou. Metacognitive retrieval- augmented large language models. arXiv preprint 16X1v:2402.11666, 2024.
14

---

A Appendix supplementary material
A.1 Benchmark Dataset Collection
We invited 26 human annotators to label the 1.4k umanswerable samples extracted from the validation set of VizWiz VQA dataset[20]. We divided the dataset evenly, assigning each annotator segment to work on. Each sample has an image and corresponding question from the original dataset. We include several sub-tasks for each VQA question categorize how the images could be improved to better answer the question: annotators were required to choose options from the following categories Reframing, Other Actions, or Way to Answer. In the Reframing category, annotators specified the direction - left, right, up, down-for a reframing action that might reveal an answer. The 'Other Actions' includes the actions that are beyond the previous four actions, such as zoom in/out, rotation, and adjusting exposure. The 'No Way to Answer' category is used to indicate cases where there is unlikely to yield additional information by taking any actions. We also asked the annotators summarize their selected options with one sentence, which can be open-ended and qualitative comments. This is used for sanity checks and helps our further cleaning. After the initial round of annotations, engaged four additional annotators (validators) to review the labels, noting any errors or disagreements. These validators conducted two rounds evaluation and discussion before finalizing the benchmark dataset. After the validation, select the Reframing and 'No Way to Answer' categories to join our benchmark dataset. The instructions given to annotators are:
Task Overview
Thank you for joining this task! In Seaell-uusstoonnnstsnee (VQA), some image-question pairs are marked "unanswerable" due to insufficient information in the image. Our goal is to determine if specific camera adjustments or other actions could potentially make these questions answerable.
For each image-question pair, your task is to identify if any adjustments or guidance could potentially make the question answerable. Please select the most applicable option from the drop-down cells and provide a brief explanation in the summary column. Below a description of each option.
Reframing: Choose "Left, Right, Up". or "Down" if moving the camera a specific direction could reveal information answer the question. Choose "Leave it unchanged" if the current image already contains all the information needed to answer the question.
Other Actions: Select "Zoom in/out, Rotate", or "Adjust exposure" if these camera settings might improve visibility or clarify the image content.
Choose this option if no adjustment would help, such cases with lost context, ambiguity, or obstructions.
No Way to Answer:
Summary Column:
In the summary column, provide one-sentence explanation of your choice to clarify the reasoning behind your selection.
Table 3: Instruction to annotators.
15

---

Image
Queetton
Summary
Retraming
Other Actions
No Way Answer
Braised Chiclan Almonds
Qusstian how Kong de look thés the micoowsvwe
Figure 5: A screenshot of the annotation work.
A.2 Training details
We experiments Ubuntu 22.04.4 LTS system equipped with 4*NVIDIA RTX A6000 GPU and AMD Ryzen 24-Cores CPU.
The training details for LLaVA models follow the default settings in [27], with training epoch of 3.
The training details for InstructBlip-77 are shown follows:
Super-parameters
Value
learning rate
5e-5
LoRA
128
LoRA alpha
256
batch size
training epoch
5
16

---

A.3
The model's performance under different settings
TLaVA-Tb
0
right 08 cOWN UP
27
0
20
6
2
10F
39
24
Actia
144
10
50
20
18
16
77
2
10
rgge
20
13
161
243
197
20
22
16
112
100
23
39
93
StatfTel
down lom
right
up
down eft
right
UD
down
rIght
0 right lift down up
15
up
166
13
107
11
16
D right leth
30
30
Actua
100
47
19
Actual
56
15
23
JC 9.1.19
107
13
280
82
12
260
17
241
12
123
down right
riGht
down right
2
0
Actial night (eft down up
29
39
2
7
0.3-03
Twch
33
34
15
37
10
36
19
60
1
37
101
65
17
182
0
29
33
223
51
20
176
38
X
right
up down eft
right
Actual 0 right Wif 0own Up
Hp
rgtht dreen
28
5
17
13
11
4
10
9
21
36
33
3
30
20
12
13
26
12
25
18
26
101
245
22
64
18
72
100
35
150
right
down =ft
right
doan
right
107
37
19
0 right (2f Gown Up
17
29
13
weth
48
49
Actial
14
30
86
23
25
757
102
0
14
65
202
15
257
28
100
172
right
up
right
right
Actual right left dieen up
15
14
37
4
19
17
61
30
Artual Hight
3.9
14
16
54
31
277
14
31
238
37
170
194
up down
up
right
GFI--yy
C271-40
Figure 6: The heatmap of model's predictions under different settings. The denotes the perturbation ranges we applied when generating synthetic data.
17

---

A.4
Examples of VLMs performances with zero-shot settings
LLaVA-1.5-78
Leave unchanged
Question:
temperature?
current
degraas
Left
GPT-4v
Leave unchanged
sorry, image provided blury accurataly read tamperatura on the thormostat If you can provide clearer image, would happy to help you with the information displeyai.
options
The surrent temperature 70 degrees Fahrenhelt.
GPT-4o
Left image provided
InstructBlip-7b
Leeve unchanged
display temperature. making impossible read the lomperature, Piease provide dizarer mage the displey showing the temperature.
fully visible,
70
LLaVA-1.5-76
options
What is the captcha show?
Question:
The captcha shows ai keyboard.
Left
GPT-vv
uncharged
Sorry. can't assist with that request.
LLaVA-1.5-130
None the other options
The captcha shows red square.
GPT-4
other options
image
displaying
typo
legee-eeeeeeee used computing determino no whother or the user human. However, the text the nol dder enough read. cannot determine the spezific displayed
InstructBlip-7b
None the other options
CAPTCHA
LLaVA-1.5-70
Left
Could you me who this to and please?
Question:
image features enowman hokling ladder, with bird porched on the ladder. The snowman is standing next bird feeder. The siane depicted postcard, which addressed "Burington, postmarked
Right
Down
I'm sorry. but can't assist with identflinn meking out the envelope. you something else, feel free
fast
LLaVA-1.5-13b Leave unchanged
GPT-40
None other
envelope is addressed "Lee"" postmarked from
sorry, can't idemniy the sender recipient the image provided image shows part envelope snowman ilustrsion and some handwitten tnxt, but the narres and addresses are not fully visible. you could provide deaarer image of the full address and rames, might be able help with more details.
InstructBlip-7b
Right
no
Figure 7: The examples of the VLMs' zero-shot performance our Directional Guidance benchmark dataset.
18

---

LLaVA-1.5-70
options
What kind of cologno this?
Question:
The image does not depict cologne bottle any sultasse, laptop. focus the badroom setting and not on any collogne or scent-relsted items.
objects.
Right
GPT-v
Right
I'm somy, can't provide sceatiic product
bottle
or brand, making imposuible to delerm mike what. kind of colagre & you havaa a diearer imsuge more informetion colegge, migght you
LLaVA-1.5-130 None of the other options
image wooden
suikaas and a laptop it There no cologne visible in the image.
GPT-4
Right
The image does not provide a dlear view of the cologne bottle making it td ideniiyy the brand type of colognn. you can provide clearer image or more disails about the bottl, such as any visiblle taxt distinctive features, would be easter to help identily callggee
InstructBlip-7b
Leeve unchenged
colognne
LLaVA-1.5-7b
Leave it unchanged
This a Tony Pacco's chill seuce. louul like to know if it's the hot the mild?
Question:
The image shows can of Tony Pacco's chili sauce, which is setcch
Up
GPT-4v
Up
BESUUCH
The prowided tibrry,
ciaatty
right indicaae
LLaVA-1.5-13b
None of the other options
spoc lubed calor cudi sy pechepppng
image can Tony Pacco's sauce, does not provide enough information to determine whether it hot or mild. The labal on the can only indicattes that is chill souce, but the specific heat level cannot be determined from the alone.
dicombil this inagg Yournoop wontto chock
rore eliont any intormaaoooiin
lingue
GPT-4o
The image provident doen not clearyy ahoue whetter thi informalion would indlouled lubed, neir product hown
peomener location
nocomesund look og any lest or symboote et vaille, wont theck the eig pedianss lect addidonal descripionss itbu..
InstructBlip-7b
None of the other options
LLaVA-1.5-7b
Leave it unchanged
Question: What yogurt
This is yogurt brand called "YYoplatt"
Up
GPT-4v
Down
LLaVA-1.5-13b
None the other
somy. image you've provided
bluny
yogurt contalner details logos, migght
yogurt yogurt
image any
assist you
card plugged the yogurt container. It not possible to determine the specific brand or typo of yogurt from the image alone.
GPT-4o
Up
The image is too bluryy clearly identify the brand type of yogurt. The text on contaner is legitle. you can provide clearer image or more detalls, might be able help better.
InstructBlip-7b
unchanged
strawberry
Figure 8: The examples of the VLMs' zero-shot performance our Directional Guidance benchmark dataset.
19

---

LLaVA-1.5-70
Leave uuccanned
Question: What this?
A yellow tag with the number 96 16 placed sheet.
Down
GPT-4v
Right
diapem daltro, Moweree
LLaVA-1.5-13b
Leave unchanged
show desee up of Morio surtis poballty a bullion the
image
bed
lookk tuu ououilor
comforter a yellow measuring tape
which might part of packaging a product could be produst code or idennfffrr The ovirail connat and rest scare are not dlear from
placed on top it. The measuring tape is lying on the bed, possiblly indicaling that someone measuring the size of bed the
GPT-4o
Down
InstructBlip-7b
unchanged
packaging thelpher ipttr"U" which Bhaly site (Medium)) packuging also
label
"ourear prottd on The oe the amage dark and ve part (lappoas to o souut poukaging. plourily cldb og accousooy..
LLaVA-1.5-7b
Right
What type of medicine is this?
Question:
with image slow roothbrushhs vorlous
loothpashh The contaier is placed on a tatie, and them in remore contal eeatt.. The med callons Booly
Down
GPT-4v
Down
control suggeets
I'm serry, but cannot provide specific informaiion about medicatirn baseed on images. It's importent consult healthcare profersionel refer the label the medication for accurate information about med cine.
that the bable en git be in living mom or aamilar space where people relex and watch
LLaVA-1.5-138
Leave unchanged
The image shows a bottle medicnee. specificlly a bottle pills, sttting
GPT-4o
Down
table. The pills white and blue color.
However, without more information about the specitic markings or labels on the
Image clear enough latel
bottlo, is not possible to dotormine the exact type of medicnn it contains.
distinuaahing fisalansa that be used to defermine the type medicino if you need know wit bype of med cane il bs. necormmend checking the label on the bottl or consulting heae professional.
InstructBlip-76
oplions
vitamin
LLaVA-1.5-70
Right
What medication?
Question:
mecication
Leave it unchanged
GPT-4v
Leave uncharged
to acetaminophenn
symptoms.
The rediciion in the image-ix "Atel Sirus usad to melinen epipptonss anoocianed common
which analgesic antipyretic, dextromethorphaii antitusiiv, medication designed to cold symptoms, such congestion, sore throet, headache,
topically
intections, sa ranal headaghes. sches and saine. The PPud" may indicnn additional scthe digreciant or stronger fermutation, Alwyys read the lstll and use meddications es dtricled
LLaVA-1.5 1.5-13
unchanged
Advil
GPT-40
Leave it unchengged
The medication the image white Advil tablot, which pain raliovar inllammatory medication.
Sinus Plus. " iss used rellowe symploos ass00 slud with colds and ainus do ripostonn. This
on typically contains Ruprolon infor ponudoosteddin decongestoee gestant, Alway follow dosage inetrctions the sbet or as
InstructBlip-7b
Leave It unchanged
vitamin
Figure 9: The examples of the VLMs' zero-shot performance on our Directional Guidance benchmark dataset.
20

---

Right
Which number do need dial emergencies?
Question:
To dial for emergencles, should use the number 555.
Leave it unchanged
GPT-4v
uncharged
the image provided indizatee that for emergency, you ahould dial 555. However, please be aware that this number not stancard emergency n. number. In many countries the slandard emergency number bers are difirentt such Stakes and Cannala
LLaVA-1.8-130
Leave
To dial for emergennces, you need to dial 555.
United
Europase
emergency cii
Unionn, spacifie
Always your emergenccy
use
InstructBlip-7b
fullow instructions thore.
911
GPT-4o
Leave
For emangencies, you 665.
LLaVA-1.5-7b
None the other options
Question: What color my dog?
Blue
the other options
None the options
I'm sorry. but the image you've provided doos not show dog, appears dosseuu photo of porson wearing denim gamment possibbl jacket or shirt. If have
LLaVA-1.5-13b
None of the other options
dog
of dog you'd to share, happy help with queedions about
GPT-4o
None of the other options
The image you proviteed
show
InstructBlip-Tb
Leave unchanged
dog. It appoars be close-up of person wearing tlue denim shirt. Please provide an image of your you like
identity
LLaVA-1.5-70
options
Question:
The remaining belance of your account
What remaining account?
None of the other options
None of the other options GPT-4v
I'm mie auch as checking your noceutt tokence ondilionllyy the image you provided doss not babenne
LLaVA-1.5-133
sorry. cannol provide remaining balance your accouni as am an Al language model and do not have information,
discemtioo catadl check your accsunt betarza, plesen contact your bank fruencial cirecly,
personal financiaal your balance, you should check your benk statement or contact your bank directly.
account infarmation secure do shale personal dotall onlino with unve rilidd sources.
GPT-4o None of the other options
image you provided Miurry and axaastance with account vou, please proeido dioner image or check socount through
InstructBlip-7b
None of the other options
app.
Figure 10: The examples of the VLMs' zero-shot performance on our Directional Guidance bench- mark dataset.
21

---

A.5
Prompts used for baselines
INSTRUCTBLIP-7E: TWO-ROUND PROMPT
Round
""Image question
Image: <inuge>
Question: [QUESTION]
ussistant helps adjust options are as follows:
image answer given question. Given image question pair, your
"itaming" the given image contains partial needed information and for beller answering the given question.
certain direction gather more information
B: 'leave it unchanged" - if the image contains enough information to answer the given question, or if the given question is already answered, this option.
"none of the other options' the previous options don't 't fit, or the given question not related to the given image, or the given question cannot be unswrred, choose this option.
If the given question unrelated the given image, select C: the other options".
way
given image answering given question,
of ihe options".
Please only output the selected option for the given image and question pair.
A short answer in the question
Round 2
"'Image question
Image: <imuge>
Question: [QUESTION]"
the previous sctting, the framing. which move to the correspondiag direction to better answer the question.
image part of needed information for answering.
You'te assistant who helps the camera in direction to better answer the question. Given the image and question pair, options are us follows:
D: 'left' - the image is truncated on the left and the camera framing should be moved left to better answer the question based on the given image, choose option.
'right" image truncated right camera framing should right answer question given image, choose this option.
'up'- the image truncated top and framing should be moved up better the question based the given image, choose this option.
G: 'down' if the image is truncated on the batton and the camera framing should be moved down better answer the question based on the given image, this option.
Please only output the selected option for direction.
A shoet answer to the question is:
Table 4: Prompt for InstructBlip-7b.
22

---

GPT4: SINGLE- PROMPT
You are an assistive technology specializing in visual question answering, i.c. the task of providing natural language answer question about given image. To better answer the question (QUESTION) based on the given image. please choose one of the six options (A. 'Leave it unchanged', B. 'Up'. 'Left', D. 'Right', E. 'Down', F. "None the other options') the camera framing. The definitions of each of the options are given below:
A: 'Leave unchamged" - The question can be answered bused on the given image without the need for changing camera framing and there is visible complete ohject the image to answer question. The image clear shows the object question without any truncation need for reframing. The cutire objoct is visible and identifiable.
'None the other options' The question cannot wnswered based the given image, even with change in frumning, the question seems to be unrelated to the content of the image provided, or the question is incomplete or the question is unrelated to the image or there nu visible text on the imuge tu answer the question.
the answer to question. ie., visible text or object is partially visible in the image or the specific text content not clear and quality of the image or the piece of text partially obscured, and the necessary details answer the question are not discemible, please choose one of the below camera framinn accordingly.
angle
C: 'Up" The camera framing should be moved up to better answer the question based on the given image.
D: 'I.efi" - The camera framing should he moved left to hetter answer the question based on the given image.
E: 'Right" The camera framing should be moved right better answer the question based on the given image.
F:: "Down' The camera framing should he moved down to hetter the question based the given image.
Output format required is:
Option and its value.
GPT4: TWO-ROUND PROMPT
Roand
You are an assistive technology specializing in visual question answering. Le.. the task of providing a natural language answer to question about given image, To beller it unchanged', B. 'None of the other options', C. 'Move camera') ") on the camem framing. The definitions of ench of the options are given helow:
the question (QUESTION) based the image, please chouse
given options (A. 'Leave
A: 'Leave unchanged - The question can be answered bused on the given image without the need for changing camera framing and there is visible or complete object the image to answer question. The image is clear shows the object question without any truncation need for reframing. The cutire objoct is visiblc and identifiable.
B: 'None the other options' The question question seams to he unrelated to the content of the image provided, or the question is incomplete or the question is unrelated to the image or there no visible on the image to answer the question.
answered based the given image, even with change in camera frumning, the
'Move camera' If answer the question i.c., visible text or object partially visible in image or specific text content is not clear due to the angle and quality of the image the piece of text partially obscured, and the necessary details answer the question not discernible.
Ourput format required is:
Option and its value,
Reason for choosing that option, option A 'Leave unchanged" selected, give answer question based on the given image.
Round 2
You assistive technology specializing visual question answering. the of providing a natural language answer question about a given image, To better answer the question (QUESTION) based on the given image or previous context (RESULT). please choose one of the given four options (A. 'Up'. B. 'Left', C. 'Down', D. 'Right') on the camera framing. The definitions of each of the options are given below:
the answer to the question i... visible text or object is partially visible in the image or the specific text content is not clear duee to the angle and quality of the image or the piece of text is partially obscured, and the necessary details one of the below camera framing accordingly.
the question are discernible, please choose
A: 'Up' The camera framing should be moved up to better answer the question based on the given image.
"Left'
The framing should be moved left better answer the question based on the given image.
'Down'
The camera framing should moved down to better
question based given image.
'Right" camera framing should be moved right to better answer question based given image.
Output format required is:
Option and its value,
Reason for choosing that option.
Table 5: Prompt for GPT4.
23

---

1L-13-I.--1I:: SINGLE-ROUND PROMPT
<image>
[QUESTION]
To improve the image and answer the above question, how should the framing should be changed in order to answer the question? Please seloct the most suitable ways adjust the framing based following description:
Select "none of other options' if is not related the image the question has grammar ervor. If the Question sentence instead of real question, seloct "none of the other options'. :
correct way to change framing for answering the ahove question, which includes: the question
Select 'leave it unchanged if the above question be answered without adjusting the framing and the question has related with the image, otherwise choose 'oone of the other options'. the question neither 'nune
other options 'leave unchanged', and image has part of the needed information answering and the camera can move the corresponding direction to better answer the question, choose the framning option which you helieve is the most suitable in order answer the question above.
Options:
left up down right
Please double check to make sure that you ure sure about your answer und make sure that you give same priorities to each option, and give me the option only.
ASSISTANT:
LLaVA-1.57D TWO-ROUND PROMPT
Round 1
USER: <image>
(QUESTION)
In order to improve the image answer the given question, please choose which entegory this question-image pair belongs to and the priorities each option are the same: 'none of the other options' or 'Iraming' or 'leave it unchanged", The definitions are given below:
Select 'none of the other options' there no correct direction to move the camera to answer the ahove question, which includes: the question unrelated the image, the question has grammar error.
Select 'framing' if the image has part of the needed information for answering and the camera can move to the corresponding direction to better answer the question.
Seleet 'leave unchanged" the image contains cnough information for answering the above question without moving the camera better answer the question.
the question is unrelated the image, select "none of the other options'.
there is no answering the question, select 'none of the other options".
Please answer the selected category above and double check the answer. Make sure that you choose 'none of the other options" and 'Traming' and 'leave it unchanged' based on their definitions and priocity of these options are the same.
ASSISTANT:
Ourput format required is:
Option and its value,
Reason for choosing that option,
option A 'Leave unchanged" selected, give answer question based on the given image.
Round
USER: <image>
(QUESTION)
the previous setting, the result is 'framing" which means that the image has part of the needed information for answering, and the camera can the corresponding direction better answer the question. Please choose the suitable the four options for moving camera for better answering the question by the directions: left, right, up, down. The definitions are given below:
left if the image is truncated on left and the camera framing should he moved left to better answer the question based on the given image.
right if the image truncated right and the camera framing should be moved right to better answer the question based the given image.
up the imago truncated on top and the framing should be moved up better answer the question based the given image.
down if image is iruncated down cameera frumning should moved down better answer the question given image.
Answer the selected direction only.
ASSISTANT:
Table 6: Prompt for LLaVA-1.5.
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction of our paper, we outline three key claims: First, we identify a significant gap between how humans and VLMs handle situations where there isn't enough visual information to answer question. Humans look for more information to help them answer, while VLMs often either make up an answer or simply say it is unanswerable. Second, we present benchmark dataset that clearly shows the gap. Third, we demonstrate the effectiveness our proposed novel framework that's ' designed to mitigate the gap. The framework effectiveness uses data augmentation to mimic the human cognitive process, thereby improving the VLMs' ability to handle insufficient information. Our experimental results and contributions align with these claims and also demonstrate the generalizability.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We analyzed the limitations in section 7, We presented the limitations of our task setting and the sensitivity of the proposed method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
25

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We don't have theoretical results. We are presenting the empirical results.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We present the steps of collecting the benchmark dataset in Supplementary Materials A. We also illustrate the details of our proposed methodology with Algorithm 1. We present the models we used in Section 4.3-Model Selection, and the training details are in Section 4.3-Fine-tune setting and Supplementary Materials.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
26

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We released the code and full benchmark dataset through our project page (link).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //iip. https/cccccbcc///ceece.eec.cclct gwwgge/dde/e/ee
httpa//guideeeeeeeuuuiiieeeeeee Sttp://ooooSSSeiiiiiiiiiaa www.summiissioooooocccoooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings are presented in Section 4.3 and Supplementary Material Section A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
27

---

Answer: [No]
Justification:
We didn't include the statistical significance analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We present the computer resources in Supplementary Materials Section A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eerrp...eeeee
Answer: [Yes]
Justification: The research presented in the paper conforms the NeurlPS Code of Ethics in all respects
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
28

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we discussed the potential socictal impacts of our work in the introduction part. We also analyzed the potential negative societal impacts in Section 7, pointing out the limitations regarding reliability and the risk of the model providing erroncous information.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We introduced the double verification process of collecting benchmark dataset in the Supplementary Materials (Section A.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
29

---

Justification:
The dataset and code package we used are properly credited. We noted and followed their license and terms use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithee thoooee com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We released the new assets and documents through the project url.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We present the human annotation instructions and screenshot Supplementary material A.I. In our annotation process, all participants received fair compensation based on the average market rate for similar studies.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
30

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Our research is pending IRB exemption.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "hFvgZRr3Cw", "reviewer": "Reviewer_eg4t", "summary": "This paper evaluates and expands the self-reflection capabilities of multiple MLLMs, so that MLLMs can actively evaluate the adequacy of given query information and give suggestions on how to find more reliable information. This paper proposes a hierarchical cognitive process pattern theory and creates a manually annotated dataset for evaluation.", "questions": "Evaluate the difference and experimental effect between this method and the large model self-reflection method.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Methods for active reflection and evaluation of given information are meaningful for enhancing the quality of MLLM responses.", "weaknesses": "1. The rationality of the proposed hierarchical cognitive process pattern, which comprises three levels, merits validation. First, are these three levels both necessary and sufficient for the cognitive process? If the issues at these three levels are addressed, can we then resolve the problems associated with the MLLM cognitive process? Second, does the evaluation benchmark and the ability to expand MLLM, as presented in this paper, align with these three levels?\n\n2. The method proposed in this paper is fundamentally similar to the MLLM self-reflection method. Can it be directly implemented through self-reflection without requiring additional training?\n\n3. A comparable method to the one presented in this paper is DualFocus [1], which not only actively evaluates image quality but also performs more sophisticated active grounding.\n\n4. Lastly, the evaluation dataset used in this paper is relatively small, which makes the evaluation results less convincing.\n\n[1]Cao Y, Zhang P, Dong X, et al. DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models[J]. arXiv preprint arXiv:2402.14767, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the significance of active reflection and evaluation methods. We would like to highlight that our main contribution is not only improving response quality but also providing guidance to acquire more information. \n\nWe are glad to discuss the following points:\n\n***W1. The alignment of the proposed hierarchical cognitive process***\n\n**Response:**\nThank you for questioning the hierarchical cognitive process pattern we proposed. We appreciate the opportunity to clarify the rationale behind it.\n\nThe three levels—Response Generation, Awareness of Knowledge Limits, and Knowledge Acquisition Direction—are analogies to study a model's behavior in a human cognitive way. Several studies in the field, as mentioned in our related works, evaluate model intelligence in a similar way. Our work aims to advance models from mere self-knowledge to actively seeking additional information when needed. Understanding the real cognitive process of MLLMs remains an open question. However, we believe this analogical perspective can inspire us to understand and improve MLLM performances.\n\nIn our proposed task: If a model can identify the information sufficiency and provide a relevant response, it reflects the first level (knowing what's known). If the model can identify when the available visual  information is insufficient to answer the question, it demonstrates the second layer (knowing what's unknown). Finally, if the model can suggest a direction to answer the question, it exhibits a sense of knowledge acquisition direction (knowing where to know the unknown).\n\n***W2. Comparison with training free self-reflection approach***\n\n**Response:**\nThank you for your feedback. We do not consider our method to share significant similarities with the self-reflection approach. As you mentioned, the self-reflection method can be training-free. In contrast, our main method explores generating synthetic training data to improve the model's performance, which is fundamentally different. \n\nHowever, we agree that exploring self-reflection in our task would be meaningful. Currently, we use zero-shot CoT to explore the model’s baseline performance, and training-free self-reflection can be an additional way to reflect the model’s capability.\n\nHowever, the existing self-reflection/self-critic approaches focus on LLMs in the LLM-agent or tool-learning domains. With MLLMs, self-reflection is still underexplored and still lacks a unified or fixed way. Therefore, we made a complementary experiment based on some relevant self-reflection papers. (See response to Q1).\n\n***W3. Comparison with DualFocus [1]***\n\n**Response:**\nWe would like to emphasize the significant difference between the two works. Specifically, our task is in a scenario where the required information may not be present or only partially present in the image. Instead of focusing on locating answerable cases, our task aims to develop VLMs' ability to guide users to seek additional information to answer questions. This capability is crucial for handling unanswerable cases.\n\nWe believe that both works address different challenges. We appreciate the reviewer's suggestion and will put DualFocus in our related work.\n\n***W4. The size of evaluation dataset***\n\n**Response:**\nWe appreciate the reviewer's concern on the size of our evaluation dataset. There are several points we would like to clarify in this regard:\n\nReal-World Data Collection and Cleaning: Our dataset was collected from real-world scenarios and underwent multiple rounds of rigorous data cleaning to ensure its quality. The nature of the Directional Guidance Task, which involves identifying cases with insufficient visual information, inherently results in a smaller dataset. This is because many real-world VQA datasets contain primarily high-quality, well-framed images, and the occurrence of low-quality or ill-framed cases is relatively rare. Also, to the best of our knowledge, there is no existing benchmark dataset that supports evaluating the Directional Guidance ability.\n\nDiverse Guidance Types: Despite the smaller size, we have carefully composed different groups of guidance types to maintain the comprehensiveness of our dataset. This diversity ensures that our dataset is sufficient to evaluate the model's performance across various scenarios relevant to our task. Each example in our dataset is thoughtfully chosen to represent a unique challenge in guiding the acquisition of additional information.\n\nWe also draw attention to similar work in the field, such as the paper “https://arxiv.org/abs/2404.12390” where the evaluation dataset sizes are of a comparable scale. This precedent underscores that meaningful and rigorous evaluations can still be conducted with datasets of this size. \n\nIn future work, we plan to expand our dataset further to include more diverse scenarios. However, we believe that our current dataset, despite its size, provides a valid foundation for evaluating our proposed task.\n\n***Q1: Evaluate the difference and experimental effect between this method and the large model self-reflection method.***\n\n**Response:**\nContinuing from W2, we implemented a self-reflection approach by constantly reassessing the current information and incorporating the image and all conversational contexts. It includes three steps: identifying answerability, finding the key object, and suggesting proper directional guidance. Each step involves local self-reflection to confirm the information, and interact with adjacent steps until reaching an initial answer. Finally, we integrate all conversations and self-reflect the answer with the global context.\n\nWe tested this self-reflection pipeline with GPT-4o-mini. With a series of carefully designed self-reflection pipeline and prompt templates, the F1, ACC, and ACC(F) are 0.53, 0.53, and 0.23 correspondingly, which is comparable to our reported baseline. As a reference, our best performing model achieves 0.63, 0.63, 0.43.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OIP4Ga5rep", "reviewer": "Reviewer_Eiqe", "summary": "- The paper proposes a recourse for unanswerable visual questions — rather than simply abstain, can a VLM indicate how to adjust an image so that the question becomes answerable?\n- The authors introduce the Directional Guidance VQA task and a dataset for the task.\n- The authors propose a data generation pipeline to produce data that can train models for the directional guidance task.", "questions": "- Table 1 is confusing and looks aesthetically unpleasant. \n- Your table numbers are wrong. Ex: in L295 you refer to Table 6, but this is actually Table 1 and Section 6. Your `\\label` is in the wrong place.", "limitations": "Not applicable.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The problem studied by the paper is practical and has the potential to have a real-world impact.\n- There have been a number of works on VQA models that can abstain, but abstention alone is not ideal for real-world deployments because it would be frustrating to be told that your question was unanswerable.\n- Providing a remedy for unanswerable visual questions via directional feedback is a useful and novel contribution.\n- The framing of the task is simple and reasonable.\n- The data generation pipeline is clever and uses a model-in-the-loop technique to systematically generate training instances.", "weaknesses": "Since this is somewhat of a resource paper, it would be nice to have multiple vision-language models of different sizes. This is also a bit of a concern given the fact that a model-in-the-loop technique is used to create a training dataset for the task, since different models would ostensibly have different weaknesses. \n\n\nAnother weakness is that this is a very narrowly scoped task and dataset that is maybe a better fit for a CV conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your recognition of the contribution and potential impact of our proposed study, and that's exactly what we are targeting. We also expect to facilitate many meaningful real-world applications such as AI assistants for visually impaired individuals.\n\nWe appreciate the opportunity to discuss the following points.\n\n***W1. Since this is somewhat of a resource paper, it would be nice to have multiple vision-language models of different sizes. This is also a bit of a concern given the fact that a model-in-the-loop technique is used to create a training dataset for the task, since different models would ostensibly have different weaknesses.***\n\n**Response:** To validate our training pipeline, we conducted a proof-of-concept using two mainstream open-source VLMs of different sizes. This initial approach was aimed at demonstrating the feasibility and effectiveness of our method across different model architectures and sizes.\n\nWe share your concern regarding the model-in-the-loop technique, particularly the potential for the training data to be influenced by the specific weaknesses of the models used. This is indeed similar to unsolved challenges in active learning, where the quality of training data is dependent on the model's performance during data collection.\n\nMoving forward, our work will focus on exploring methods to enhance the stability and robustness of our framework. This could include regularization techniques to mitigate biases introduced by model-specific weaknesses and integrating multiple models to diversify the training data.\n\n***W2. Another weakness is that this is a very narrowly scoped task and dataset that is maybe a better fit for a CV conference.***\n\n**Response:** Thank you for your feedback. While our study starts from a specific scenario, it is both important and generalizable. This new task fills the gap in current research regarding the handling of unknown cases and explores the capability of VLMs to acquire additional information when visual information is insufficient. Moreover, our proposed Directional Guidance Framework can be adapted and potentially applied to other tasks/modalities such as understanding query intention and reducing hallucination, where we must deal with insufficient or unclear information. These demonstrate a broader applicability of our work beyond a specific task.\n\nWe appreciate your acknowledgment of the importance and uniqueness of our study in the strength part. This guidance capability is crucial when deploying vision-language models in real-world applications, not only for specific applications, including assisting visually impaired individuals, but also for advancing the general understanding of multi-modal AI systems. Therefore, we believe the core question of whether VLMs can guide us in acquiring more information is fundamental for broader AI communities.\n\n***Q1:Table 1 is confusing and looks aesthetically unpleasant.***\n\n**Response:** Thank you for your feedback. We will edit the table in the next version to avoid confusion. For example, we will consider splitting the table into two or more parts to make the information clearer and easier to navigate, and more consistent shading will be used to highlight different sections of the table.\n\n***Q2: Your table numbers are wrong. Ex: in L295 you refer to Table 6, but this is actually Table 1 and Section 6. Your `\\label` is in the wrong place.***\n\n**Response**: Thank you for pointing out the issue with the table numbers and labels. We have reviewed and corrected the placement of the `\\label` command to ensure that the tables are referenced correctly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y2GV3Ho9GS", "reviewer": "Reviewer_SnhN", "summary": "VLMs offer an interesting opportunity to describe their uncertainty if the images given to them are of poor quality. This paper explores a framework for estimating whether a VLM can identify if its ability to correctly answer a visual question would improve were it given a better picture. The paper introduces a new labeled dataset for VQA where questions involve identifying how best to take better pictures. The paper introduces also a finetuning scheme with synthetically generated examples leveraging the labeled dataset.", "questions": "- Why is LLaVA-1.5 13b worse than 7b?", "limitations": "Limitations are lightly discussed", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem is interesting, relevant, and important\n- The novel data set is an important contribution after it is released\n- The performance improvement after fine-tuning on LLaVA is impressive.", "weaknesses": "- It would be interesting to see how the model performs with different prompting techniques. Beyond vanilla CoT, can we try methods covered in (https://arxiv.org/pdf/2402.07927)\n- The problem specifically focuses on qualitative spatial reasoning with respect to moving the camera. Can this methodology be used for quantitative reasoning (e.g., Spatial-VLM, Spatial-RGPT), by identifying not just which direction, but by how much to move the camera? \n- The two perturbation ranges of 0.1-0.9 and 0.3-0.7 are confusing. What exactly are you differentiating with these categories? It would have been interesting to evaluate over disparate ranges (e.g., 0.1-0.3, 0.3-0.5, 0.5-0.7, …) so that we can see how performance drops with the perturbations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the unique contribution of our study and the effectiveness of our training method. We believe these advancements highlight our work's potential to drive further research.\n\nWe are glad to discuss the following points:\n\n***W1. It would be interesting to see other method Beyond vanilla CoT***\n\n**Response:**\nWe agree that exploring diverse CoT methods for this task is meaningful. In our study, we experimented with a series of prompting techniques and presented the ones with the best performance based on our experiments. We will add more failure cases in the appendix in our next version to provide a comprehensive view of the challenges faced. As suggested by reviewer 1, we also made a complementary experiment with more complex zero-shot prompting techniques with the self-reflection mechanism. However, we also found that a more complex prompt does not guarantee a better performance, and that remains an interesting question.\nAs prompt design remains an open question, we acknowledge that our settings may not fully tap the model's potential to achieve the best performance. Our primary aim was to test the model's baseline performance under general prompt settings, but we appreciate the suggestion to investigate further prompting methods as outlined in the referenced paper. We plan to explore these advanced techniques in future work to benchmark the current VLMs in a more comprehensive way.\n\n***W2. Can this methodology be used for quantitative reasoning (e.g., Spatial-VLM, Spatial-RGPT), by identifying not just which direction, but by how much to move the camera?***\n\n**Response:**\nWe acknowledge that our current methodology focuses on qualitative spatial reasoning and recognize this as a limitation.\n\nTheoretically, our guidance framework can be extended to more general and quantitative scenarios. By simulating spatial drift, we can customize the ratio of drift and produce synthetic training data with quantitative values. This potential extension would allow models to identify not just the direction but also the extent of camera movement required. We plan to explore this in future work.\n\nIn our current task, we focused on assisting visually impaired individuals, which informed our data and question design. We assumed it would be challenging for users to accurately move the camera by a precise quantitative value. However, we agree that quantitative guidance would be particularly meaningful in applications such as robotics, where precise tracking of target objects is crucial. One challenge lies in calculating the ground-truth for the extent of movement needed. We are actively working on this direction, leveraging simulated environments such as Ge, Yunhao, et al. \"BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation, CVPR 2024.\" \n\n***W3. The two perturbation ranges of 0.1-0.9 and 0.3-0.7 are confusing. What exactly are you differentiating with these categories? It would have been interesting to evaluate over disparate ranges (e.g., 0.1-0.3, 0.3-0.5, 0.5-0.7, …) so that we can see how performance drops with the perturbations***\n\n**Response:**\nWe understand the confusion regarding the perturbation ranges. In our manuscript, we included a discussion on the rationale behind choosing the ranges of 0.1-0.9 and 0.3-0.7. (with * in the table below)\n\nThese ranges were selected to balance between image quality and sufficient data generation. Severe perturbations (e.g., 0.9) can lead to significant information loss, making the images harder to interpret, while minimal perturbations (0.1) do not challenge the model enough, resulting in fewer positive guidance data points.. To explore the effect of different perturbation range settings, we present the supplementary experiment results below,\n\n\nLLaVA 7B Performance Across Different Perturbation Ranges\n\n| Perturbation Range | Overall F1 | Overall Accuracy | ACC(F) |\n|:------------------:|:----------:|:----------------:|:------:|\n|      0.1-0.3       |    0.19    |       0.24       |  0.31  |\n|      0.3-0.5       |    0.49    |       0.49       |  0.38  |\n|      0.5-0.7       |    0.49    |       0.49       |  0.43  |\n|      0.7-0.9       |    0.50    |       0.49       |  0.40  |\n|      *0.1-0.9       |    0.57    |       0.58       |  0.44  |\n|      *0.3-0.7       |    0.47    |       0.48       |  0.33  |\n\nThese findings generally align with our discussion: as the value of the perturbation range increases, the overall accuracy and F1 score improve quickly and stay relatively stable around 0.49. However, if we focus on the accuracy  on four reframing directions(ACC(F)), it improves as the perturbation range increases but drops when reaching the highest range (0.7-0.9). This demonstrates there might be a tradeoff between the diversity of directional training data and the complexity of perturbed samples. A broader range (0.1-0.9) may lead to a balance and generally good performance across metrics. However, we can obeserve a general improvement with all perturbation ranges on the ACC(F). We acknowledge that studying the optimized way to choose from different ranges of perturbations could provide additional insights into performance variations. This is an interesting direction for future work, and we will consider a more detailed analysis in our next version.\n\n***Question:Why is LLaVA-1.5 13b worse than 7b?***\n\n**Response:**\nThank you for your question. We believe the difference might be due to the sensitivity of prompt settings, especially in a zero-shot scenario. Interestingly, we found that the larger model, LLaVA-1.5 13b, performed worse with a two-round prompt compared to a one-round prompt, highlighting the instability of prompt settings for different models.  We believe this could be a valuable finding and points to the need for further research to understand and optimize prompting strategies across model sizes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their time and valuable feedback. We are encouraged that the reviewers recognize our contribution in threefold:

- This work identifies a meaningful problem with real-world impact: "Can VLMs guide us in acquiring more information beyond simply abstaining?" (Reviewer *Eiqe* and Reviewer *SnhN*)
- We present a novel human-annotated Directional Guidance Benchmark Dataset (Reviewer *SnhN*)
- Our paper introduces an effective synthetic data generation and training pipeline to improve the quality of MLLM/VLM responses. (Reviewer *eg4t*, Reviewer  *Eiqe*, and Reviewer *SnhN*)
---
Additionally, we would like to address some common concerns raised by the reviewers:

- **Concern 1**. Compared with more training-free methods (such as more sophisticated CoT and self-reflection prompting) for evaluating the baseline performance. 

The prompt design is an open question, and we don't rule out that there will always be a better prompt design to get a performance boost. 
We developed a series of prompting techniques to the best of our knowledge as mentioned in Section 4, and we will include more experiment results (including failure cases) in the revision. Meanwhile, we would like to highlight that the focus of this work is on the design of the task and the (data generation and training) pipeline that significantly improves the VLM's performance on this task.

- **Concern 2**. The task is narrowly scoped and limited.

While our study is based on a specific task, its implications and applications are wide-ranging. We are glad most reviewers emphasize the importance and necessity of this task (Reviewer *Eiqe* and *SnhN*) and hit one of the pain points of existing VLMs (Reviewer *Eiqe*). We also appreciate Reviewer *Eiqe* for highlighting that our task framing is straightforward yet has a potential real-world impact. Moreover, both the task and our proposed method can be generalized to other scenarios and potentially mitigate the struggles when models face insufficient information. Our contributions have the potential to make a significant impact on both the research community and practical deployments of VLMs, affirming the relevance and importance of our work in the broader AI landscape.

Meta-review: The reviewers find the problem being studied to be relevant and important, the proposed task framing to be simple and reasonable, the proposed data generation pipeline to be clever and the generated data to be novel and useful, and the performance gains to be impressive. The reviewers had raised some concerns, but the rebuttal successfully addressed most of them and all reviewers recommend acceptance. The authors are recommended to improve the final paper version by following the reviewer recommendations.
Decision: Accept (poster)