Title: Right this way: Can VLMs Guide Us to See More to Answer Questions?
Venue: NeurIPS 2024
Accepted: yes
Keywords: visual accessibility

self-knowledge

vision language models
Abstract: In question-answering scenarios, humans can assess whether the available information is sufficient and seek additional information if necessary, rather than providing a forced answer. In contrast, Vision Language Models (VLMs) typically generate direct, one-shot responses without evaluating the sufficiency of the information. To investigate this gap, we identify a critical and challenging task in the Visual Question Answering (VQA) scenario: can VLMs indicate how to adjust an image when the visual information is insufficient to answer a question? This capability is especially valuable for assisting visually impaired individuals who often need guidance to capture images correctly. To evaluate this capability of current VLMs, we introduce a human-labeled dataset as a benchmark for this task. Additionally, we present an automated framework that generates synthetic training data by simulating ``where to know'' scenarios. Our empirical results show significant performance improvements in mainstream VLMs when fine-tuned with this synthetic data. This study demonstrates the potential to narrow the gap between information assessment and acquisition in VLMs, bringing their performance closer to humans.

Right this way: Can VLMs Guide Us to See More to Answer Questions?

Abstract

In question-answering scenarios, humans can assess whether the available information is sufficient and seek additional information if necessary, rather than providing a forced answer. In contrast, Vision Language Models (VLMs) typically generate direct, one-shot responses without evaluating the sufficiency of the information. To investigate this gap, we identify a critical and challenging task in the Visual Question Answering (VQA) scenario: can VLMs indicate how to adjust an image when the visual information is insufficient to answer a question? This capability is especially valuable for assisting visually impaired individuals who often need guidance to capture images correctly. To evaluate this capability of current VLMs, we introduce a human-labeled dataset as a benchmark for this task. Additionally, we present an automated framework that generates synthetic training data by simulating 'where to know' scenarios. Our empirical results show significant performance improvements in mainstream VLMs when fine-tuned with this synthetic data. This study demonstrates the potential to narrow the gap between information assessment and acquisition in VLMs, bringing their performance closer to humans. Our dataset and code are available at: https://github.com/LeoLee7/Directional_guidance .

1 Introduction

In recent years, Vision-Language Models (VLMs) have made significant strides in general multimodal tasks such as visual recognition and Visual Question Answering (VQA) [1, 49]. This progress has opened up a vast potential for various applications, including enhancing visual accessibility for visually impaired individuals [4, 20], supporting decision-making in autonomous systems [50, 32], enabling interactive technologies [39], etc. Despite these advances, VLMs still fall short of human capabilities. Humans can intuitively assess whether the available information is sufficient to answer a question and seek additional details when necessary [36, 11]. In contrast, VLMs typically tend to provide direct, single-response outputs even when information is insufficient to answer the question accurately. This limitation reduces their effectiveness in real-world applications [13]. To address this issue, recent studies have explored ways to teach VLMs to assess information sufficiency [42]. These studies aim to have VLMs either provide concrete answers or label questions as unanswerable , using benchmark datasets from real user questions like VizWiz [20].

However, a significant gap remains in handling unanswerable cases: deciding what actions to take when VLMs identify a question to be unanswerable . Humans naturally possesses the ability to seek additional details when faced with unanswerable questions - a challenge often encountered in real-world VQA tasks due to poor image quality, ambiguous questions, or loss of context [3, 9]. To

Question:

What is this?

Sorry, I'm not sure. Can you please

move the camera to right?

Question

: Can you tell me

what color my shirt is?

Sorry, it's hard to tell.

Moving camera seems not helpful.

Let's try...

Question:

What room is this?

It's 1141.

You don't have to move camera.

the best of our knowledge, no existing benchmarks focused on 'what to do' after the model identifies information insufficiency. This active process of information acquisition, fundamental to human cognition, has not been replicated in VLMs and remains largely unexplored.

To narrow the gap between VLMs and human intelligence, we suggest going beyond improving accuracy on answer generation or merely deciding on information sufficiency. Instead, we focus on enhancing the model's capability to provide constructive feedback when encountering unanswerable questions. In response to this challenge, we introduce a novel VQA task aimed at providing Directional Guidance, which aligns with real-world needs, particularly for visually impaired individuals. As indicated in previous studies [9], a common issue is that many images taken by visually impaired users are ill-framed. Our task aims to guide users on how to reframe their images during the interactive VQA process. This task evaluates the model's ability to understand visual direction and determine a potential direction to obtain more relevant information.

Moreover, to empower VLM with such guiding capability, we propose an automatic VQA data augmentation framework. This framework begins by prompting a pretrained VLM to filter a set of answerable questions from the given VQA dataset. The corresponding images are then perturbed using predefined rules that crop relevant visual information, making it more challenging for the model to answer the questions correctly. Finally, the VLM is fine-tuned using this augmented dataset, with the task of providing Directional Guidance on resolving the predefined perturbations. This approach simulates information inadequacy scenarios and holds promising potential for enhancing the model's ability to guide users in acquiring relevant information.

To validate the effectiveness of the approach, we contribute a manually labeled test set containing the Directional Guidance for real-world unanswerable datasets with images taken by visually impaired individuals. Our experiments on three popular open-source VLMs show significant improvements in the models' performance on the Directional Guidance task after fine-tuning with our synthetic training data. Notably, the best-performing model outperforms GPT-4o (CoT) [31] by 3% accuracy score.

The contributions of this study are:

Directional Guidance task: We define a novel VQA task. As shown in Figure 1, the proposed task assesses the model's ability to identify the information sufficiency and provide Directional Guidance when needed.

Directional Guidance dataset: We create a human-labeled test set to benchmark the guidance-providing capability of VLMs.

Directional Guidance framework: We propose a data-efficient framework for training models on the Directional Guidance task. This framework includes synthetic training data generation and model fine-tuning, which can be generalized with any VQA dataset with grounding information.

2 Related Work

Directional Visual Understanding. Many studies have identified that current VLMs struggle to interpret and understand spatial relationships within an input image, especially on fundamental visual concepts like relative directions [15, 40]. This ability is important for interactive VQA applications

like autonomous agents [18, 17], visual navigation, and assistive technologies designed for visually impaired individuals [24, 13]. To enhance VLMs' capability to understand directional relationships, researchers construct extensive training data [44], add assistive visual prompt [47, 30], or include collaborative VLMs to communicate and ensemble their decisions [8]. However, these methods often require heavy data collection or introduce additional models. Previous studies have investigated visual learning through simulation [18], but they rely on virtual interactive environments that may not accurately reflect real-world scenarios. Another trend involves generating training data by asking questions about directional relationships in existing images [29, 27], but this also requires additional involvement of advanced models. In our study, we aim to improve the model's directional understanding with simple data augmentation methods, using images collected from real users.

Assistive technology for visually impaired individuals. Over the past decade, applications like VizWiz [4] and Be My Eyes [13] have used real-time video connections to enhance visual accessibility. VLMs present a more accessible and responsive solution to satisfy the user's needs, as they can provide immediate responses when given a photo-query pair. However, as noted in [9], visually impaired users often face challenges in capturing clear images. In real-world conditions, many images suffer from quality issues such as blurriness, obstructions, and improper exposure, making them difficult to recognize. These issues often result in divergence in human annotations [9, 3]. Moreover, even when the images are clear, the questions may still be difficult to answer due to the off-framing of the target objects [9]. Addressing these challenges typically requires multiple rounds of queries and adjustments to properly frame the key object. For VLMs, these difficulties may be amplified because their training data typically lack examples of unrecognizable images. Additionally, to align with the multiple adjustment interaction offered by human operators in Be My Eyes applications, VLMs need to offer honest and effective guidance to navigate to target objects.

Self-knowledge. Self-knowledge refers to the model's ability to recognize what is known and unknown [22]. When confronted with unanswerable questions due to ambiguity or insufficient information, VLMs/LLMs often generate hallucinated responses [26, 46]. Previous research has introduced methods to help LLMs understand limitations regarding unknowns [48, 2, 34, 41]. Subsequent studies, such as [12], have explored explaining unanswerability by constructing known and unknown datasets through data augmentation and refining base models with a self-curation method. For VLMs, [25] presents a robust visual instruction tuning dataset that includes negative instructions at different semantic levels, i.e. nonexistent object manipulation, existent object manipulation, and knowledge manipulation, all implemented by GPT-4. Although these studies validate the benefits of data augmentation, they have focused on generating negative or unknown data primarily within the language modality. Instead, our study extends this exploration into multimodal data by incorporating the vision modality.

3 The Cognitive Question: From What's Unknown and Where to Know

To understand how a statistical model conceptualizes the world, one effective approach is to draw an analogy to human cognition. In the meta task in NLP (i.e., Question-Answering as other core NLP tasks can be transformed into QA), human cognitive processes in problem-solving and learning are multifaceted, involving not just the retrieval of stored information but also the recognition of one's knowledge boundaries and the strategic acquisition of new knowledge. To simulate these processes, we propose a hierarchical cognitive process pattern comprising three levels:

Response Generation (knowing what's known): At the foundational level, the model utilizes its existing knowledge base and basic analysis capabilities to generate responses to queries. This process mirrors the human cognitive function of retrieving known information from memory, akin to recall or recognition tasks in cognitive psychology[28, 35]. It reflects the model's ability to combine available information into coherent answers.

Awareness of Knowledge Limits (knowing what's unknown): The second level reflects the model's metacognitive ability to evaluate its own knowledge state, recognizing when it lacks sufficient information to answer a question accurately [14, 36, 37]. This awareness is crucial for intellectual honesty and mirrors the human cognitive process of monitoring and evaluating one's understanding and capabilities, a key aspect of metacognition [51, 38, 11].

Knowledge Acquisition Direction (knowing where to know the unknown): At the most advanced level, the model identifies pathways for acquiring new knowledge when existing

information is insufficient. This ability to seek out and engage in learning opportunities mirrors the human cognitive strategies for addressing knowledge gaps, such as identifying resources, formulating questions, or modifying learning strategies. It signifies the model's capacity for self-guided learning and adaptation, similar to strategic learning and problemsolving in human cognition.

As mentioned in Section 1, most existing works on VLMs cognitive questions are focused on the first two levels [42, 15, 5], and the third level is mostly under-explored. We argue that the challenges lie in the difficulty of collecting suitable data for benchmark and training data: there are few VQA samples that exhibit both awareness of knowledge limits and knowledge acquisition direction. Therefore, in our study, we focus on benchmark dataset curation and training data generation.

4 Method

4.1 Directional Guidance Task

We define our Directional Guidance task as follows: in the context of VQA, given an image-question pair < I, Q > , the model M should determine whether the image needs to be reframed. To be specific, if the target object is only partially visible and not sufficient to answer the question, the model should give clear guidance for the reframing direction (left, right, up, or down). Otherwise, the model should inform whether the question is already answerable (no need to change) or remains unanswerable even with potential reframing (none of the other options). This task mirrors real-world scenarios where visually impaired individuals need guidance to position their cameras correctly through many attempts. Although the target object might be only partially visible on each attempt, with continuous adjustments under guidance, the user can always capture a better view and finally have a better chance to get the question answered. This task goes beyond simply detecting the ill-framing issue of the image: it assesses whether the framing issue impacts the model's ability to answer the specific question posed. For example, reframing may not be necessary if the question can be directly answered with the available visual information even if the image is ill-framed. We regard these guide responses as an additional output that complements the original VQA answering process.

This task exemplifies three levels of the hierarchical cognitive pattern discussed in Section 3. Instead of a binary classification of answerable/unanswerable as proposed in [9], this task emphasizes the model's ability to effectively utilize available visual information. It requires the model to assess what is known and determine where to acquire extra information, standing in the transition from unanswerable to answerable.

4.2 Directional Guidance Dataset

Benchmark dataset. To evaluate model performance in our task setting, we created a benchmark dataset derived from VizWiz dataset families [20, 7]. The VizWiz dataset consists of real VQA queries collected from visually impaired individuals [20]. From this dataset, we used all the unanswerable samples (1.4k) from the validation set as the training set may have potential leakage issues during the pre-training process of VLMs. We invited 26 human annotators to identify ill-framed photos and label the most promising direction to move the camera, by which the reframing action could potentially help to answer the question (more details are available in Appendix A.1). After cleaning and re-evaluation, we collected 291 samples where reframing could potentially lead to an answer, and 230 samples unlikely to be answered even with reframing. The rest samples are where the human annotators have disagreements. The details of the data collection are presented in the Supplementary Materials. To ensure a balanced distribution in the test set, we randomly selected 300 samples from the VizWiz-grounding test set and simulated the case where the current image already has sufficient information to answer the question, under the assumption that the visual evidence could be theoretically grounded in the image. Combining those three groups, we get a high-quality Directional Guidance benchmark dataset including 821 samples. Despite the size of the dataset being relatively small, this reflects the inherent challenge of the task, where ill-framed images are rare in standard VQA datasets but commonly seen in real-world scenarios. Furthermore, our dataset's diversity and comprehensiveness make it suitable for evaluating model performance on the target task, providing a valuable foundation for future studies.

Question:

What is this?

Image:

-Train Set Generation:

Guidance Label:

Guidance Label:

perturbation

(i.e. move to right)

The grounding box

The grounding area

Question:

What is this?

The grounding box

Leave it unchanged

Reversed

'a dog'

'a horse'

'unanswerable'

...

The grounding area

model

Training dataset. We propose a data augmentation process to simulate the ill-framed samples, instead of collecting ad-hoc images that suit the task. Initially, we take all the training samples from a dataset pool - the validation set of VizWiz-grounding dataset [7] as it includes the visual groundings for each answerable VQA query. With that visual grounding information, manual perturbations have been applied to simulate ill-framing. Specifically, we identify the bounding box surrounding the target object and divide it into 10 zones, horizontally and vertically. We then choose a specific zone for cropping, resulting in an image that has some missing information while retaining a part of the target object. With a series of perturbations, we observe the consistency of the model's response to the initial VQA query and capture the cases where an ill-framing issue impacts the question-answering. As the VizWiz is an open-ended task, we use precision as the evaluation:

P ( w ) and T ( w ) denote a word from the model prediction and from the ground-true answer. Precision calculates how many words in the predictions also appear in the ground-truth answer, and we set a threshold e to identify the correctness. Following [19], only non-stop words have been taken into consideration. Figure 2 and algorithm 1 outline the process of generating training data with guidance labels.

Another crucial case in the benchmark test set involves samples that remain unanswered even after adjusting the camera. One more data argumentation technique has been placed: we mismatch the questions and images from the same dataset pool to create new pairs with different semantic information. Most questions in the original dataset pool are generic, as a highly frequent question is 'What is this?' without semantic information. Correspondingly, our GPT-4 enabled argumentation helped rephrase the paired question and answer. For example, given an image I i with the question Q i 'What is this?' and an answer A i 'laptop,' the new question Q ′ i will be rephrased to 'What's the color of this laptop?.' Then, we mismatch the Q ′ i with another irrelevant image I j to form a new pair. This augmentation generates complex, real-world queries where straightforward answers are infeasible, compelling models to learn deeper semantic information.

Algorithm 1: Generate synthetic training set with data augmentation Input: Output: Initialize dataset Define a range of perturbation magnitudes foreach

4.3 Experiment settings

Model Selection. To verify the feasibility and effectiveness of our approach for different model architectures and sizes, we analyze the experiments of four mainstream open-source large models with different sizes, including: LLaV A-1.5 [27], InstructBlip [10], GPT-4o [31], and CLIP [33]. First, we benchmark the test set on the LLaVA-1.5, InstructBlip, and GPT-4o on the zero-shot setting. A series of prompts has been designed to test their zero-shot performances, serving as our baselines. Next, we generate a training dataset using algorithm 1 and apply LoRA [21] fine-tuning on the open-sourced models. We anticipate the effectiveness of our proposed training framework will be reflected by the improvement of model performance compared with the zero-shot baseline.

Task format. To quantitatively analyze the model's ability to provide guidance, we format the task with a basic VQA multiple choice template: ' <image>{Original_Question} To improve the image and answer the question, how should the camera be moved? A.Leave it unchanged. B.Left. C.Right. D.Up. E.Down. F.None of the other options. ' Each option reflects the model's decision of Directional Guidance: The leave it unchanged option indicates that the current image contains all the necessary information to answer the question. The four directional options suggest that the relevant object is only partially visible, and further image adjustment is needed. The None of the other options implies that moving the camera will not help because the question is inherently unanswerable, i.e. due to the ambiguity, or the relevant object is absent from the current image. We use the F1 score and accuracy as the evaluation metrics and also analyze the confusion matrix of the different options.

Zero-shot prompt setting. For the zero-shot baseline, we enhance the basic template with additional instructions and explanations tailored for each model. We designed two prompt settings to accommodate their varying capabilities. The first setting is a single-round query where the model makes predictions from six options directly. The second setting is a two-round prompt, following the Chain-of-Thought [43] process. This two-round prompt decomposes the tasks and works as follows: Initially, we prompt the model to determine if the target object is fully present in the image, partially visible, or if the question is unanswerable. The corresponding options are: leave it unchanged , reframe , and none of the other options . If the model indicates that the target object is only partially visible, we then ask it to decide a specific direction for movement: left , right , up , or down . To ensure reproducibility, we include all prompts we used in Supplementary Materials A.5.

Fine-tune setting. In our training framework, we utilize data augmentation to generate potential samples with guidance labels. We assess the consistency of the model's predictions before and after perturbations and categorize the samples into two groups. Samples where the model fails to predict post-perturbation are considered positive, and their Directional Guidance labels are assigned one of four directions: left , right , up , or down . Conversely, samples where the model maintains correct predictions are labeled as negative, with the Directional Guidance label set to leave it unchanged . Upon analyzing these groups, we observed that negative samples predominated the generated training set. To ensure a balanced distribution within the training dataset, we under-sampled the negative samples to align with the average count of the four directional categories. We also adjusted the number of None of the other options samples to achieve an even distribution across the entire training set.

After generating the training set, we format the new pairs into a standardized instruction fine-tuning layout: each sample, comprising < I ′ , question > , is supplemented with option choices and instructions. Since the task requires models to respond with a single letter, the prediction process is equivalent to a classification task. Following the settings in [27], the loss is only computed on the token for the chosen letter and the < eos > (end of the sentence) token. Also, to prevent the model from memorizing the letter distribution, we randomly shuffle the association between letters and options, ensuring each letter (from A to F) is paired with an option randomly in each training sample. When fine-tuning each model, most training configurations follow the officially suggested settings, and more training details are presented in Supplementary Materials A.2.

None-generative Models. Since the task has been simplified to a classification problem, we also investigate whether a non-generative model with a simpler architecture could suffice. Accordingly, we add a linear probe layer onto CLIP and perform a classification head, using a vision encoder (CLIPViT-L-336px) aligned with LLaVA-1.5 and a text encoder in the default setting. We concatenate the image feature and the text feature as the input for the classification head. Since the CLIP model can not generate open-ended answers, we use the synthetic training dataset generated by LLaV A-1.5 13b.

Left

Right

Down

Up

What is in the can?

Up

What flavor of tea is this?

Down

I was wondering if you

could read what was on

the screen? I hope I got a

good picture.

Right

Can you read the screen

to me? THanks.

Left

(a)

(b)

(c)

(d)

(e)

BOISCHER

（b

1)

（

b2)

（

b3)

（

b4)

Fine-tuned

（

（

a2)

（

a3)

（

a4)

Zero-shot baseline

GPT-4o

InstructBlip 7b

LLaVA-1.5 7b

LLaVA-1.5 13b

CLIP with linear probing

InstructBlip 7b

LLaVA-1.5 7b

LLaVA-1.5 13b

dn

right left down

40

Actual

4

×

dn

right left down

Actual

2

dn

right left down

Actual

6

×

102

101

2

100

1

dn

right left down

Actual

0

O

×

dn

right left down

Actual

1

×

dn

right left down

Actual

0

2

8

×

10

10

0

10°

5 Results

5.1 Directional Guidance benchmark dataset and baseline performance

Fig. 3 (a) shows the distribution of four directions in our benchmark dataset. The horizontal directions are the most common, with left at 38.5% and right at 29.6%. The figure displays four typical samples from each direction. We also identified a frequent scenario where users need to take another photo and attempt a different question, as shown in Fig. 3 (e). This pattern reveals a common challenge for visually impaired individuals: without continuous guidance, the user and assistant can easily lose the context of the original VQA. These findings emphasize the importance of providing clear, sequential dialogue-based guidance for effectively adjusting the camera position.

As mentioned in Section 4.3, we use different prompt settings for each group of models that suit their capabilities. For the 7b models, we use the two-round prompt because these models benefit from a more structured, step-by-step approach, which helps them handle the task more effectively. In contrast, we tested the LLaVA-1.5 13b and GPT-4o model with a single-round of prompting to see if they were capable of this task. The prediction results are presented in Fig. 4, from (a1) - (a4). We observe that three open-sourced models (LLaVA-1.5 7b/13b, and Instructblip 7b) show similar behaviors: these models tend to avoid predicting the reframing cases and mistakenly categorize them

as either leave it unchanged or none of the other options. With the two-round prompt, LLaVA-1.5 7b and InstructBlip 7b make some correct predictions in the left and right categories. However, the correct and incorrect predictions are nearly balanced, with frequent misclassifications in the opposite direction. For example, the number of true left predictions is equivalent to the number of erroneous left predictions that were intended to be right . For GPT-4o, there are fewer errors in categorizing reframing cases into the wrong categories, and more cases within the reframing category are correctly predicted. However, contradictory predictions also occur frequently within the reframing cases. The result demonstrates that all the models are generally incapable of accurately predicting the reframing cases under zero-shot prompt settings. However, every baseline model performs well in the none of the other options category. We present the examples from each model in the Supplementary material A.4 to visualize the model's pretrained capability on the Directional Guidance Task.

5.2 Model's performance after fine-tuning

We present the heat maps of the fine-tuned model predictions in Fig. 4. By comparing the prediction results between the zero-shot baselines and the fine-tuned models, we observe significant and consistent improvements in prediction performance, demonstrating the effectiveness and generalizability of our proposed method. Although there is considerable potential to improve the overall accuracy, the fine-tuned models reduce confusion between reframing , leaving it unchanged (O), and none of the other options (X). The fine-tuned models are more likely to provide directional guidance on the reframing cases. Moreover, the predictions within reframing cases show noticeable improvement, as indicated by the clear diagonal line in the heat map. Another interesting finding is the substantial reduction in wrong predictions with opposite directions (e.g., predicting an up case as down ). This clarity is meaningful, as it lowers the chance of users receiving conflicting guidance, thereby enhancing safety and efficiency in real-world applications. Overall, the fine-tuned models reduce errors across all options, showing significant improvement in both cross-category and within reframing predictions.

To evaluate our training framework's sensitivity to different settings, we conducted groups of comprehensive experiments. We used three metrics to quantitatively assess the model's performance: overall F1 score, overall Accuracy, and Accuracy on the reframing cases denoted as ACC(F). The metrics for the baseline models are presented in Table 1. In some baseline experiments, we found that the zero-shot setting did not always ensure a standard output format. In such cases, we performed post-processing and excluded samples with predictions that did not fall within our options. The total number of excluded samples was fewer than 10, and this only occurred in the zero-shot baseline models. The results, as shown in Table 1, include a combination of different settings from two aspects: varying perturbation ranges and the impact of shuffling letters and options in the training data. Regarding the choice to shuffle, we observed that randomly mixing letters and options does not consistently enhance performance. For instance, with a perturbation range of 0.1-0.9, the unshuffled approach often outperformed the shuffled version, while with a perturbation range of 0.3-0.7, shuffling generally resulted in inferior performance.

The CLIP with linear probing method achieves comparable performance with the zero-shot performance of InstructBlip, but it's still not able to provide informative guidance (the accuracy for random choice for a six classification task is 16.6%). This suggests that simple CLIP-based encoders, lacking integration with a language model, may not be sufficient for this task. While the task largely relies on the model's perception of salient features, the contextual information within questions is also essential. For instance, the VizWiz dataset includes many generic questions such as 'What is the color of this?' Many of these questions are answerable even though the photos are heavily ill-framed. Since these questions do not concern spatial details, the appropriate guidance is to leave it unchanged . This underlines a key distinction between our task and other image quality detection tasks, especially those focusing on ill-framing solely on image modality.

6 Discussion

In this section, we analyze the effect of different settings, including perturbation range and shuffling operations, on the generation of training data. A detailed heatmap of the model's predictions is presented in Supplementary Materials with Figure 6. The perturbation range determines the crop

ratios used to generate the training samples, ranging from 0.1 (minimal crop) to 0.9 (maximum crop). We observed that positive Guidance samples tend to cluster at high crop ratios, while lower ratios often correspond to negative samples (where the Guidance is leave it unchanged ). Therefore, a range of 0.3-0.7 leads to a more balanced selection of training data, while a range of 0.1-0.9 provides a more comprehensive and varied dataset. This setting can affect the model's performance due to the balance between the diversity and complexity of the generated training samples, and the trade-off works as follows: when the perturbation becomes more severe (e.g., at a ratio of 0.9), images are aggressively corrupted. This increases the chance of obtaining positive Guidance samples, as the model is more likely to fail in predicting these heavily perturbed samples, which it could have predicted accurately without perturbation. However, it also results in significant information loss and greater challenges in detecting objects, making it a harder sample to learn. Conversely, a moderate perturbation ratio results in less aggressive cropping, allowing the model to access more information and better respond to the original question. However, this can lead to fewer positive Guidance samples, as the perturbation does not sufficiently challenge the predictions. The differences with the shuffling settings could be attributed to the regularization effect, which prevents the model from memorizing fixed patterns and increases training difficulty, especially when training data is scarce. In scenarios with less training data, shuffling acts as a form of data augmentation, increasing the diversity of training examples and making the model more robust. However, shuffling might add unnecessary complexity to a larger training dataset derived from a wider perturbation range, making it harder for the model to learn effectively. In such cases, the unshuffled approach allows the model to quickly identify and leverage consistent patterns, facilitating faster and more efficient learning processes.

Ablation Study To gain a more comprehensive understanding of how different perturbation ranges affect model performance, we conducted a more fine-grained ablation study with LLaVA1.5-7b. Specifically, we evaluated perturbation ranges of 0.1-0.3, 0.3-0.5, 0.5-0.7, and 0.7-0.9, alongside our main experiments of 0.1-0.9 and 0.3-0.7. The results, presented in Table 2, reveal that as the perturbation range increases from 0.1-0.3 to 0.5-0.7, both overall F1 scores and overall accuracy show substantial improvements, stabilizing around 0.49. However, at the highest perturbation range of 0.7-0.9, we observe a slight decrease in the ACC(F) metric, suggesting that overly aggressive perturbations may introduce excessive complexity, hindering the model's ability to accurately identify relevant objects. Notably, the broader range of 0.1-0.9 achieves the highest overall F1 and accuracy scores, suggesting that a wide perturbation range strikes an effective balance between data diversity and sample complexity. Additionally, all perturbation ranges demonstrate improvements in ACC(F), with enhanced reframing direction performance as perturbation increases, except at the highest range. These findings support our initial discussion by emphasizing the trade-off between data diversity and the complexity of perturbed samples. Future work could explore optimized strategies for selecting perturbation ranges, potentially employing dynamic or adaptive methods to further improve model performance based on specific dataset characteristics.

7 Limitation and Future Work

In this study, we focused specifically on guiding image reframing directions as a proof of concept. Although reframing is one of the most common needs when assisting visually impaired individuals, some other aspects that impact the VQA process could also be explored, such as orientation, exposure, and focus. Our data augmentation framework can be extended to these aspects and generate training data in a similar way, and we plan to explore this in future work. Second, the directional guidance has been simplified to a classification task on directions, which may not fully capture the complexity of real-world scenarios. For example, effective reframing might require combining multiple directions-such as moving both up and left-or even zooming out. A more informative guidance in practice would also consider additional parameters like the magnitude of the reframing action. Those complexities can confuse the model, leading to inaccurate evaluations. To enhance clarity and reduce ambiguity in our benchmark dataset, we included only cases that received consistent annotations from multiple annotators, which resulted in a limited size in our benchmark test set. Additionally, our preliminary experiments are designed to validate the effectiveness of our proposed framework rather than to maximize the model's performance. Consequently, the current method cannot fully guarantee the reliability of the model's prediction, and it still requires cautious deployment in highrisk scenarios. Moving forward, we aim to refine the task design and data generation framework, adapting more effectively to complex, real-world applications. In addition, theoretically, our guidance framework can be extended to more general and quantitative scenarios. By simulating spatial drift, we can customize the ratio of drift and produce synthetic training data with quantitative values. This potential extension would allow models to identify not just the direction but also the extent of camera movement required. This makes such quantitative guidance particularly meaningful in applications such as robotics, where precise tracking of target objects is crucial, for example, in calculating the ground truth for the extent of movement needed [16]. Furthermore, we expect our unsupervised data generation framework to alleviate the pressing data needs of studies exploring LLM or VLM for spatial or temporal reasoning tasks [6, 45, 23].

8 Conclusion

In this paper, we introduced a novel task and benchmark dataset within the context of Visual Question Answering (VQA) aimed at improving the self-knowledge of Vision-Language Models (VLMs). Our task specifically evaluates how well VLMs can assess the sufficiency of visual information and determine the necessary actions to reframe an image to obtain additional information. To address the challenge of limited training data, we proposed an automated framework that generates synthetic data by simulating unanswerable scenarios through perturbations applied to answerable cases. Our results show that current high-performing VLMs, including LLaVA and GPT-4o, struggle with this task, revealing a gap in their ability to handle incomplete or ambiguous visual inputs. However, when fine-tuned with the synthetic training data generated by our framework, the models significantly outperformed the zero-shot baseline on real-world data.

This study highlights the importance of self-knowledge in VLMs, particularly their ability to recognize the boundaries of available information and take appropriate actions when confronted with incomplete or misleading data. By mimicking human cognitive processes, our approach presents a promising solution for enhancing models' self-knowledge and robustness, especially in real-world applications that require accurate and adaptive responses. This is particularly relevant for assistive technologies, such as those designed for visually impaired individuals, where providing timely and effective guidance is essential. As VLMs continue to evolve, the ability to recognize their knowledge boundaries and make informed decisions will be critical for their successful deployment in dynamic environments. Future work can explore additional strategies for refining this self-knowledge, potentially leading to more robust models capable of learning in complex, uncertain scenarios.

9 Acknowledgment

We sincerely appreciate all collaborators who contributed to this project for their invaluable insights and support. Special thanks to our human labelers for their diligent efforts in creating the benchmark dataset. We are also grateful to Prof. David Lee and Prof. James Davis for their expert guidance, constructive feedback, and encouragement throughout this research.

We thank the VizWiz community for providing an inspiring platform that significantly influenced our work. We acknowledge the use of the VizWiz VQA and VizWiz Grounding datasets(both under CC-BY 4.0)). Our model development was based on LLaVA (Apache-2.0 license) and InstructBLIP (in compliance with the license terms of LLaMA and Vicuna).

This material is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-24-1-0149.

References

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV) , 2015.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations , 2023.

Nilavra Bhattacharya, Qing Li, and Danna Gurari. Why does a visual question have different answers? In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4271-4280, 2019.

Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology , pages 333-342, 2010.

Yuhang Cao, Pan Zhang, Xiaoyi Dong, Dahua Lin, and Jiaqi Wang. Dualfocus: Integrating macro and micro perspectives in multi-modal large language models. arXiv preprint arXiv:2402.14767 , 2024.

Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14455-14465, 2024.

Chongyan Chen, Samreen Anjum, and Danna Gurari. Grounding answers for visual questions asked by visually impaired people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19098-19107, June 2022.

Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models are visual reasoning coordinators. Advances in Neural Information Processing Systems , 36, 2024.

Tai-Yin Chiu, Yinan Zhao, and Danna Gurari. Assessing image quality issues for real-world problems. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3646-3656, 2020.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems , 36, 2024.

Andreas Demetriou and Smaragda Kazi. Self-awareness in g (with processing efficiency and reasoning). Intelligence , 34(3):297-317, 2006.

Yang Deng, Yong Zhao, Moxin Li, See-Kiong Ng, and Tat-Seng Chua. Gotcha! don't trick me with unanswerable questions! self-aligning large language models for responding to unknown questions. arXiv preprint arXiv:2402.15062 , 2024.

Be My Eyes. Introducing be my eyes virtual volunteer, 2023. Accessed: 2024-05-21.

John H Flavell. Metacognition and cognitive monitoring: A new area of cognitivedevelopmental inquiry. American psychologist , 34(10):906, 1979.

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390 , 2024.

Yunhao Ge, Yihe Tang, Jiashu Xu, Cem Gokmen, Chengshu Li, Wensi Ai, Benjamin Jose Martinez, Arman Aydin, Mona Anvari, Ayush K Chakravarthy, et al. Behavior vision suite: Customizable dataset generation via simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22401-22412, 2024.

Akshay Gopalkrishnan, Ross Greer, and Mohan Trivedi. Multi-frame, lightweight & efficient vision-language models for question answering in autonomous driving. arXiv preprint arXiv:2403.19838 , 2024.

Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4089-4098, 2018.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3608-3617, 2018.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 , 2022.

Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, and Hyunwoo J Kim. Large language models are temporal and causal reasoners for video question answering. arXiv preprint arXiv:2310.15747 , 2023.

Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics , 11:635-651, 2023.

Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations , 2023.

Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253 , 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

George Mandler. Recognizing: The judgment of previous occurrence. Psychological Review , 87:252-271, 1980.

Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens Van Der Maaten. Learning by asking questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 11-20, 2018.

Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872 , 2024.

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

SungYeon Park, MinJae Lee, JiHyuk Kang, Hahyeon Choi, Yoonah Park, Juhwan Cho, Adam Lee, and DongKyu Kim. Vlaad: Vision and language assistant for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 980-987, 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 784-789, 2018.

Henry L. Roediger and Andrew C. Butler. The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences , 15:20-27, 2011.

Gregory Schraw and David Moshman. Metacognitive theories. Educational psychology review , 7:351-371, 1995.

Nick Shannon and Bruno Frischherz. Metathinking: The Art and Practice of Transformational Thinking . Management for Professionals. Springer International Publishing, Cham, Switzerland, 1st edition, 2020.

Zhen Tan, Jie Peng, Tianlong Chen, and Huan Liu. Tuning-free accountable intervention for llm deployment-a metacognitive approach. arXiv preprint arXiv:2403.05636 , 2024.

Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209 , 2024.

Yile Wang, Peng Li, Maosong Sun, and Yang Liu. Self-knowledge guided retrieval augmentation for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 10303-10315, Singapore, December 2023. Association for Computational Linguistics.

Yuhao Wang, Yusheng Liao, Heyang Liu, Hongcheng Liu, Yu Wang, and Yanfeng Wang. Mmsap: A comprehensive benchmark for assessing self-awareness of multimodal large language models in perception. arXiv preprint arXiv:2401.07529 , 2024.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022.

Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, et al. Q-instruct: Improving low-level visual abilities for multi-modality foundation models. arXiv preprint arXiv:2311.06783 , 2023.

Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. Large language models can learn temporal reasoning. arXiv preprint arXiv:2401.06853 , 2024.

Diji Yang, Kezhen Chen, Jinmeng Rao, Xiaoyuan Guo, Yawen Zhang, Jie Yang, and Yi Zhang. Tackling vision language tasks through learning inner monologues. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 19350-19358, 2024.

Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting. Advances in Neural Information Processing Systems , 36, 2024.

Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuan-Jing Huang. Do large language models know what they don't know? In Findings of the Association for Computational Linguistics: ACL 2023 , pages 8653-8665, 2023.

Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and answering binary visual questions. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Xingcheng Zhou, Mingyu Liu, Bare Luka Zagar, Ekim Yurtsever, and Alois C Knoll. Vision language models in autonomous driving and intelligent transportation systems. arXiv preprint arXiv:2310.14414 , 2023.

Yujia Zhou, Zheng Liu, Jiajie Jin, Jian-Yun Nie, and Zhicheng Dou. Metacognitive retrievalaugmented large language models. arXiv preprint arXiv:2402.11626 , 2024.

A Appendix / supplementary material

A.1 Benchmark Dataset Collection

We invited 26 human annotators to label the 1.4k unanswerable samples extracted from the validation set of VizWiz VQA dataset[20]. We divided the dataset evenly, assigning each annotator a segment to work on. Each sample has an image and corresponding question from the original dataset. We include several sub-tasks for each VQA question to categorize how the images could be improved to better answer the question: annotators were required to choose options from the following categoriesReframing , Other Actions , or No Way to Answer . In the Reframing category, annotators specified the direction left , right , up , down -for a reframing action that might reveal an answer. The 'Other Actions' includes the actions that are beyond the previous four actions, such as zoom in/out, rotation, and adjusting exposure. The 'No Way to Answer' category is used to indicate cases where there is unlikely to yield additional information by taking any actions. We also asked the annotators to summarize their selected options with one sentence, which can be open-ended and qualitative comments. This is used for sanity checks and helps our further cleaning. After the initial round of annotations, we engaged four additional annotators (validators) to review the labels, noting any errors or disagreements. These validators conducted two rounds of evaluation and discussion before finalizing the benchmark dataset. After the validation, we select the Reframing and 'No Way to Answer' categories to join our benchmark dataset. The instructions given to annotators are:

Task Overview

Thank you for joining this task! In Visual-Question-Answering (VQA), some image-question pairs are marked "unanswerable" due to insufficient information in the image. Our goal is to determine if specific camera adjustments or other actions could potentially make these questions answerable.

For each image-question pair, your task is to identify if any adjustments or guidance could potentially make the question answerable. Please select the most applicable option from the drop-down cells and provide a brief explanation in the summary column. Below is a description of each option.

Reframing: Choose " Left, Right, Up ", or " Down " if moving the camera in a specific direction could reveal information to answer the question. Choose " Leave it unchanged " if the current image already contains all the information needed to answer the question.

Other Actions: Select " Zoom in/out, Rotate ", or " Adjust exposure " if these camera settings might improve visibility or clarify the image content.

No Way to Answer: Choose this option if no adjustment would help, such as cases with lost context, ambiguity, or obstructions.

Summary Column: In the summary column, provide a one-sentence explanation of your choice to clarify the reasoning behind your selection.

Image

Question

Summary

Reframing

OtherActions

NoWaytoAnswer

BraisedChicken

&Almonds

Question:For how long doIcook this in the

microwave?

A.2 Training details

We run experiments on an Ubuntu 22.04.4 LTS system equipped with 4*NVIDIA RTX A6000 GPU and AMD Ryzen 24-Cores CPU.

The training details for LLaV A models follow the default settings in [27], with a training epoch of 3.

The training details for InstructBlip-7b are shown as follows:

A.3 The model's performance under different settings

GPT-4v

GPT-4o

Shuffled

p: 0.3-0.7

p: 0.1-0.9

Unshuffled

Baseline

p: 0.3-0.7

p: 0.1-0.9

LLaVA-7b

InstructBlip-7b

LLaVA-13b

right left down up

2

2

Actual

1

×

dn

right left down

Actual

2

0

×

dn

right left down

5

Actual

2

×

dn

right left down

7

Actual

2

0

right left down up

Actual

2

2

×

right left down up

8

Actual

0

2

×

dn

right left down

9

Actual

7

×

dn

right left down

Actual

3

right left down up

一

Actual

LO

2

8

7

dnL

right left down

m

0

Actual

5

5

6

×

dn

right left down

3

8

Actual

一

4

0

×

102

2

101

m

100

102

m

2

101

2

3

102

7

101

3

1

100

102

2

101

m

0

100

102

m

2

101

8

2

100

102

寸

式

101

7

4

100

A.4 Examples of VLMs performances with zero-shot settings

LLaVA-1.5-7b

Leave it unchanged

Question:

The current temperature is 72 degrees

What is the current temperature?

Fahrenheit.

Left

GPT-4v

Leave it unchanged

I'm sorry,but the image provided is too

LLaVA-1.5-13bNone of the other options

blurrytoaccuratelyreadthetemperature

on thethermostat.If you canprovidea

The current temperature is 70 degrees

clearerimage,Iwouldbehappytohelp

Fahrenheit.

youwith the information displayed.

GPT-40

Left

The image provided does not clearly

InstructBlip-7b

Leave it unchanged

show thecurrent temperature.The

displayonthe deviceisnotfullyvisible,

70

makingit impossibletoread the

image of the display showing the

temperature.

LLaVA-1.5-7bNone of the other options

Question:

The captcha shows a keyboard.

What is the captcha show?

Left

GPT-4v

Leave it unchanged

Sorry,Ican't assist with thatrequest.

LLaVA-1.5-13b

None of the otheroptions

The captcha shows a red square.

GPT-40

None of the other options

The image shows a computer screen

displayingaCAPTCHA,which isa type

of challenge-response test used in

computing to determinewhetherornot

InstructBlip-7b

None of the other options

the userishuman.However,the text on

red

thescreenisnotclearenoughtoread,

soIcannotdeterminethespecific

characters orwords displayed in the

CAPTCHA.

LLaVA-1.5-7b

Left

Question:

Theimagefeaturesasnowmanholding

Could you tell me who this to and from

aladder,withabirdperchedonthe

Right

please?

ladder.Thesnowmanisstandingnext

toabirdfeeder.Thesceneisdepicted

GPT-4v

Down

on a postcard,which is addressed to

I'm sorry,butI can't assist with

"Lee." The postcard is postmarked

from ""Burlington, VT 05401.""

theenvelope.lfyou needhelpwith

something else, feel free to ask!

LLaVA-1.5-13b

Leave it unchanged

GPT-40

The envelope is addressed to"Lee"

None of the other options

I'msorry,butIcan't identifythesenderor

image showspart ofan envelopewitha

snowmanillustrationandsome

InstructBlip-7b

Right

handwrittentext,butthenamesand

addresses arenotfullyvisible.lf you could

no

provideaclearerimageof thefulladdress

andnames,Imightbe abletohelpwith

more details.

LLaVA-1.5-7bNone of the other options

Question:

Theimage doesnot depict a cologne

What kind of cologne is this?

bottle or anyscent-related objects.It

Right

GPT-4v

Right

shows a bedroom with a bed,a

suitcase, and a laptop. The focus is on

ImsorrybutIcantprovidespecificprodut

thebedroomsetting andnotonany

identification ordetailsbasedonthisimage.The

cologneorscent-relateditems.

photodoesnot clearly showthecolognebottle

orbrand,makingit mpossibletodeterminewhat

kind ofcologneit is.lfyouhavea clearerimage

ormore information about thecologne,Imight

LLaVA-1.5-13b

Noneof theotheroptions

be able to help you better.

Theimage showsa woodenshelf with a

suitcase and a laptop on it. There is no

GPT-40

Right

cologne visible in the image.

Theimagedoesnot providea clearviewof

the colognebottle,makingit difficult to

identifythebrandortypeofcologne.lf you

aboutthebottlesuchasanyvisibletextr

canprovideaclearerimageormore details

InstructBlip-7b

Leave it unchanged

distinctivefeatures,itwouldbeeasierto

cologne

help identify the cologne.

LLaVA-1.5-7b

Leave it unchanged

Question:

The image shows a can of TonyPacco's

This is a TonyPacco's chili sauce.Iwould

chili sauce,which is a hot sauce.

like to know if it's the hot or the mild?

Up

GPT-4v

Up

Theimageyou'veprovidedisblurryandcannot

clearly read the text that might indicatewhether the

Tony Packo's chili sauce ishot or mild.If there is a

LLaVA-1.5-13b

None of the otheroptions

specific label orcolorcoding onthepackaging that

differentiates the hot from the mild variety, it's not

The image shows a canof TonyPacco's

labelmorecloselyorlookforany additionalmarkings

discernible in this image.You may want to check the

chilisauce,butitdoesnotprovide

enoughinformationtodeterminewhether

it ishotormild.Thelabel on the can only

GPT-40

Up

indicatesthatitischilisauce,butthe

Theimageprovideddoesnotclearlyshowwhether

specificheatlevelcannotbedetermined

theTony Packo's chilisauce is hot or mild.Typically,

from the image alone.

this information would be indicated on the label, often

near theproduct name or in aprominent location on

the can. lf you have the can in hand, I would

InstructBlip-7b

None of the other options

notvisible,youmightwanttochecktheingredients list

indicate the heat level, such as“Hot” or “Mild."If it's

hot

orany additional descriptions on the label.

LLaVA-1.5-7b

Leave it unchanged

Question:

This is a yogurt brand called "Yoplait.""

What kind of yogurt is this?

dn

GPT-4v

Down

LLaVA-1.5-13b

None of the other options

I'm sorry,but the image you've provided is

tooblurrytodiscern anyspecificdetails

Theimage shows aperson wearinga

about theyogurtcontainer oritscontents.

blue shirt with ayogurt containerontheir

If youcanprovidea clearerimageorany

lap. The yogurt container is white and

visibletextorlogos,Imightbe ableto

has a blue label.The person is sitting on

assist you further.

a chair,and there is a cord plugged into

the yogurt container.lt is not possible to

determinethespecificbrandortypeof

GPT-40

Up

yogurt from the image alone.

The image istooblurry to clearly identify

thebrandor typeofyogurt.Thetext on the

containerisnot legible.lfyou canprovide

InstructBlip-7b

Leave it unchanged

a clearerimage ormoredetails,Imight be

able to helpbetter.

LLaVA-1.5-7b

Leaveit unchanged

Question:

Ayellowtagwiththenumber9516on it

What is this?

is placed on a white sheet.

Down

GPT-4v

Right

Themageisquitedarkandblurrymakingitdifficult

LLaVA-1.5-13b

Leave it unchanged

todiscernspecificdetails.Howeveritappearsto

show a close-upof afabric surface,possiblya

bedsheetorothertextile,withayellowobjectatthe

Theimagefeaturesabedwithawhite

bottomof theimagethat looks likea labelor tag,

comforterand a yellowmeasuring tape

which mightbepart of packaging for a product.The

label includes a letter“M"and some numbers,which

placed on topof it.The measuring tape

could bea product code oridentifier.The overall

is lying on thebed,possibly indicating

contextand therestof thescene arenotclearfrom

thatsomeoneismeasuringthesizeof

the imageprovided.

the bed or the room.

GPT-40

Down

InstructBlip-7b

Leave it unchanged

Theimage showsaclose-upofayellowandblue

packagingwiththeletterMonit,whichlikely

indicatesthe size(Medium）.The packaging also hasa

label

cut-out for hanging on a display rack and a number

“09185printed onit.Therest of theimageisdarkand

does not provide additional context.Based on the

visibleaitapprstobeproductpackaging

possibly for clothing or an accessory.

LLaVA-1.5-7b

Right

Question:

Theimage showsaplasticcontainerfilledwith

What type of medicine is this?

various medications,including toothbrushes and

toothpaste.The container is placed ona table,and

Down

GPT-4v

Down

inthe containerarelikelyusedfordentalhygiene,

there is aremote control nearby.The medications

and thepresence of theremotecontrol suggests

I'msorry,butIcannotprovidespecific

that the table might be in a living room or a similar

informationaboutmedicationsbasedon

space where people relax and watch TV.

images.It'simportanttoconsulta

healthcareprofessional orrefertothelabel

LLaVA-1.5-13b

Leaveitunchanged

on themedicationfor accurateinformation

aboutanymedicine.

The image shows abottle of medicine,

specificallyabottleofpills,sittingona

GPT-40

Down

table.The pills are white and blue in color.

However,without moreinformation about

Theimage isnotclearenoughtoidentify the

thespecificmarkingsorlabelsonthe

specifictypeof medicineinthebottle.The

bottle,it is notpossible to determine the

label isnotvisible,and there are no

exact type of medicine it contains.

distinguishingfeaturesthatcanbeused to

determinethetypeof medicine.lf you need to

InstructBlip-7bNone of the other options

checking thelabelonthebottleorconsulting

vitamin

withahealthcareprofessional.

LLaVA-1.5-7b

Right

Question:

Thismedication is Advil Cold&Sinus

What is this medication?

Leave it

Plus,which is a combination of

unchanged

GPT-4v

Leave it unchanged

medicationstorelievecoldandsinus

symptoms.It contains acetaminophen,

Themedication in the imageis"Advil Cold&Sinus

whichis ananalgesic andantipyretic,and

Plus."It is an over-the-counter medication typically

dextromethorphan,whichis an

used torelieve symptomsassociatedwiththe

antitussive.Themedicationis designed to

congestioninusressureadachesandminr

common cold and sinus infections, such as nasal

providerelief fromcoldandsinus

aches and pains.The“Plus"may indicate additional

symptoms,suchasnasal congestion,sore

active ingredientsora strongerformulation,but the

throat,headache,and fever.

specificdetailsarenotvisibleintheimageprovided.

Alwaysread the label and usemedications as

directed.

LLaVA-1.5-13b

Leave it unchanged

Advil

GPT-40

Leave it unchanged

Themedication in the imageis a white

The medication in the image is"Advil Cold&

Advil tablet,whichisapainrelieverand

Sinus Plus."It is used to relieve symptoms

anti-inflammatorymedication.

associated with coldsand sinus congestion.This

72

reliever and anti-inflammatory) and

medication typically contains ibuprofen(a pain

InstructBlip-7b

Leave it unchanged

pseudoephedrine(a decongestant).Always follow

the dosage instructions onthe label oras

vitamin

prescribed bya healthcare professional.

LLaVA-1.5-7b

Right

Question:

To dial for emergencies, you should use

Which number doI need to dial for

Leave it

the number555.

emergencies?

unchanged

GPT-4v

Leave it unchanged

Intheimageprovideditindicatesthatforan

emergencyyoushoulddial555.However,

LLaVA-1.5-13b

Leave it unchanged

pleasebeawarethat thisnumberisnota

standardemergencynumber.Inmany

To dial foremergencies,you need to

countries,thestandardemergencynumbers

dial555.

aredifferent,suchas911intheUnited

States andCanada,112in theEuropean

Inemergency

Union,and otherspecificnumbers in

dial555

different countries.Always make sure touse

the correctemergencynumberforyour

InstructBlip-7b None of the other options

location.If this image isfrom a specific

privatesystemororganization,followthe

instructions provided there.

911

GPT-40

Leave it unchanged

Foremergencies,you need to dial 555.

LLaVA-1.5-7b

Noneof theotheroptions

Question:

Blue

What color is my dog?

None of the

otheroptions

GPT-4v

None of the otheroptions

I'msorry,but theimageyou'veprovided

doesnot showa dog.lt appearstobe a

LLaVA-1.5-13b

None of the other options

close-upphotoofaperson wearinga

denimgarment,possibly ajacketorshirt.lf

Your dog is black.

you have an imageof yourdogyou'd like

toshare,I'dbehappy tohelpyou with any

questions about it.

GPT-40

None of the otheroptions

The image you provided does not show

InstructBlip-7b

Leave it unchanged

a dog.It appears tobe a close-upof a

personwearingabluedenimshirt.

Pleaseprovide animage of your dog if

youwould likemetohelpidentify its

color.

LLaVA-1.5-7b

None of theotheroptions

Question:

The remaining balance of your account

None of the

What is the remaining balance of my

is $o.

account?

otheroptions

GPT-4v

None of the other options

I'msorry,butIcannotassist withpersonalaccount

inquiries such aschecking youraccount balance.

LLaVA-1.5-13b

None of the other options

Additionally,theimageyouprovideddoesnot

containanyinformationregarding anaccount

I'm sorry,but I cannot provide the

balance orany otherdiscernibledetails.Tocheck

remainingbalanceofyouraccount as

orfinancialinstitutiondirectlyusetheirofficial

youraccountbalance,pleasecontactyourbank

Iam an Al language model and do not

website,mobileapp,rvisitanTM.Makesuret

have access to your personal financial

keep youraccount informationsecureanddonot

information.Tofindoutyouraccount

sharepersonal details online withunverified

balance,youshouldcheckyourbank

sources.

statement orcontact yourbank

GPT-40

Noneof theotheroptions

directly.

Theimage youprovided istoo blurry and

overexposed to discem any details.If you need

InstructBlip-7b

None of the other options

assistancewithyouraccountbalance,please

provide a clearerimageorcheckyouraccount

0

throughyourbank'sofficial websiteormobile app.

A.5 Prompts used for baselines

-----INSTRUCTBLIP-7B: TWO-ROUND PROMPT -----

Round 1

''Image question pair

Image: <Image> Question: { QUESTION }''

You're an assistant who helps us adjust the view of the image to better answer the given question. Given the image and question pair, your options are as follows:

A: 'framing' - If the given image contains partial needed information and the camera can move in a certain direction to gather more information for better answering the given question.

B: 'leave it unchanged' - if the image contains enough information to answer the given question, or if the given question is already answered, choose this option.

C: 'none of the other options' - If the previous options don't fit, or the given question is not related to the given image, or the given question cannot be answered, choose this option.

- If the given question is unrelated to the given image, select C: 'none of the other options'.

- If there is no way to use the given image for answering the given question, select C: 'none of the other options'.

Please only output the selected option for the given image and question pair.

A short answer to the question is:

Round 2

''Image question pair

Image: <Image> Question: { QUESTION }''

In the previous setting, the result is framing, which means that the image has part of the needed information for answering, and the camera can move to the corresponding direction to better answer the question.

You're an assistant who helps us move the camera in direction to better answer the given question. Given the image and question pair, your options are as follows:

D: 'left' - if the image is truncated on the left and the camera framing should be moved left to better answer the question based on the given image, choose this option.

E: 'right' - if the image is truncated on the right and the camera framing should be moved right to better answer the question based on the given image, choose this option.

F: 'up' - if the image is truncated on top and the camera framing should be moved up to better answer the question based on the given image, choose this option.

G: 'down' - if the image is truncated on the bottom and the camera framing should be moved down to better answer the question based on the given image, choose this option.

Please only output the selected option for direction.

A short answer to the question is:

-----GPT4: SINGLE-ROUND PROMPT -----

You are an assistive technology specializing in visual question answering, i.e., the task of providing a natural language answer to a question about a given image. To better answer the question { QUESTION } based on the given image, please choose one of the six options (A. 'Leave it unchanged', B. 'Up', C. 'Left', D. 'Right', E. 'Down', F. 'None of the other options') on the camera framing. The definitions of each of the options are given below:

A: 'Leave it unchanged' - The question can be answered based on the given image without the need for changing camera framing and there is visible text or complete object in the image to answer the question. The image is clear and shows the object in question without any truncation or need for reframing. The entire object is visible and identifiable.

B: 'None of the other options' - The question cannot be answered based on the given image, even with a change in camera framing, or the question seems to be unrelated to the content of the image provided, or the question is incomplete or the question is unrelated to the image or there is no visible text on the image to answer the question.

If the answer to the question, i.e., visible text or object is partially visible in the image or the specific text content is not clear due to the angle and quality of the image or the piece of text is partially obscured, and the necessary details to answer the question are not discernible, please choose one of the below camera framing accordingly.

C: 'Up' - The camera framing should be moved up to better answer the question based on the given image.

D: 'Left' - The camera framing should be moved left to better answer the question based on the given image.

E: 'Right' - The camera framing should be moved right to better answer the question based on the given image.

F: 'Down' - The camera framing should be moved down to better answer the question based on the given image.

Output format required is:

Option and its value.

-----GPT4: TWO-ROUND PROMPT -----

Round 1

You are an assistive technology specializing in visual question answering, i.e., the task of providing a natural language answer to a question about a given image. To better answer the question { QUESTION } based on the given image, please choose one of the given options (A. 'Leave it unchanged', B. 'None of the other options', C. 'Move camera') on the camera framing. The definitions of each of the options are given below:

A: 'Leave it unchanged' - The question can be answered based on the given image without the need for changing camera framing and there is visible text or complete object in the image to answer the question. The image is clear and shows the object in question without any truncation or need for reframing. The entire object is visible and identifiable.

B: 'None of the other options' - The question cannot be answered based on the given image, even with a change in camera framing, or the question seems to be unrelated to the content of the image provided, or the question is incomplete or the question is unrelated to the image or there is no visible text on the image to answer the question.

C: 'Move camera' - If the answer to the question i.e., visible text or object is partially visible in the image or the specific text content is not clear due to the angle and quality of the image or the piece of text is partially obscured, and the necessary details to answer the question are not discernible.

Output format required is:

Option and its value,

Reason for choosing that option,

If option A 'Leave it unchanged' is selected, give answer to question based on the given image.

Round 2

You are an assistive technology specializing in visual question answering, i.e., the task of providing a natural language answer to a question about a given image. To better answer the question { QUESTION } based on the given image or previous context {RESULT} , please choose one of the given four options (A. 'Up', B. 'Left', C. 'Down', D. 'Right') on the camera framing. The definitions of each of the options are given below:

If the answer to the question i.e., visible text or object is partially visible in the image or the specific text content is not clear due to the angle and quality of the image or the piece of text is partially obscured, and the necessary details to answer the question are not discernible, please choose one of the below camera framing accordingly.

A: 'Up' - The camera framing should be moved up to better answer the question based on the given image.

B: 'Left' - The camera framing should be moved left to better answer the question based on the given image.

C: 'Down' - The camera framing should be moved down to better answer the question based on the given image.

D: 'Right' - The camera framing should be moved right to better answer the question based on the given image.

Output format required is:

Option and its value,

Reason for choosing that option.

-----LLAVA-1.5-13b: SINGLE-ROUND PROMPT -----

USER: <image>

{QUESTION}

To improve the image and answer the above question, how should the framing should be changed in order to answer the question? Please select the most suitable ways to adjust the framing based on the following description:

Select 'none of the other options' if there is no correct way to change the framing for answering the above question, which includes: the question is not related to the image or the question has grammar error. If the Question is a sentence instead of a real question, select 'none of the other options'.

Select 'leave it unchanged' if the above question can be answered without adjusting the framing and the question has no related with the image, otherwise choose 'none of the other options'.

If the question is neither 'none of the other options' or 'leave it unchanged', and if the image has part of the needed information for answering and the camera can move to the corresponding direction to better answer the question, choose the framing option which you believe is the most suitable in order to answer the question above.

Options:

left up down

right

Please double check to make sure that you are sure about your answer and make sure that you give same priorities to each option, and give me the option only.

ASSISTANT:

-----LLaVA-1.5-7b TWO-ROUND PROMPT-----

Round 1

USER: <image>

{QUESTION}

In order to improve the image to answer the given question, please choose which category this question-image pair belongs to and the priorities of each option are the same: 'none of the other options' or 'framing' or 'leave it unchanged'. The definitions are given below:

Select 'none of the other options' if there is no correct direction to move the camera to answer the above question, which includes: the question is unrelated to the image, or the question has a grammar error.

Select 'framing' if the image has part of the needed information for answering and the camera can move to the corresponding direction to better answer the question.

Select 'leave it unchanged' if the image contains enough information for answering the above question without moving the camera to better answer the question.

- If the question is unrelated to the image, select 'none of the other options'.

- If there is no answering the question, select 'none of the other options'.

Please answer the selected category above and double check the answer. Make sure that you choose 'none of the other options' and 'framing' and 'leave it unchanged' based on their definitions and the priority of these options are the same.

ASSISTANT:

Output format required is:

Option and its value,

Reason for choosing that option,

If option A 'Leave it unchanged' is selected, give answer to question based on the given image.

Round 2

USER: <image>

{QUESTION}

In the previous setting, the result is 'framing' which means that the image has part of the needed information for answering, and the camera can move to the corresponding direction to better answer the question. Please choose the most suitable one of the four options for moving camera for better answering the question by the directions: left, right, up, down. The definitions are given below:

left - if the image is truncated on left and the camera framing should be moved left to better answer the question based on the given image.

right - if the image is truncated on right and the camera framing should be moved right to better answer the question based on the given image.

up - if the image is truncated on top and the camera framing should be moved up to better answer the question based on the given image.

down - if the image is truncated on down and the camera framing should be moved down to better answer the question based on the given image.

Answer the selected direction only.

ASSISTANT:

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction of our paper, we outline three key claims: First, we identify a significant gap between how humans and VLMs handle situations where there isn't enough visual information to answer a question. Humans look for more information to help them answer, while VLMs often either make up an answer or simply say it is unanswerable. Second, we present a benchmark dataset that clearly shows the gap. Third, we demonstrate the effectiveness of our proposed novel framework that's designed to mitigate the gap. The framework effectiveness uses data augmentation to mimic the human cognitive process, thereby improving the VLMs' ability to handle insufficient information. Our experimental results and contributions align with these claims and also demonstrate the generalizability.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We analyzed the limitations in section 7. We presented the limitations of our task setting and the sensitivity of the proposed method.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We don't have theoretical results. We are presenting the empirical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We present the steps of collecting the benchmark dataset in Supplementary Materials A.1. We also illustrate the details of our proposed methodology with Algorithm 1. We present the models we used in Section 4.3-Model Selection, and the training details are in Section 4.3-Fine-tune setting and Supplementary Materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We released the code and full benchmark dataset through our project page (link).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental settings are presented in Section 4.3 and Supplementary Material Section A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We didn't include the statistical significance analysis.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We present the computer resources in Supplementary Materials Section A.2. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research presented in the paper conforms to the NeurIPS Code of Ethics in all respects

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, we discussed the potential societal impacts of our work in the introduction part. We also analyzed the potential negative societal impacts in Section 7, pointing out the limitations regarding reliability and the risk of the model providing erroneous information.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We introduced the double verification process of collecting benchmark dataset in the Supplementary Materials (Section A.1).

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The dataset and code package we used are properly credited. We noted and followed their license and terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We released the new assets and documents through the project url.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We present the human annotation instructions and screenshot in Supplementary material A.1. In our annotation process, all participants received fair compensation based on the average market rate for similar studies.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [No]

Justification: Our research is pending on an IRB exemption.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
VLMs offer an interesting opportunity to describe their uncertainty if the images given to them are of poor quality. This paper explores a framework for estimating whether a VLM can identify if its ability to correctly answer a visual question would improve were it given a better picture. The paper introduces a new labeled dataset for VQA where questions involve identifying how best to take better pictures. The paper introduces also a finetuning scheme with synthetically generated examples leveraging the labeled dataset.

STRENGTHS:
- The problem is interesting, relevant, and important
- The novel data set is an important contribution after it is released
- The performance improvement after fine-tuning on LLaVA is impressive.

WEAKNESSES:
- It would be interesting to see how the model performs with different prompting techniques. Beyond vanilla CoT, can we try methods covered in (https://arxiv.org/pdf/2402.07927)
- The problem specifically focuses on qualitative spatial reasoning with respect to moving the camera. Can this methodology be used for quantitative reasoning (e.g., Spatial-VLM, Spatial-RGPT), by identifying not just which direction, but by how much to move the camera? 
- The two perturbation ranges of 0.1-0.9 and 0.3-0.7 are confusing. What exactly are you differentiating with these categories? It would have been interesting to evaluate over disparate ranges (e.g., 0.1-0.3, 0.3-0.5, 0.5-0.7, …) so that we can see how performance drops with the perturbations

QUESTIONS:
- Why is LLaVA-1.5 13b worse than 7b?

LIMITATIONS:
Limitations are lightly discussed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

comments: SUMMARY:
- The paper proposes a recourse for unanswerable visual questions — rather than simply abstain, can a VLM indicate how to adjust an image so that the question becomes answerable?
- The authors introduce the Directional Guidance VQA task and a dataset for the task.
- The authors propose a data generation pipeline to produce data that can train models for the directional guidance task.

STRENGTHS:
- The problem studied by the paper is practical and has the potential to have a real-world impact.
- There have been a number of works on VQA models that can abstain, but abstention alone is not ideal for real-world deployments because it would be frustrating to be told that your question was unanswerable.
- Providing a remedy for unanswerable visual questions via directional feedback is a useful and novel contribution.
- The framing of the task is simple and reasonable.
- The data generation pipeline is clever and uses a model-in-the-loop technique to systematically generate training instances.

WEAKNESSES:
Since this is somewhat of a resource paper, it would be nice to have multiple vision-language models of different sizes. This is also a bit of a concern given the fact that a model-in-the-loop technique is used to create a training dataset for the task, since different models would ostensibly have different weaknesses. 


Another weakness is that this is a very narrowly scoped task and dataset that is maybe a better fit for a CV conference.

QUESTIONS:
- Table 1 is confusing and looks aesthetically unpleasant. 
- Your table numbers are wrong. Ex: in L295 you refer to Table 6, but this is actually Table 1 and Section 6. Your `\label` is in the wrong place.

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

comments: SUMMARY:
This paper evaluates and expands the self-reflection capabilities of multiple MLLMs, so that MLLMs can actively evaluate the adequacy of given query information and give suggestions on how to find more reliable information. This paper proposes a hierarchical cognitive process pattern theory and creates a manually annotated dataset for evaluation.

STRENGTHS:
Methods for active reflection and evaluation of given information are meaningful for enhancing the quality of MLLM responses.

WEAKNESSES:
1. The rationality of the proposed hierarchical cognitive process pattern, which comprises three levels, merits validation. First, are these three levels both necessary and sufficient for the cognitive process? If the issues at these three levels are addressed, can we then resolve the problems associated with the MLLM cognitive process? Second, does the evaluation benchmark and the ability to expand MLLM, as presented in this paper, align with these three levels?

2. The method proposed in this paper is fundamentally similar to the MLLM self-reflection method. Can it be directly implemented through self-reflection without requiring additional training?

3. A comparable method to the one presented in this paper is DualFocus [1], which not only actively evaluates image quality but also performs more sophisticated active grounding.

4. Lastly, the evaluation dataset used in this paper is relatively small, which makes the evaluation results less convincing.

[1]Cao Y, Zhang P, Dong X, et al. DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models[J]. arXiv preprint arXiv:2402.14767, 2024.

QUESTIONS:
Evaluate the difference and experimental effect between this method and the large model self-reflection method.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13