TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions

Authors: Xingwei He, Qianru Zhang, A-Long Jin, Yuan Yuan, Siu Ming Yiu
Primary area: datasets and benchmarks
Keywords: Vision-Language Models, Unanswerable Questions, Benchmark, Hallucination, Trustworthiness

Abstract:
Large Vision-Language Models (LVLMs) have achieved remarkable progress on visual perception and linguistic interpretation. Despite their impressive capabilities across various tasks, LVLMs still suffer from the issue of hallucination, which involves generating content that is incorrect or unfaithful to the visual or textual inputs. Traditional benchmarks, such as MME and POPE, evaluate hallucination in LVLMs within the scope of visual question answering (VQA) using answerable questions. However, some questions are unanswerable due to insufficient information in the images, and the performance of LVLMs on such unanswerable questions remains underexplored. To fill in this research blank, we propose TUBench, a benchmark specifically designed to evaluate the reliability of LVLMs using unanswerable questions. TUBench comprises an extensive collection of high-quality, unanswerable questions that are meticulously crafted using ten distinct strategies. To thoroughly evaluate LVLMs, the unanswerable questions in TUBench use images from four diverse domains as visual contexts: screenshots of code snippets, natural images, geometry diagrams, and screenshots of statistical tables. These unanswerable questions are tailored to test LVLMs' trustworthiness in code reasoning, commonsense reasoning, geometric reasoning, and mathematical reasoning related to tables, respectively. We conducted a comprehensive quantitative evaluation of 28 leading foundational models on TUBench, with Gemini-1.5-Pro, the top-performing model, achieving an average accuracy of 69.2\%, and GPT-4o, the third-ranked model, reaching 66.7\% average accuracy, in determining whether questions are answerable. Furthermore, our manual analysis of the model outputs reveals that: (1) Gemini-1.5-Pro provides both correct answers and explanations in only 41\% of cases, and (2) hallucinations are the primary cause of error, accounting for 58.5\% of the incorrect explanations generated by Gemini-1.5-Pro. These findings highlight that TUBench presents a significant challenge to current LVLMs, and offers a new perspective for evaluating hallucinations and trustworthiness through the lens of unanswerable questions.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730343403012

thread_type:
official_review

<thread_object>
creator: reviewer_653o

<content>
summary:
This paper introduces a new benchmark called TUBench designed to evaluate Large Vision-Language Models (LVLMs) for reliability using unanswerable questions. Unlike previous benchmarks that focus on answerable visual questions, TUBench uses unanswerable queries based on images from diverse domains like code snippets, natural scenes, geometry, and statistical tables to test the models' trustworthiness. Through experiments on 28 models, the study reveals that hallucination is a primary challenge, with even top models like Gemini-1.5-Pro achieving only modest success in distinguishing answerability, highlighting significant gaps in current LVLM capabilities.

strengths:
1. This paper focuses on unanswerable questions across diverse image types (e.g., code snippets, natural scenes, geometric diagrams, and statistical tables), TUBench provides a unique angle for assessing LVLM trustworthiness, setting it apart from benchmarks that focus only on answerable questions.
2. TUBench’s dataset of 2,354 questions includes 1,151 unanswerable questions constructed using 10 strategies. The manual curation ensures high quality and relevance, providing nuanced and challenging unanswerable queries.
3.The paper includes detailed analyses of model errors, specifically investigating the role of hallucination and overconfidence in LVLMs when faced with unanswerable questions. This analysis underscores a critical shortcoming in current models.

weaknesses:
1. Can the authors provide suggestions for future LVLM training to solve the "unanswerable" questions? From both the model perspective and training data perspective.

2. Although human-annotated questions ensure high quality, manual creation of unanswerable questions may introduce subjective biases or inconsistencies, which could affect reproducibility or lead to unintentional patterns that models might exploit.

questions:
The questions are mentioned in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730510304139

thread_type:
official_review

<thread_object>
creator: reviewer_gyxq

<content>
summary:
The paper introduces TUBench, a novel benchmark specifically designed to assess the reliability of large vision-language models (LVLMs) when faced with unanswerable questions. TUBench includes a diverse set of high-quality, manually crafted unanswerable questions across four visual domains. This benchmark evaluates trustworthiness by focusing on LVLMs' ability to refuse to answer when information is insufficient. The contributions are:
1. A systematic dataset of 2,354 questions split into answerable and unanswerable subsets, carefully crafted to challenge LVLMs.
2. A comprehensive evaluation of 28 LVLMs, revealing substantial room for improvement in identifying unanswerable questions.
3. Analysis showing hallucination as a primary source of errors, highlighting the gap in current LVLMs' reliability in real-world applications.

strengths:
1. TUBench introduces a new problem and a new dataset for evaluating Large Vision-Language Models (LVLMs) by focusing on unanswerable questions.
2. The TUBench is well-structured with various domains that can happen in real life. The evaluation scale is large, containing 28 MLLMs. 
3. The paper is mostly well-written and easy to follow.

weaknesses:
1. Some examples may be a bit not rigorous, for instance, in the bottom left subfigure in Fig 1, if the reason that the code snippet is not complete leads to the unanswerable for the first two questions, then the last two questions are also not answerable: other part of the code may change the result.
2. The original performance of MLLMs on the GeoQA dataset seems very low; in this case, it does not make much sense to expect them to judge the answerability of the question by removing some conditions. Mostly, one has to know how to solve the problem to judge the answerability of it. About this point, can the authors show the best results of MLLMs on the answerable subset of the dataset?

questions:
1. Line 414 refers to Figure 13, but Figure 13 does not show that GPT4o is wrong in question answerability; it just got the wrong answer. This example seems marginal. 
2. The paper could address more on the significance/importance of the proposed problem: what bad outcomes in which situations would the weakness of MLLMs on this problem lead?
3. Does the 'hallucination' mentioned in the paper mean 'object hallucination'? How is the error in the UCR dataset classified in The result in Fig 5 (b)?
4. It is not quite clear how exactly OAAC is calculated, can the author provide the equation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693516669

thread_type:
official_review

<thread_object>
creator: reviewer_ky3c

<content>
summary:
This paper presents a new benchmark to evaluate the reliability of LVLMs with both answerable and unanswerable questions, which is different from the SOTA ones that focus on answerable questions. The benchmark contains images from four domains including code snippet screen shots, natural images, geometry diagrams and statistical table screen shots. The benchmark is used to evaluate 28 foundational models in terms of determining question's answerability and others. The evaluation results indicate that the STOA LVLMs do not perform well in answering unanswerable questions.

strengths:
1. The presented work is well motivated. While the existing benchmarks overlook unanswerable questions or used heuristic rules to automatically generated unanswerable questions, this proposed new benchmark that create unanswerable yet closely related questions and present a mixture of answerable and unanswerable ones to evaluate LVLMs. 

2. The evaluation of 28 leading foundation LVLMs using the new benchmark reveals low accuracy even for the best-performing model, which indicates the need for improvement. 

3. The paper is well organized and written, with clear and detailed description of the benchmark, the evaluation methodology and evaluation results.

weaknesses:
1. It would be better to study and compare the performance of the LVLM models when they are presented with only answerable questions and with a mixture of answerable and unanswerable questions. Such a ablation study would help to quantify the impact of the presence of unanswerable questions. The current study only uses a mixture of answerable/unanswerable questions. 

2. It is claimed that identifying answerability is more challenging, but this claim needs more clear/systematic justification. For example, on page 9 line 450 - "unanswerable questions are challenging for current VLMs, with out 44% of Gemini-1.5-pro's outputs providing correct answers and explanations"; however, according to Fig 5 (a), it appears that for both answerable and unanswerable questions, only 41% of Gemini-1.5-Pro's outputs provide correct answers and explanations. Does this indicate correctly answering the answerable questions is also challenging?

questions:
Among the evaluation metrics, F1 and 2ACC are already used to measure the performance of identifying unanswerable questions, why is OACC introduced to measure the combination of both accuracy of answerability classification and the accuracy for answerable questions? Is the redundant? Is it better to have the third metric measure only the accuracy of answerable questions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695016496

thread_type:
official_review

<thread_object>
creator: reviewer_m7wd

<content>
summary:
This paper introduces TUBench, a benchmark aimed at assessing the reliability of Language-Vision Models (LVLMs) by using unanswerable questions. TUBench features a diverse set of high-quality unanswerable questions, created using ten distinct strategies, to rigorously test LVLMs’ responses. The benchmark draws from four varied visual domains—code snippets, natural images, geometry diagrams, and statistical tables—to examine trustworthiness in specific reasoning areas: code, commonsense, geometry, and mathematical reasoning.

strengths:
1. This paper evaluates the trustworthiness of VLMs across four reasoning domains applying 10 approaches to address unanswerability.
2. The paper provides in-depth analysis of model performance, including sensitivity to hallucination and question answerability, which gives a nuanced understanding of model limitations.

weaknesses:
1. The generation of approach for UVQA and UTabMWP domains are less diverse; specifically in UVQA, perturbations in relations/attributes using scene graphs to generate unanswerable questions have been already addressed in current literature [a].  The fixed number of perturbations can easily be included in a few-shot way to the VLM to teach unanswerability to VLM.
2. The effect of changing the ordering of MCQ options is not explored, although prior works have found bias in VLMs in option ordering.

(a) https://arxiv.org/pdf/2403.10534

questions:
Comment: Recent works tried to address unanswerability [a] by perturbing relations/attributes and hallucinations [b] of VLMs which are worth exploring.

Q1. UCR, UVQA are posed in Y, N, U option order. Has there been analysis with changing the option order?

Q2. In UTabMWP, table being incomplete derives in Unanswerable situation which is very easy to embed into the VLM with few-shot examples.

Q3. UGeoQA contains questions in Chinese and most VLMs are predominantly English. Therefore, it is unclear if the ineffectiveness of the VLMs derive from inability to process Chinese or actual reasoning.

Q4. What the 'Other' category of explanation evaluation categories consists?




(a) https://arxiv.org/pdf/2403.10534
(b) https://www.researchgate.net/publication/375595895_HALLUSIONBENCH_You_See_What_You_Think_Or_You_Think_What_You_See_An_Image-Context_Reasoning_Benchmark_Challenging_for_GPT-4Vision_LLaVA-15_and_Other_Multi-modality_Models
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presentend a new benchmarks called TUBench, aimed at examining the trustworthness of VLMs. The authors pointed that LLMs and VLMs incline to produce answers to questions, even for those that are not answerable given the input image and text. To evaluate the VLMs more comprehensively, the authors developed a new QA datasets which explicitly contains both answerable and unanswerable questions for given images. The authors deliberatly curated the image QA pairs from four different domains and perform some variantions to the original QA pairs to make them unanswerable. Based on this new datasets, the authors conducted extensive evaluations for 28 leading VLMs. The experimental results show that the most advanced model like Gemini or GPT-4o still struggle to determine whether the question is answerable or not. This finding calls for further research and development of more reliable VLMs when handling unanswerable visual questions.

The main strength of this work is that the authors proposed a new benchmark to evaluate the trustworthness of VLMs. Through extensive evaluations for various VLMs, the authors showed that the top performed models still lack the ability to distinguish unanswerable questions from answerable ones. Clearly, this study conveys some new insights regarding the incapability of VLMs to cope answerable and unanswerable questions. Nevertheless, all reviewers pointed out that the proposed benchmark is heuristic and prone to unintendend bias introduced by the method used to synthesize unanswerable questions. This as a result will render biased evaluation results on the VLMs. With this, the ACs also agree and have a concern about the rigorousness of the proposed benchmark, given that all samples are designed heuristically and the total number of samples is also very limited. Furthermore, the ACs agreed with the reviewer gyxQ that the definition of answerable and unanswerable questions is a bit vague. For example, the question at Fig. 1 top right, "Is the output of the code ‘Hello’?" is considered as unanswerable question. However, this is more like a question that the model cannot answer deterministically. In this sense, the ACs suggest the authors make the definition of different types of questions more clear, so that the evaluations on top of this become more rigorous and indicative to the real capability of VLMs.

Based on the reviews and discussions durring rebuttal session, the ACs think this work still has some clear drawbacks which could be addressed by the authors in the revision. However, at this point, it does not pass the bar of this venue.