VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large
  Vision-Language Models in Fact-Seeking Question Answering
Authors: name: Yanling Wang; name: Yihan Zhao; name: Xiaodong Chen; name: Shasha Guo; name: Lixin Liu; name: Haoyang Li; name: Yong Xiao; name: Jing Zhang; name: Qi Li; name: Ke Xu
Published: 2025-03-09 07:25:32

Abstract: Large vision-language models (LVLMs) have demonstrated remarkable
achievements, yet the generation of non-factual responses remains prevalent in
fact-seeking question answering (QA). Current multimodal fact-seeking
benchmarks primarily focus on comparing model outputs to ground truth answers,
providing limited insights into the performance of modality-specific modules.
To bridge this gap, we introduce VisualSimpleQA, a multimodal fact-seeking
benchmark with two key features. First, it enables streamlined and decoupled
evaluation of LVLMs in visual and linguistic modalities. Second, it
incorporates well-defined difficulty criteria to guide human annotation and
facilitates the extraction of a challenging subset, VisualSimpleQA-hard.
Experiments on 15 LVLMs show that even state-of-the-art models such as GPT-4o
achieve merely 60%+ correctness in multimodal fact-seeking QA on VisualSimpleQA
and 30%+ on VisualSimpleQA-hard. Furthermore, the decoupled evaluation across
these models highlights substantial opportunities for improvement in both
visual and linguistic modules. The dataset is available at
https://huggingface.co/datasets/WYLing/VisualSimpleQA.