VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large
  Vision-Language Models in Fact-Seeking Question Answering
Authors: name: Yanling Wang, name: Yihan Zhao, name: Xiaodong Chen, name: Shasha Guo, name: Lixin Liu, name: Haoyang Li, name: Yong Xiao, name: Jing Zhang, name: Qi Li, name: Ke Xu

Large vision-language models (LVLMs) have demonstrated remarkable
achievements, yet the generation of non-factual responses remains prevalent in
fact-seeking question answering (QA). Current multimodal fact-seeking
benchmarks primarily focus on comparing model outputs to ground truth answers,
providing limited insights into the performance of modality-specific modules.
To bridge this gap, we introduce VisualSimpleQA, a multimodal fact-seeking
benchmark with two key features. First, it enables streamlined and decoupled
evaluation of LVLMs in visual and linguistic modalities. Second, it
incorporates well-defined difficulty criteria to guide human annotation and
facilitates the extraction of a challenging subset, VisualSimpleQA-hard.
Experiments on 15 LVLMs show that even state-of-the-art models such as GPT-4o
achieve merely 60%+ correctness in multimodal fact-seeking QA on VisualSimpleQA
and 30%+ on VisualSimpleQA-hard. Furthermore, the decoupled evaluation across
these models highlights substantial opportunities for improvement in both
visual and linguistic modules. The dataset is available at
https://huggingface.co/datasets/WYLing/VisualSimpleQA.