VisOnlyQA: Large Vision Language Models Still Struggle with Visual
  Perception of Geometric Information
Authors: name: Ryo Kamoi; name: Yusen Zhang; name: Sarkar Snigdha Sarathi Das; name: Ranran Haoran Zhang; name: Rui Zhang
Published: 2024-12-01 19:46:22

Abstract: Errors in understanding visual information in images (i.e., visual perception
errors) remain a major source of mistakes in Large Vision Language Models
(LVLMs). While further analysis is essential, there is a deficiency in datasets
for evaluating the visual perception of LVLMs. In this work, we introduce
VisOnlyQA, a new dataset designed to directly evaluate the visual perception
capabilities of LVLMs on questions about geometric and numerical information in
scientific figures. Our dataset enables us to analyze the visual perception of
LVLMs for fine-grained visual information, independent of other capabilities
such as reasoning. The evaluation set of VisOnlyQA includes 1,200
multiple-choice questions in 12 tasks on four categories of figures. We also
provide synthetic training data consisting of 70k instances. Our experiments on
VisOnlyQA highlight the following findings: (i) 20 LVLMs we evaluate, including
GPT-4o and Gemini 1.5 Pro, work poorly on the visual perception tasks in
VisOnlyQA, while human performance is nearly perfect. (ii) Fine-tuning on
synthetic training data demonstrates the potential for enhancing the visual
perception of LVLMs, but observed improvements are limited to certain tasks and
specific models. (iii) Stronger language models improve the visual perception
of LVLMs. In summary, our experiments suggest that both training data and model
architectures should be improved to enhance the visual perception capabilities
of LVLMs. The datasets, code, and model responses are provided at
https://github.com/psunlpgroup/VisOnlyQA.