CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning
  Capabilities of VLMs
Authors: name: Ai Jian; name: Weijie Qiu; name: Xiaokun Wang; name: Peiyu Wang; name: Yunzhuo Hao; name: Jiangbo Pei; name: Yichen Wei; name: Yi Peng; name: Xuchen Song
Published: 2025-05-30 01:34:25

Abstract: Vision-Language Models (VLMs) have demonstrated remarkable progress in
multimodal understanding, yet their capabilities for scientific reasoning
remains inadequately assessed. Current multimodal benchmarks predominantly
evaluate generic image comprehension or text-driven reasoning, lacking
authentic scientific contexts that require domain-specific knowledge
integration with visual evidence analysis. To fill this gap, we present CSVQA,
a diagnostic multimodal benchmark specifically designed for evaluating
scientific reasoning through domain-grounded visual question answering.Our
benchmark features 1,378 carefully constructed question-answer pairs spanning
diverse STEM disciplines, each demanding domain knowledge, integration of
visual evidence, and higher-order reasoning. Compared to prior multimodal
benchmarks, CSVQA places greater emphasis on real-world scientific content and
complex reasoning.We additionally propose a rigorous evaluation protocol to
systematically assess whether model predictions are substantiated by valid
intermediate reasoning steps based on curated explanations. Our comprehensive
evaluation of 15 VLMs on this benchmark reveals notable performance
disparities, as even the top-ranked proprietary model attains only 49.6\%
accuracy.This empirical evidence underscores the pressing need for advancing
scientific reasoning capabilities in VLMs. Our CSVQA is released at
https://huggingface.co/datasets/Skywork/CSVQA.