Title: Are Vision Language Models Ready for Clinical Diagnosis? A 3D CT Benchmark for Lesion-centric Visual Question Answering
Primary area: datasets and benchmarks
Keywords: vision language models, medical image analysis, benchmarking

Abstract: Vision-Language Models (VLMs) have shown promise in various 2D visual tasks, yet their readiness for 3D clinical diagnosis remains unclear due to stringent demands for recognition precision, reasoning ability, and domain knowledge. To systematically evaluate these dimensions, we present CTLesionVQA, a diagnostic visual question answering (VQA) benchmark targeting abdominal lesions in CT scans. It comprises 9,262 CT volumes (3.7M slices) from 17 public datasets, with 395K expert-level questions spanning four categories: Recognition, Measurement, Visual Reasoning, and Medical Reasoning. CTLesionVQA introduces unique challenges, including small tumor detection and clinical reasoning across 3D anatomy. Benchmarking four advanced VLMs (RadFM, M3D, Merlin, CT-CHAT), we find current models perform adequately on measurement tasks but struggle with lesion recognition and reasoning, and do not meet clinical needs. Two key insights emerge: (1) large-scale multimodal pretraining plays a crucial role in a VLM's performance, making RadFM stand out among all VLMs. (2) Our dataset exposes critical differences in VLM components, where proper image preprocessing and design of vision modules significantly affect 3D perception.

TL;DR: We propose a modular dataset CTLesionVQA, consisting of recognition, measurement, reasoning questions, to explore how good the SOTA medical VLMs can solve real-world diagnosis problems.