Multimodal ChatGPT for Medical Applications: an Experimental Study of
  GPT-4V
Authors: name: Zhiling Yan; name: Kai Zhang; name: Rong Zhou; name: Lifang He; name: Xiang Li; name: Lichao Sun
Published: 2023-10-29 16:26:28

Abstract: In this paper, we critically evaluate the capabilities of the
state-of-the-art multimodal large language model, i.e., GPT-4 with Vision
(GPT-4V), on Visual Question Answering (VQA) task. Our experiments thoroughly
assess GPT-4V's proficiency in answering questions paired with images using
both pathology and radiology datasets from 11 modalities (e.g. Microscopy,
Dermoscopy, X-ray, CT, etc.) and fifteen objects of interests (brain, liver,
lung, etc.). Our datasets encompass a comprehensive range of medical inquiries,
including sixteen distinct question types. Throughout our evaluations, we
devised textual prompts for GPT-4V, directing it to synergize visual and
textual information. The experiments with accuracy score conclude that the
current version of GPT-4V is not recommended for real-world diagnostics due to
its unreliable and suboptimal accuracy in responding to diagnostic medical
questions. In addition, we delineate seven unique facets of GPT-4V's behavior
in medical VQA, highlighting its constraints within this complex arena. The
complete details of our evaluation cases are accessible at
https://github.com/ZhilingYan/GPT4V-Medical-Report.