Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Authors: name: Yuxuan Qiao; name: Haodong Duan; name: Xinyu Fang; name: Junming Yang; name: Lin Chen; name: Songyang Zhang; name: Jiaqi Wang; name: Dahua Lin; name: Kai Chen
Published: 2024-06-20 17:54:03

Abstract: Vision Language Models (VLMs) demonstrate remarkable proficiency in
addressing a wide array of visual questions, which requires strong perception
and reasoning faculties. Assessing these two competencies independently is
crucial for model refinement, despite the inherent difficulty due to the
intertwined nature of seeing and reasoning in existing VLMs. To tackle this
issue, we present Prism, an innovative framework designed to disentangle the
perception and reasoning processes involved in visual question solving. Prism
comprises two distinct stages: a perception stage that utilizes a VLM to
extract and articulate visual information in textual form, and a reasoning
stage that formulates responses based on the extracted visual information using
a Large Language Model (LLM). This modular design enables the systematic
comparison and assessment of both proprietary and open-source VLM for their
perception and reasoning strengths. Our analytical framework provides several
valuable insights, underscoring Prism's potential as a cost-effective solution
for vision-language tasks. By combining a streamlined VLM focused on perception
with a powerful LLM tailored for reasoning, Prism achieves superior results in
general vision-language tasks while substantially cutting down on training and
operational expenses. Quantitative evaluations show that Prism, when configured
with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on
par with VLMs 10 times larger on the rigorous multimodal benchmark MMStar.
The project is released at: https://github.com/SparksJoe/Prism.